JP2019129470A - Image processing device - Google Patents

Abstract

To provide an image processing device that detects good motion vectors of a main subject and a background even when the magnification and rotation components of an image change due to the movement of a subject.SOLUTION: When an image of a main subject position is converted into polar coordinates, the image is corrected on the basis of the magnification and rotation variation calculated by first magnification and first rotation variation calculation means for calculating the magnification and the rotation variation of the image. When the first motion vector detection is performed by using the corrected image, a motion vector of a main subject is detected, and the image is corrected on the basis of the magnification and rotation variation calculated by a method different from the first magnification and rotation variation calculation means. When the second motion vector detection is performed by using the corrected image, the motion vector of the background subject is detected.SELECTED DRAWING: Figure 2

Description

  The present invention relates to an image processing apparatus.

  As a method for detecting image shake due to shake and motion of the imaging apparatus, there is a method of calculating a movement amount (motion vector) between images using a vector detection method such as template matching.

  The motion detection method using template matching is weak to changes in the template image, and if the texture of the image changes due to a change in magnification or a rotational change in the in-plane direction of the image, the performance of vector detection is degraded.

  For this reason, in the movement vector detection method disclosed in Patent Document 1, when the optical zoom magnification varies, the reference image or the comparison image is set so that the reference image and the comparison image have the same magnification based on the information of the optical zoom magnification. The movement vector is detected after the electronic zoom correction is performed on the image.

  In the camera shake correction method disclosed in Patent Literature 2, a rotation component in the image plane direction of the imaging device is calculated by performing polar coordinate conversion on the calculated motion vector, and the rotation component extracted from the calculated motion vector The motion vector is detected by removing the

JP, 2008-160274, A JP, 2012-199814, A

  However, when the subject is moving, the positional relationship between the subject and the imaging device changes, so it is not possible to obtain a change in image magnification from the information on the optical zoom magnification as in Patent Document 1.

  In the method of extracting rotation components by polar coordinate conversion of motion vectors as in Patent Document 2, different motion vectors coexist when the subject and the background move differently in the image, so the rotation component extraction result is erroneous. The motion vector detection accuracy may deteriorate.

In order to solve the above problems, an image processing apparatus according to the present invention provides:
Image main subject position information acquisition means;
A first magnification for calculating the magnification of the image and the amount of change of the image by polar coordinate conversion of the image of the main object position based on the main object position information;
A second magnification for calculating an image magnification and a rotation change amount by a method different from the calculated method, a rotation change amount calculating means;
Image correction means for correcting image magnification and rotation;
Vector detection means for performing motion vector detection using the corrected image;
Have
An image correction is performed based on the first magnification, the magnification calculated by the rotation change amount calculation means, and the rotation change amount, and the first motion vector is detected using the corrected image, thereby moving the motion vector of the main subject. To detect
The image is corrected based on the second magnification, the magnification calculated by the rotation change amount calculation means, and the rotation change amount, and the motion of the background subject is detected by performing the second motion vector detection using the corrected image. It is characterized by detecting a vector.

  According to the image processing apparatus of the present invention, it is possible to detect good motion vectors of the main subject and the background even when the magnification of the image and the rotation component change due to the movement of the subject.

Block diagram showing the functional configuration of the digital camera Block diagram showing the configuration of the image processing unit in the first embodiment Flowchart showing the overall operation in the first embodiment Schematic diagram for explaining image polar coordinate conversion processing Schematic diagram explaining an example of logarithmic polar transformation of an image A schematic diagram for explaining an example of log polar coordinate conversion of an image based on main subject position information Schematic diagram explaining electronic correction processing for the purpose of vector detection of the main subject Schematic diagram explaining electronic correction processing for the purpose of vector detection of the background Block diagram showing the configuration of the image processing unit in the second embodiment Flowchart showing the overall operation in the second embodiment Schematic diagram illustrating a background image position / region selection method according to the second embodiment.

  DESCRIPTION OF EMBODIMENTS Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the drawings. In the embodiment described below, an example in which the present invention is applied to a digital camera as an example of image processing will be described.

(First embodiment)
FIG. 1 is a block diagram showing a functional configuration of a digital camera according to an embodiment of the present invention.

  The control unit 101 is, for example, a CPU, reads out an operation program of each block included in the digital camera 100 from the ROM 102, expands it in the RAM 103, and executes it to control the operation of each block included in the digital camera 100.

  The ROM 102 is a rewritable non-volatile memory, and stores parameters and the like necessary for the operation of each block in addition to the operation program of each block included in the digital camera 100.

  The RAM 103 is a rewritable volatile memory, and is used as a temporary storage area of data output in the operation of each block included in the digital camera 100.

  The optical system 104 is an optical system composed of lenses and the like, and includes a focus lens, a zoom lens, and their drive systems. The imaging unit 105 is an imaging device such as a CMOS sensor, photoelectrically converts an optical image formed on the imaging device by the optical system 104, and outputs an obtained analog image signal to the A / D conversion unit 106.

  The A / D conversion unit 106 applies A / D conversion processing to the input analog image signal, and outputs the obtained digital image data to the RAM 103 for storage. The optical magnification information obtained from the drive position of the zoom lens at the timing corresponding to the stored image data is output to the RAM 103 and stored.

  The focusing processing unit 107 has a function of performing focusing processing of a conventionally used method such as a contrast method or a phase difference method, and a drive system of a focus lens included in the optical system 104 is a control unit 101. Control is performed so that the subject is automatically focused. As a method of specifying the main subject position or area to be focused, a method of automatically detecting and a method of specifying by the user can be selected.

  Automatic detection methods include a method using changes in luminance and color of an image (refer to JP-A-11-190816), a method using distance information (refer to JP-A-2007-96437), and a luminance gradient called saliency The method of using the change (Japanese Patent Laid-Open No. 2000-207564), the user of the imaging apparatus or the manufacturer of the imaging apparatus using a classifier such as neural networks, deep learning, and trees in advance as represented by face recognition etc. The imaging device manufacturer or user can obtain the main subject by calculating the likelihood of the main subject likeness of the image of the main subject region by learning in advance the multidimensional feature quantities extracted from the image of the main subject candidate object. Various methods have already been proposed, such as a method of flexibly recognizing an arbitrary subject considered as a main subject.

  Any method may be used as long as it can detect the presence or absence, the position, and the area of the main subject in the image. The focusing processing unit 107 detects the main subject in the image, and outputs the information on the position and area of the main subject to the RAM 103 for storage.

  The posture change acquisition unit 108 includes a position and posture sensor such as a gyro, an acceleration sensor, and an electronic compass, and measures the position and posture change of the imaging apparatus. The attitude sensor is attached to an arbitrary axis orthogonal to the optical axis of the optical system. When the attitude sensor is a rotation sensor, it is attached to each axis of the imaging device in the yaw direction, pitch direction, and roll direction, measures attitude change due to rotation around each axis, outputs attitude change data to the RAM 103, and stores Let The posture change acquisition unit 108 has an integral processing unit and not only rates such as sampling timing (for example, 1 KHz) of a gyro that is a posture change information sensor but also samplings such as 60 fps and 120 fps that are frame rates of photographed images. At the same time, posture change information at different rates may be output at the same time.

  The image processing unit 109 uses the captured image stored in the RAM 103 to calculate the image magnification variation, the rotation variation amount, the image magnification, the electronic correction for the rotation component variation amount, and the motion vector using the corrected image. The detection process is performed.

  The recording medium 110 is a detachable memory card or the like, and an image A / D converted by the A / D converter 106 stored in the RAM 103 or an image processed by the image processor 109 is recorded as a recorded image. .

  FIG. 2 is a diagram for explaining the configuration of the image processing unit 109 shown in FIG.

  The image processing unit 109 includes an image input unit 201, a main subject information acquisition unit 202, an image polar coordinate conversion processing unit 203, a vector detection processing unit 204, a main subject image scale / rotation information acquisition unit 205, and a scale / rotation. It comprises an image correction processing unit 206 and an imaging device scale / rotation information acquisition unit 207.

  Of the captured images stored in the RAM 103, an image used for vector detection is input to the image input unit 201, and the captured image is temporarily stored and held.

  The main subject information acquisition unit 202 acquires main subject position / region information output by the focusing processing unit 107 corresponding to the time when the captured image input to the image input unit 201 is captured.

  The image polar coordinate conversion processing unit 203 performs polar coordinate conversion processing on the photographed image input to the image input unit 201 based on the main subject position / area information acquired by the main object information acquisition unit 202. The captured image that has been subjected to polar coordinate conversion processing is temporarily stored and held.

  The vector detection processing unit 204 performs vector detection processing using an image input from the image polar coordinate conversion processing unit 203 or the scale / rotational image correction processing unit. The detected vector information is stored in the RAM 103.

  The main subject image scale / rotation information acquisition unit 205 acquires the scale / rotation information of the main subject image using the vector information detected by the vector detection processing unit 204 using the image input from the image polar coordinate conversion processing unit 203. .

  The scale / rotation image correction processing unit 206 is input from the image input unit 201 using the scale / rotation information acquired from the main subject image scale / rotation information acquisition unit 205 or the imaging device scale / rotation information acquisition unit 207. Image correction is performed on the image. The corrected image is output to the vector detection processing unit 204.

  The imaging device scale / rotation information acquisition unit 207 acquires the scale / rotation information of the background image using the posture change information of the imaging device acquired by the posture change acquisition unit 108 and the optical magnification information acquired by the optical system 104. To do.

  The control of the image processing unit 109 configured as described above will be described in detail using the flowchart shown in FIG.

  In step 301, the captured image is input to the image processing unit via the image input unit 201 and temporarily stored.

  In step 302, the control sequence is branched depending on whether or not vector detection is performed on the main subject. If the main subject is to be processed, the process proceeds to step 303. If not, that is, if vector detection is performed on the background, the process proceeds to step 307. In this embodiment, the flow of steps 303 to 306 will be described first.

  In step S 303, the main subject information acquisition unit 202 acquires main subject position / area information output by the focusing processing unit 107.

  In step 304, the image polar coordinate conversion processing unit 203 performs polar coordinate conversion processing on the captured image stored in step 301 based on the main subject position / region information acquired in step 303.

  In this embodiment, the captured image is mapped to the logarithmic polar coordinate space based on the main subject position / region information. The method of image polar coordinate conversion processing will be described in detail with reference to FIG.

  FIG. 4 is a schematic view illustrating a method of mapping an image into a polar coordinate space. FIG. 4A is a schematic view showing XY plane coordinates of an image before mapping, and FIG. 4B is a schematic view showing an image after mapping to polar coordinates. The image mapped to the polar coordinates in FIG. 4B is a pixel image represented by a radial radius 、 in the horizontal direction and a deflection angle φ in the vertical direction. The pixel value of the pixel filled in gray in FIG. 4B is obtained by using the pixel value at the position on the XY plane coordinate in FIG. 4A shown by the upper arrow corresponding to the polar coordinate value. It is obtained by inverse interpolation. Here, reverse interpolation is used, but there may be a method of interpolating pixels that do not contain values using a filter or the like after forward interpolation. In the present embodiment, mapping is performed in a logarithmic polar coordinate space, and the conversion equation is expressed by the following.

In Equation 1, x and y are xy coordinate values of the image before mapping, and x _centor and y _centor are set transformation center coordinate values of the pre-mapping coordinates. As shown in Equation 2, the moving radius ρ is calculated by multiplying the xy coordinate of Equation 1 by logarithmic polar coordinate conversion by a coefficient M. The declination φ is calculated by multiplying the angle (0 to 360 °) of the xy coordinate I by a coefficient K as shown in Equation 3. The coefficient M is expressed by Equation 4, w is the horizontal size of the pre-mapping image, and R is the set maximum moving radius. The coefficient K is expressed by Equation 5, and h is the vertical size of the pre-mapping image. x _center and y _center are set according to the xy coordinate position on the pre-mapping image of the main subject acquired in step 303. R is set to an arbitrary n times the maximum moving radius obtained from the diagonal of the pre-mapping image, or R is the maximum obtained from the size of the main subject calculated from the main subject area information acquired in step 303. Arbitrary n times the moving radius may be set. This is to make the resolution of the main subject center to be high because the resolution of the conversion central portion is high and the peripheral portion has low characteristics in the logarithmic polar coordinate conversion. The converted image is temporarily stored.

  Here, with reference to FIG. 5, an example of log polar coordinate conversion of an image will be described. 5 (a) and 5 (c) are images before mapping, FIG. 5 (b) is an image obtained by logarithmic polar transformation of FIG. 5 (a), and FIG. 5 (d) is a logarithm of FIG. 5 (c). It is the image which carried out polar coordinate conversion. Further, in this example, logarithmic polar coordinate conversion is performed using the central coordinates of the image at the conversion center coordinates of Expression 1. Therefore, in both FIGS. 5B and 5D, the pixel area obtained by converting the vicinity of the image center coordinates before mapping has the highest resolution, and the resolution decreases as the converted pixel area moves away from the image center coordinates. ing.

In step 305, the vector detection processing unit 204 performs a template matching process using the images obtained by the logarithmic polar transformation obtained in step 304. Here, a general method is used for template matching. In the present embodiment, matching processing is performed using a plurality of template images obtained by dividing an image, and a vector value of each matching result is processed by a statistical value such as a histogram to thereby translate a translation vector value (V Calculate _x , V _y ).

  Here, template matching between log polar transformed images will be described with reference to FIG. The image in FIG. 5C is an image that is rotated and enlarged in the clockwise direction on the sheet surface with the image center coordinates as the center with respect to the entire image in FIG. When the images after the mapping of FIG. 5B and FIG. 5D are compared, it is found that the image of FIG. 5D is shifted in the lower right direction with respect to the image of FIG. 5B. Recognize. This is because the amount of change when the image before mapping in FIG. 5C is rotated appears downward in the drawing, and the amount of change in the scaled scale appears in the right of the drawing as the change in the shift component. . As described above, by detecting the vector translation component of the image after log-polar coordinate conversion by template matching, it is possible to convert the rotation and scale components of the image before mapping and obtain it.

In step 306, using the vector value calculated in step 305, the main subject image scale / rotation information acquisition unit 205 detects the scale of the main subject image and the rotational fluctuation component. A conversion formula of the vector values (V _x , V _y ) into the scale fluctuation component r and the rotation fluctuation component θ (°) is expressed as follows.

  Here, how the image after logarithmic polar coordinate conversion changes according to the position of the conversion center coordinate of Expression 1 will be described using FIG. FIGS. 6A and 6D are images before mapping, and a dotted frame represents the position of the main subject acquired by the main subject information acquisition unit 202. When the images in FIGS. 6A and 6D are compared, the background is not moved with respect to the photographing device, and only the main subject is moving, and the image of FIG. In contrast to (a), the main subject is rotated and enlarged in the clockwise direction on the paper. FIGS. 6 (b) and 6 (e) are images obtained by subjecting the images in FIGS. 6 (a) and 6 (d) to log polar coordinate conversion using the image center coordinates as conversion center coordinates. FIGS. 6C and 6F are images obtained by performing log-polar conversion on the images in FIGS. 6A and 6D, respectively, using the central coordinates of the main subject at the center of the dotted line frame as the conversion central coordinates. It is. When the converted images in FIG. 6 (b) and (e) are compared, no shift due to rotation or scale change is observed as the entire image. On the other hand, comparing the images after conversion of FIGS. 6C and 6F with each other, the image of FIG. 6F is shifted in the lower right direction with respect to the image of FIG. 6C. I understand. This is because the image after polar coordinate conversion has a high resolution in the area near the conversion center coordinates and low characteristics in the peripheral area, so when the polar conversion center coordinates are the center coordinates of the main subject, the rotation and scale change of the main subject It can be seen that the component is detectable.

  Next, the flow of steps 307 to 308 will be described.

  In step 307, the imaging apparatus scale / rotation information acquisition unit 207 uses the optical magnification information output from the optical system 104 and the posture change information output from the posture change acquisition unit 108 to perform scale fluctuation components r between images for which vector detection is performed. The rotation fluctuation component θ (°) is calculated. Here, the fluctuation component is acquired at the frame rate of vector detection.

  Next, step 308 in which the two flows branched in step 302 merge will be described.

  In step 308, electronic correction is performed on the image used for vector detection acquired in step 302, using the scale fluctuation component r between images for performing vector detection acquired in step 306 or step 307 and the rotational fluctuation component θ (°). Process. The electronic correction process may be performed on either of the images because the scale and rotation components between the template images to be subjected to template matching need only be aligned. The electronic correction process performed here may use a general image conversion process. For example, the scaling process and the in-plane rotation are performed so that the scale fluctuation component r and the in-plane rotation fluctuation component θ (°) are aligned by Helmart deformation or the like. Apply processing.

  In step 309, template matching processing is performed using the image subjected to the electronic correction processing in step 308. Here, in template matching, matching processing is performed using a plurality of divided template images as in step 305. The division of the template may make the entire image uniform. In addition, when aiming at vector detection of the main subject as in the process of the flow of steps 303 to 306, a large number of templates of the image position of the main subject are arranged, and the process of the flow of steps 307 to 308 When aiming at a background vector as in the case of time, many templates may be arranged at the image position of the background.

  Here, the electronic correction processing when detecting the main subject vector will be described with reference to FIG. The images in FIG. 7A and FIG. 7B are images before electronic correction processing in which template matching is performed. In the image of FIG. 7B, the scale of the main subject only varies with respect to the image of FIG. The image in FIG. 8C is an image obtained by electronically correcting the scale variation of the main subject so as to align with the image in FIG. 7B with respect to the image in FIG. Thus, it can be seen that the images of FIG. 7B and FIG. 7C have the same magnification of the main subject. Therefore, it is possible to improve the vector detection accuracy of the main subject by performing template matching using the images of FIGS. 7B and 7C.

  Next, electronic correction processing when detecting a background vector will be described with reference to FIG. The images in FIGS. 8A and 8B are images before electronic correction processing for template matching. In the image of FIG. 8 (b), rotational fluctuation occurs in the background with respect to the image of FIG. 8 (a). The image of FIG. 8C is an image obtained by electronically correcting the background rotation variation with respect to the image of FIG. 8B with respect to the image of FIG. Thus, it can be seen that the images in FIGS. 8B and 8C have the same background inclination. Therefore, by performing template matching using the images of FIG. 8B and FIG. 8C, it is possible to improve the background vector detection accuracy.

  In step 310, error processing of the vector detected in step 309 is performed. To explain with the example of FIG. 7, when comparing the images of FIG. 7B and FIG. 7C, the scale of the main subject is the same, but the scale of the background is largely shifted. In such a case, in the flow of detecting the vector of the main subject, the vector component of the background is likely not to be detected correctly. Therefore, processing for eliminating such vectors as errors is performed. For example, processing may be performed in which a detected vector of a region not intended to obtain a vector is regarded as an error using the position / region information of the main subject in step 303. Alternatively, histogram processing may be performed on the detected vector, and a vector having a value that is not smaller than a certain threshold from the peak of the histogram may be regarded as an error. Alternatively, an error vector may be used if the template matching similarity is below a certain threshold.

  As described above, the control of the image processing unit 109 has been described in detail using the flowchart shown in FIG. Up to this point, the operation of the present proposal has been described. By performing the above processing, even if there is a change in magnification or rotation component of the image due to movement of the subject, it is possible to detect good motion vectors of the main subject and the background.

Second Embodiment
The image display device according to the first embodiment acquires the scale and rotation variation component of the main subject by an image processing method using template matching of the polar coordinate conversion image of the image, and the background scale and rotation variation component are obtained from the imaging device. Good motion vector detection is implemented by performing electronic correction processing of a template matching detection image using information obtained by using optical system control information and position and orientation sensor information. The image display apparatus according to the second embodiment uses the template matching of the polar-transformed image of the image as in the case of the main subject without using the above-described information acquired from the imaging apparatus for the background scale / rotation fluctuation component. An example of acquisition using a conventional image processing method will be described.

FIG. 9 is a view for explaining the configuration of the image processing unit 109 of FIG. 1 in the image display device according to the second embodiment. The image processing unit 109 includes an image input unit 201, a main subject information acquisition unit 202, an image polar coordinate conversion processing unit 203, a vector detection processing unit 204, a main subject image scale / rotation information acquisition unit 205, and a scale / rotation. It comprises an image correction processing unit 206, a background image scale / rotation information acquisition unit 907, and a background area selection processing unit 908.
The image input unit 201 and the main subject information acquisition unit 202 are the same as the description of FIG.

The background area selection processing unit 908 uses the main subject position / area information acquired by the main subject information acquisition unit 202 to select position / area information used for scale / rotation detection of the background image.
The image polar coordinate conversion processing unit 203 uses the captured image input to the image input unit 201 as the main subject position / region information acquired by the main subject information acquisition unit 202 or the position of the background region selected by the background region selection processing unit 908. -Polar coordinate conversion processing is performed based on the region. The photographed image subjected to the polar coordinate conversion process is temporarily stored and held.

  The vector detection processing unit 204, the main subject image scale / rotation information acquisition unit 205, and the scale / rotation image correction processing unit 206 are the same as those described with reference to FIG.

  The background image scale / rotation information acquisition unit 907 acquires the scale / rotation information of the background image using the vector information detected by the vector detection processing unit 204 using the image input from the image polar coordinate conversion processing unit 203.

  Control of the image processing unit 109 configured as described above will be described in detail using the flowchart shown in FIG.

  Step 301 is the same as that described with reference to FIG.

  In step 1002, the main subject information acquisition unit 202 acquires the main subject position / area information output by the focusing processing unit 107.

  At step 1003, the control sequence is branched depending on whether or not vector detection is performed on the main subject. If the main subject is to be processed, the process proceeds to step 304. If not, that is, if vector detection is performed on the background, the process proceeds to step 1004. The flow of steps 304 to 306 is the same as the explanation of FIG.

  In step 1004, the background area selection processing unit 908 selects a position / area used for scale / rotation detection of the background image using the main subject position / area information acquired in step 1002. Here, a background image position / region selection method will be described with reference to FIG. FIG. 11 is a schematic view of the acquired image, and the area surrounded by the dotted line 1100 is the main subject position / area. In this example, the area surrounded by dotted line 1101 is selected as the image area including the background texture as much as possible among the areas excluding the main subject position / area of dotted line 1100 as the position / area used for the scale / rotation detection of the background image. doing. The image position / area to be selected is preferably an image area excluding the main subject position / area and an image area including as much of the background texture as possible, and by doing so for scale / rotation detection of the background image described later. Template matching can be performed accurately. Specifically, in the background position / region selection method, a differential filter is used to select a position / region having a relatively large distribution of pixels with a large amount of change in luminance in the acquired image, excluding the main subject position / region. To do. Although a differential filter is used in the present embodiment, other general feature point detection methods may be used to select positions / regions in which a relatively large number of pixels with large feature amounts are distributed.

  In step 1005, the image polar coordinate conversion processing unit 203 performs polar coordinate conversion processing on the captured image stored in step 301 based on the position / region of the background image acquired in step 1004. The process here is the same as the process of step 304 in FIG. 10 except that the main subject position / area information is replaced with the position / area of the background image, and the other processes are the same.

  In step 1006, the vector detection processing unit 204 performs template matching processing using the log-polar coordinate transformed images acquired in step 1005. Since the process here is the same as that of step 305 of FIG. 10, the detailed description is omitted.

  In step 1007, using the vector value calculated in step 1006, the background image scale / rotation information acquisition unit 907 detects the scale and rotation variation component of the main subject image. Since the process here is the same as that of step 306 of FIG. 10, the detailed description is omitted.

  The flow of steps 308 to 310 is the same as the explanation of FIG.

  As described above, the control of the image processing unit 109 has been described in detail using the flowchart shown in FIG. Up to this point, the operation of the present proposal has been described. By performing the above processing, even when control information of the optical system of the imaging apparatus or position and orientation sensor information is not used, the main subject, the background, and the variation of the image magnification or rotation component due to the movement of the subject It is possible to detect each good motion vector.

(Other embodiments)
The image display apparatus according to the first embodiment is an image display according to the second embodiment in which the background scale and the rotational fluctuation component use control information of the optical system of the imaging apparatus and position and orientation sensor information. The apparatus has described an example in the case where the background scale / rotation fluctuation component does not use the above information acquired from the imaging apparatus. In addition to the above embodiment, the scale of the background or the rotational fluctuation component may be acquired using information of only one of control information of the optical system of the imaging apparatus or position and orientation sensor information. Further, as in the second embodiment, by using image processing, the acquired background scale and rotation fluctuation component, and both the control information of the optical system of the imaging apparatus and the position / orientation sensor information, or the acquisition information of only one of them is used. Thus, the background scale and rotation fluctuation component may be obtained by calculating the scale and rotation fluctuation component obtained from them by weighted addition.

  In the above description, the present invention is described as applied to an imaging apparatus such as a video camera, but this is not limited to this example. For example, the present invention may be applied to a personal computer equipped with a video camera, or may be applied to a mobile phone terminal capable of shooting a moving image. Furthermore, the present invention can be applied as a process for image data recorded on a recording medium in addition to being applied as a process for image data output in real time by photographing. Further, the processing according to the present invention may be performed by software on a personal computer on image data recorded on a hard disk of a personal computer or the like. As mentioned above, although preferable embodiment of this invention was described, this invention is not limited to these embodiment, A various deformation | transformation and change are possible within the range of the summary.

100 digital camera, 101 control unit, 102 ROM, 103 RAM,
104 optical system, 105 imaging unit, 106 A / D conversion unit, 107 focusing processing unit,
108 posture change acquisition unit, 109 image processing unit, 110 storage medium,
201 image input unit 202 main subject information acquisition unit 203 image polar coordinate conversion processing unit
204 vector detection processing unit, 205 main subject image scale / rotation information acquisition unit,
206 scale and rotation image correction processing unit,
207 Imager scale / rotation information acquisition unit,
907 background image scale / rotation information acquisition unit,
908 Background area selection processing unit

Claims (9)

Main subject position information acquisition means for images;
A first magnification for calculating the magnification of the image and the amount of change of the image by polar coordinate conversion of the image of the main object position based on the main object position information;
A second magnification for calculating an image magnification and a rotation change amount by a method different from the calculated method, a rotation change amount calculating means;
Image correction means for correcting image magnification and rotation;
Vector detection means for performing motion vector detection using the corrected image;
Have
An image correction is performed based on the first magnification, the magnification calculated by the rotation change amount calculation means, and the rotation change amount, and the first motion vector is detected using the corrected image, thereby moving the motion vector of the main subject. To detect
The image is corrected based on the second magnification, the magnification calculated by the rotation change amount calculation means, and the rotation change amount, and the motion of the background subject is detected by performing the second motion vector detection using the corrected image. An image processing apparatus for detecting a vector. The image processing according to claim 1, wherein the main subject position information acquiring unit acquires an in-focus position, or a position and an area on an image surface of the image, which is obtained from in-focus information of the image. apparatus. The first magnification / rotational change calculation unit performs polar coordinate conversion of the image using the main object position on the image surface acquired by the main object position information acquisition unit as a conversion center coordinate value of polar coordinate conversion, and after polar coordinate conversion The image processing apparatus according to claim 1, wherein a translation vector is calculated by template matching between the two images, and a magnification and a rotation change amount of the image are calculated by inverse transformation of the calculated translation vector. The second magnification / rotational change calculating means is characterized by calculating the magnification / rotational change using the fluctuation of the optical magnification at the time of photographing the image and the detection information of the position and orientation sensor at the time of photographing the image. The image processing apparatus according to claim 1. The second magnification / rotational change calculation means converts the image position of the background area in the image, except for the main subject position on the image surface acquired by the main subject position information acquisition means, into a conversion center coordinate value of polar coordinate conversion. The image is subjected to polar coordinate conversion, the translation vector is calculated by template matching between the images after polar coordinate conversion, and the calculated translation vector is subjected to polar coordinate inverse conversion to calculate the magnification and rotation change amount of the image. An image processing apparatus according to Item 1. The second magnification and rotation change amount calculation means method according to claim 5 is an area on the image image plane acquired by the main subject position information acquisition means and a region where pixels having a high luminance gradient or high feature amount are distributed. 2. The image processing apparatus according to claim 1, further comprising background image selection means for selecting a position and an area on the image image plane excluding the main subject position.   The second magnification and rotation change amount calculation means are both the fluctuation amount of the optical magnification at the time of image shooting according to claim 4 and the magnification and the rotation change amount using the detection information of the position and orientation sensor at the time of image shooting. 6. The image processing apparatus according to claim 1, wherein one of the second magnification, the magnification calculated by the rotation change amount calculation means, and the rotation change amount are calculated by weighted addition. The vector detection means is a vector value detected at a position excluding the main subject position / region on the image image plane acquired by the main subject position information acquisition means with respect to the vector value calculated by the first motion vector detection. The image processing apparatus according to claim 1, further comprising: vector reliability determination processing that determines low reliability of the image processing apparatus. The vector detection means is a vector value detected at the position of the main subject position / region on the image image plane acquired by the main subject position information acquisition means with respect to the vector value calculated by the second motion vector detection. The image processing apparatus according to claim 1, further comprising vector reliability determination processing that determines the reliability to be low.