JP2020120362A - Image processing device, image processing method, and program - Google Patents

Abstract

To provide an image processing device capable of stabilizing moving images considering position variation of a moving body when generating stabilized moving images from picked-up moving images.SOLUTION: The image processing device includes: a trajectory estimation part that estimates an imaging trajectory from picked-up moving images; a trajectory correction part that corrects the estimated imaging trajectory; and an image generation part that generates stabilized moving images according to the imaging trajectory after correction from the picked-up moving images. The trajectory correction part detects a moving body area based on movement vector detected from the picked-up moving images, corrects the imaging trajectory to reduce the variation of the position of the moving body in each frame of the generated moving images calculated from the detected moving body area, and generates stabilized moving images from which the position variation of the moving body is reduced.SELECTED DRAWING: Figure 1

Description

The present invention relates to an image processing device, an image processing method, and a program.

As a technology to electronically stabilize the fluctuation of the shooting angle of view due to camera shake, etc., the shooting trajectory is estimated from the shooting video, and the estimated trajectory fluctuation of the shooting trajectory is corrected so as to be reduced and corrected based on the shooting video. A technique for generating a moving image corresponding to a subsequent image capturing locus has been proposed. For example, Patent Document 1 proposes a technique of reducing trajectory fluctuation by performing processing for thinning out or interpolating frames so that the amount of movement of a subject between frames of a captured moving image is constant. Patent Document 2 proposes a technique for reducing trajectory fluctuation by controlling the shooting interval so that the movement amount of a specific subject between frames becomes a desired movement amount. In addition, recently, by creating a movie with stabilized fluctuations in the shooting angle of view so that it plays at a speed N times faster than the shot video, a long movie shot with an action cam becomes a short digest movie. It is said that he enjoys it.

JP 2012-60258 A JP, 2011-66873, A

However, in the technique described in Patent Document 1, the movement of the entire screen due to the movement of the imaging device is taken into consideration in order to stabilize the moving image, but the positional fluctuation of the moving body is not taken into consideration. As a result, the motion of the moving body is not stabilized, and the moving body appears at irregular positions in each frame. In particular, when a short-digest movie is generated from a long-time movie shot by a normal digital camera, if the motion of the moving body is not stabilized, the frames are thinned out and the moving body does not move smoothly. Be regular and discontinuous. Therefore, even if the movement of the entire screen is stabilized, the moving object may flicker on the screen, resulting in an unsightly image.

In addition, although the technique described in Patent Document 2 considers the position variation of the moving body, it does not control the moving body position itself on the image, only by controlling the shooting interval. Therefore, the movement of the moving body cannot be stabilized except when the moving body moves past a desired position.

The present invention has been made in view of such circumstances, and when generating a stabilized N-times speed moving image from a captured moving image, it is possible to stabilize the moving image in consideration of the position variation of the moving body. An image processing apparatus is provided.

An image processing apparatus according to the present invention includes a trajectory estimation unit that estimates an imaging trajectory from a first moving image, a trajectory correction unit that corrects the estimated imaging trajectory, and a second trajectory that corresponds to the corrected imaging trajectory. An image generation unit that generates a moving image from the first moving image, the trajectory correction unit detects a moving body region based on a motion vector detected from the first moving image, and calculates from the detected moving body region. The imaging locus is corrected so as to reduce the fluctuation of the moving body position in each frame of the second moving image.

According to the present invention, when a stabilized N-times speed moving image is generated from a captured moving image, it is possible to generate a stabilized moving image in which unsightlyness due to position variation of a moving body is reduced.

It is a figure which shows the structural example of the image processing apparatus in 1st Embodiment. It is a figure which shows the structural example of the locus|trajectory estimation part in 1st Embodiment. It is a figure showing the example of composition of the locus amendment part in a 1st embodiment. 6 is a flowchart illustrating an operation example of the image processing apparatus according to the first exemplary embodiment. 6 is a flowchart showing an example of trajectory estimation processing in the first embodiment. It is a figure explaining the matching process in 1st Embodiment. It is a figure explaining the estimation method of the camera position and orientation in a 1st embodiment. FIG. 6 is a diagram illustrating an example of a variation of an image pickup trajectory in the first embodiment. 6 is a flowchart showing an example of trajectory correction processing in the first embodiment. It is a figure explaining the detection method of the moving body area|region in 1st Embodiment. It is a figure explaining the optimization of the imaging locus in 1st Embodiment. It is a figure explaining the optimization of the imaging locus in 1st Embodiment. It is a figure showing the example of composition of the locus amendment part in a 2nd embodiment. 8 is a flowchart showing an example of trajectory correction processing in the second embodiment. It is a figure explaining the weight control of the imaging locus correction in 2nd Embodiment. It is a figure which shows the hardware structural example of the image processing apparatus in this embodiment.

Embodiments of the present invention will be described below with reference to the drawings.

(First embodiment)
A first embodiment of the present invention will be described. FIG. 1 is a block diagram showing a configuration example of an image processing apparatus 100 according to the first embodiment. The image processing apparatus 100 includes an image input unit 101, an image memory 102, a trajectory estimation unit 103, a trajectory correction unit 104, and an image generation unit 105. The image input unit 101 receives an image (captured moving image) including a plurality of frames. The image memory 102 temporarily stores and holds the image input by the image input unit 101 for one frame or a plurality of frames.

The trajectory estimation unit 103 estimates the trajectory of the camera when a moving image is captured, using images of different frames from the image input by the image input unit 101 and the image stored and held in the image memory 102. .. FIG. 2 is a block diagram showing a configuration example of the trajectory estimation unit 103. As shown in FIG. 2, the trajectory estimation unit 103 includes an image matching unit 201, a variation amount calculation unit 202, and a variation amount accumulation unit 203.

The image matching unit 201 performs matching processing of the current frame (frame number n) and the next frame (frame number n+1). For example, the image of the current frame (frame number n) is input from the image memory 102, and the image of the next frame (frame number n+1) is input from the image input unit 101. The variation amount calculation unit 202 calculates the variation amount between images using the result of the matching process by the image matching unit 201. The variation amount accumulating unit 203 accumulates the variation amounts between the images calculated by the variation amount calculating unit 202 to calculate an imaging trajectory.

The locus correction unit 104 performs a correction process based on the imaging locus at the time of imaging estimated by the locus estimation unit 103 so as to reduce the variation amount between images, and calculates a stabilized imaging locus. FIG. 3 is a block diagram showing a configuration example of the trajectory correction unit 104. As shown in FIG. 3, the locus correction unit 104 includes a motion vector detection unit 301, a moving body region detection unit 302, a moving body position calculation unit 303, and a locus optimization unit 304.

The motion vector detection unit 301 detects a motion vector on a pixel-by-pixel basis for each frame. The moving body area detection unit 302 detects the moving body area for each frame using the motion vector detected by the motion vector detection unit 301. The moving body position calculation unit 303 calculates the coordinates indicating the moving body region as the moving body position based on the moving body region detected by the moving body region detection unit 302. The trajectory optimization unit 304 corrects the imaging trajectory estimated by the trajectory estimation unit 103 by using the moving body position for each frame calculated by the moving body position calculation unit 303.

The image generation unit 105 reads out an image corresponding to the angle-of-view area corresponding to the imaging trajectory corrected by the trajectory correction unit 104 from the captured moving image and generates a stabilized moving image at N times speed of the captured moving image.

The operation of the image processing apparatus 100 according to the first embodiment will be described. FIG. 4 is a flowchart showing an operation example of the image processing apparatus 100 according to the first embodiment.

In step S401, the captured image is input to the image processing apparatus 100 via the image input unit 101. The input image is stored and held in the image memory 102. Next, in step S402, the trajectory estimation unit 103 estimates an imaging trajectory when a captured moving image is captured using the image input in step S401 and the image stored and held in the image memory 102. The trajectory estimation process is performed.

In subsequent step S403, the trajectory correction unit 104 performs trajectory correction processing for correcting the imaging trajectory estimated in step S402. The trajectory correction unit 104 generates a corrected imaging trajectory by performing processing on the estimated imaging trajectory at the time of imaging so that the change for each frame becomes smooth. Next, in step S404, the image generation unit 105 generates, for each frame, a stabilized moving image by reading an image of the angle-of-view area corresponding to the image capturing trajectory corrected in step S403 from the captured moving image.

The trajectory estimation processing by the trajectory estimation unit 103 in step S402 of FIG. 4 will be described. FIG. 5 is a flowchart showing an example of the trajectory estimation process in the first embodiment.

In step S501, the image matching unit 201 of the trajectory estimation unit 103, as shown in an example in FIG. 6, the current frame (frame number n) in FIG. 6A and the next frame (frame number n+1) in FIG. 6B. ) And perform matching processing. An example of the result of the matching process is shown in FIG. Any matching method may be used as long as it can calculate corresponding points in two frames. For example, like SIFT (Scale Invariant Feature Transform), the feature points of the current frame and the next frame may be calculated, and the correspondence search of the calculated feature points may be performed. Further, for example, like a KLT (Kanade-Lucas-Tomasi feature tracker) tracker, feature points may be calculated in the current frame, and the feature points may be tracked in subsequent frames.

In FIG. 6A, the positions 601, 602 and 603 indicated by circles are the positions of the feature points in the current frame, and the positions 604, 605 and 606 shown in FIG. It is the position of the feature point in the next frame. As a result of the matching process, corresponding points among the positions of the respective characteristic points are shown by lines 607, 608, and 609 in FIG. 6C. By the matching processing, the correspondence between the respective feature points is calculated such that the position 601 of FIG. 6A is the position 604 of FIG. 6B. In FIG. 6, only the three corresponding points are displayed as the number of characteristic points, but the number of characteristic points is not limited to this. However, since the number of parameters of the basic matrix F to be calculated later is eight, it is preferable that there are eight or more feature points for accurate calculation. Also, as for the feature points, it is better to use only the feature amount and the points with high matching precision, because the calculation accuracy of the camera position and orientation afterwards is improved, so that the correlation value and the feature amount during matching are higher than a certain threshold. You may perform the process which uses only.

In step S502, the variation amount calculation unit 202 of the trajectory estimation unit 103 uses the result of the matching processing performed in step S501 to calculate the variation amount of the camera position and orientation between images. Here, a method of estimating the camera position and orientation using the Structure from Motion will be described with reference to FIG. 7. Corresponding points between the two frames are three-dimensional same points viewed from two viewpoints. Using FIG. 7, the camera centers of the two viewpoints are C1 and C2, and the same points are three-dimensional coordinate points X. expressed. The corresponding points at which the three-dimensional coordinate point X viewed from the camera centers C1 and C2 are projected on the frame are represented by x1 and x2.

Here, the camera matrices P1 and P2 at the camera centers C1 and C2 can be expressed as the following (Equation 1) and (Equation 2), respectively.

Here, the coordinate axes with the origin as the reference are aligned with the camera center C1, I is the unit matrix, and K is the internal calibration matrix. Further, R is a rotation conversion matrix representing the orientation of the camera at the position of the camera center C2, and T is a three-dimensional translation vector. In the present embodiment, since the cameras at the camera centers C1 and C2 are the same, the internal calibration matrix K is the same.

Here, as a result of the relative positional relations R and T of the cameras, the characteristics K of the cameras, and the positions of the three-dimensional points, geometric constraints are generated at the points x1 and x2 on the image. When used, the condition for projecting the three-dimensional coordinate point X on the points x1 and x2 on the image can be derived. In this case, it is necessary to satisfy the following equation (3).

(Equation 3) is called an epipolar constraint, and the matrix F in this epipolar constraint is called a basic matrix. This matrix F is an equation representing the amount of rotation and translation between the two cameras C1 and C2, and (Equation 3) is represented as an equation for converting the point x1 into the point x2. (Formula 4) is a formula in which the matrix F is represented by the relative positional relationships R and T of the cameras, the characteristics K of the cameras, and St. St is an alternating matrix represented by a three-dimensional translation vector T.

Therefore, this basic matrix F is obtained by solving (Equation 3) using the corresponding point group between the two frames obtained in step S501. As a method of obtaining the basic matrix F, there are an 8-point method using eight sets of corresponding points, a least square method using SVD (singular value decomposition), and the like. Then, after the basic matrix F is obtained, the camera matrix is restored using (Equation 4), whereby the rotation conversion matrix R and the translation vector T between the two frames can be obtained.

FIG. 8A shows the amounts of variation of the translation vector T among the calculated amounts of variation, which are arranged in time series (ordered by frame number). In FIG. 8A, the vertical axis represents the image variation amount, the horizontal axis represents the frame number, and the vector in each frame is indicated by the arrow 801. The image fluctuation amount actually has components in three directions of x, y, and z directions, but in FIG. 8A, for convenience of explanation, it is omitted and only one component is represented. ..

Next, in step S503, the variation amount accumulating unit 203 of the trajectory estimation unit 103 accumulates the variation amounts calculated in step S502 in the time direction to calculate an imaging trajectory. The fluctuation amount accumulating unit 203 calculates a polygonal line 802 that is the imaging trajectory of FIG. 8B by accumulating the arrow 801 that is the vector of FIG. 8A in the time direction.
This is the end of the description of the trajectory estimation process in step S402.

Next, the locus correction processing by the locus correction unit 104 in step S403 of FIG. 4 will be described. FIG. 9 is a flowchart showing an example of the trajectory correction process in the first embodiment.

In step S901, the motion vector detection unit 301 of the trajectory correction unit 104 detects a motion vector in pixels on a frame-by-frame basis. As a method of detecting a motion vector in pixel units, there is a gradient method, for example. In the gradient method, it is assumed that the amount of movement of an object between two consecutive frames is minute and the brightness of a point on the object does not change even after the movement. Based on this assumption, if the pixel value of the pixel (x, y) on the image at time t is I(x, y, t) and the pixel moves by (Δx, Δy) after a time Δt, (Equation 5) is established.

Taylor expansion of the right side of (Equation 5) yields (Equation 6).

Here, in (Equation 6), ε is a high-order term regarding Δx, Δy, and Δt. Ignore that ε is sufficiently small and divide both sides by Δt. Then, by taking the limit as Δt→0, the following (formula 7) is obtained.

Here, the two unknowns u and v cannot be uniquely determined only by (Equation 7). Therefore, for example, it is assumed that the motion vectors are equal in the local region near the pixel of interest, and u and v are obtained by arranging a plurality of constraint condition expressions of (Equation 7) obtained in the local region.

In addition to the gradient method, for example, a known template matching method may be used as the method of detecting the motion vector on a pixel-by-pixel basis. Further, the motion vector may be calculated for all points with all the pixels in the image as the pixel of interest, or may be calculated only for pixels at a constant interval.

Next, in step S902, the moving body area detection unit 302 of the trajectory correction unit 104 detects the moving body area for each frame using the pixel-based motion vector detected in step S901. A method of detecting a moving body area will be described with reference to FIG.

First, consider calculation of a motion vector between two consecutive frames 1001 and 1002. As shown in FIG. 10A, a stationary subject 1003 and a moving subject 1004 are shown in the frame 1001. As shown in FIG. 10B, in the next frame 1002, the still subject 1003 moves to 1005 by the movement of the camera, and the moving subject 1004 moves to 1006 by the movement of the camera and the movement of the moving subject.

FIG. 10C shows the motion vector between the frames 1101 and 1102. The motion vector includes a motion 1007 of the entire screen due to a change in the position and orientation of the camera and a motion motion 1008 added to the motion of the moving object. However, the motion vector information alone cannot distinguish the two. Therefore, in step S502 of FIG. 5, it is used that the change (variation amount) of the position and orientation of the camera is known. For example, from the rotation R of the camera position and orientation and the parallel movement T, as shown in FIG. 10D, the movement amount 1009 of each pixel between frames due to only the change of the position and orientation of the camera is obtained. FIG. 10E shows the result of removing the movement amount of each pixel shown in FIG. 10D from the motion vector shown in FIG. 10C. The stationary area is detected as a zero vector 1010, and the moving body area is detected as a non-zero vector 1011. FIG. 10F shows a region having a non-zero vector, and a set of this region corresponds to the moving body region 1012.

Actually, since the position and orientation of the camera and the motion vector of each pixel are calculated with an error, they may not be exactly zero even in the non-moving object region. However, if the position and orientation of the camera and the motion vector of each pixel can be calculated almost accurately, the non-moving object region shows a vector value close to zero. Therefore, an area having a larger vector value may be detected as a moving body area. Alternatively, a vectorization value may be binarized into a zero vector and a non-zero vector by using a binarization method such as a discriminant analysis method, and the non-zero vector may be detected as a moving body region.

Immediately before performing the process of step S901, the image is geometrically transformed so as to cancel the change in the position and orientation of the camera between frames using the rotation R and the parallel movement T of the camera position and orientation calculated in step S502. It may be done. Then, in the subsequent step S901, motion vector detection may be executed. In this case, since the motion vector detection is performed in a state where there is no change in the camera position and orientation between frames, only the moving body area has a non-zero vector value.

Subsequently, in step S903, the moving body position calculation unit 303 of the trajectory correction unit 104 calculates the coordinates indicating the moving body region as the moving body position based on the moving body region detected in step S902. The moving body position calculation unit 303 obtains a rectangle 1013 circumscribing the moving body region 1012 and calculates its center of gravity 1014 as the moving body position, for example, as shown in FIG. Since the moving body position may be the coordinates of a point indicating the moving body region, the moving body position calculation unit 303 may calculate, for example, the average value of the coordinates of the pixels forming the moving body region as the moving body position. The former method of obtaining the center of gravity of the circumscribed rectangle requires a small amount of calculation. On the other hand, the latter method of obtaining the average value of the coordinates of each pixel can accurately calculate the moving body position even for a moving body having a complicated shape.

In step S904, the moving body position calculation unit 303 calculates the magnitude of the moving amount of the moving body. Even if there is a moving body, it is considered that there is little need to stabilize it if the amount of movement is small. Therefore, it is preferable to adjust the degree of reduction in the positional fluctuation of the moving body according to the amount of movement of the moving body when correcting the imaging trajectory in step S905 later. As the motion amount of the moving body, for example, the absolute value of the motion vector at the moving body position calculated in step S903 may be used. Alternatively, it is also possible to obtain the average of the motion vectors of the respective pixels forming the moving body area detected in step S902 and use the absolute value thereof.

In step S905, the trajectory optimization unit 304 of the trajectory correction unit 104 corrects the imaging trajectory at the time of imaging estimated in step S402 using the moving body position calculated in step S903 to optimize the imaging trajectory. The equation for optimizing the imaging trajectory can be expressed by the following (Equation 8) to (Equation 13), for example.

As the reference position, an arbitrary coordinate on the image may be set as a point at which the moving body position is desired to be fixed. For example, the center of the image or the moving body position of the frame to be processed first is set as the reference position.

Here, with reference to FIG. 11, the effect of the correction of the imaging trajectory will be described. In the graphs shown in FIGS. 11A to 11D, the vertical axis represents the variation amount of the camera position and orientation in three dimensions, and the horizontal axis represents the frame number. A polygonal line 1101 shown in FIGS. 11A to 11D indicates an input imaging locus (imaging locus at the time of imaging).

(Equation 8) is an equation for adjusting the length of the corrected imaging trajectory. A polygonal line 1102 shown in FIG. 11B is a diagram in which the length of the corrected imaging locus is shorter than that of the polygonal line 1101.

(Equation 9) is an equation for adjusting so that the change in the corrected camera position and orientation for each frame becomes smooth. A curve 1103 shown in FIG. 11C is a diagram in which the polygonal line 1102 is adjusted so that the change in the corrected image pickup trajectory for each frame becomes smooth.

(Formula 10) is a formula for adjusting the difference between the input camera position and orientation and the corrected camera position and orientation. A curve 1104 illustrated in FIG. 11D is a diagram in which the curve 1103 is adjusted so that the difference from the input imaging trajectory is reduced.

(Formula 11) is a formula for adjusting the difference between the moving body position and the reference position in the output image, and details will be described later. (Equation 12) is an equation for adjusting the imaging trajectory obtained by converting (Equation 11).

(Equation 13) is obtained by weighting and adding each of the terms of (Equation 8) to (Equation 10) and (Equation 12) with the parameters λ1 to λ4. Let it be an expression. Then, the corrected imaging trajectory can be calculated by solving the nonlinear minimization problem for p(t) and f(t) so as to minimize the value E shown in (Equation 13).

(Equation 11) and (Equation 12) will be described with reference to FIG. (Equation 11) is an equation that minimizes the difference D between the moving object position obj(t) on the two-dimensional image and the reference position c. (Equation 12) is an equation obtained by converting (Equation 11) into a three-dimensional representation of the camera position and orientation. (Equation 12) is such that the camera position/posture in the three-dimensional space such that the difference D on the two-dimensional image is 0 is pc(t), and the camera position/posture p(t) approaches pc(t). Correction.

FIG. 12A shows a scene and a camera position in the three-dimensional space. In FIG. 12 (a), 1201 denotes a camera position and orientation p _in the input. By capturing the moving body coordinates 1202 in the three-dimensional space from each camera position and orientation, the moving body position obj(t) 1203 on the two-dimensional image can be obtained.

The corrected camera position/orientation p(t) 1201 is set as 1204 by correcting the input camera position/orientation 1201 without considering the moving object position obj(t). The output angle of view from the camera position/orientation 1204 is set to 1205, and the state is shown in FIG. Here, the center of the output angle of view 1205 is set as the reference position c1206.

Since the camera position/orientation p(t) does not consider the moving object position obj(t), the deviation between the moving object position 1203 and the reference position 1206 is large at the output angle of view 1205 of the camera position/orientation p(t). As described above, when the input camera position/orientation is corrected without considering the moving object position, the moving object position 1203 exists regardless of the reference position 1206 in the corrected output angle of view of the camera position/orientation. As a result, since the moving body position changes for each frame, a moving image in which the moving body position is not stable is generated.

The corrected camera position/posture p(t) is set to 1207 by correcting the input camera position/posture 1201 in consideration of the moving object position obj(t) based on (Equation 12). The state of output from the camera position/orientation 1207 is set to 1208, and the state is shown in FIG.

Since the camera position/posture p(t) considers the moving object position obj(t), the moving object position 1203 and the reference position 1206 substantially overlap with each other in the output angle of view 1208 of the camera position/posture p(t). In this way, when the input camera position/orientation is corrected in consideration of the moving object position, the moving object position 1203 comes closer to the reference position 1206 at the corrected output angle of view of the camera position/orientation. As a result, since the moving body position is fixed near the reference position for each frame, a moving image with a stable moving body position is generated.

Here, in (Equation 13), the weight of each term can be changed by the parameter λ. These weightings can be specified arbitrarily. For example, if the value of the parameter λ2 is increased and the value of the parameter λ3 is decreased, even if the imaging locus is far from the input imaging locus as shown by the curve 1103 in FIG. Smoothing is a priority. Further, it is preferable to further reduce the position variation of the moving body as the moving amount of the moving body calculated in step S904 increases. Therefore, it is better to increase the value of the parameter λ4 as the moving amount of the moving body increases.

In step S906, the trajectory correction unit 104 determines whether the processing frame is the final frame. When the trajectory correction unit 104 determines that the processing frame is the final frame (YES), the trajectory correction processing ends, and the process returns to the flowchart shown in FIG. On the other hand, when the trajectory correction unit 104 determines that the processing frame is not the final frame, the process proceeds to step S901.

As described above, according to the first embodiment, the estimated imaging trajectory at the time of imaging is corrected so that the moving body position is fixed near the reference position for each frame in consideration of the moving body position. Then, by reading the image corresponding to the angle-of-view area corresponding to the corrected imaging trajectory from the captured moving image, it is possible to generate a moving image in which the variation in the captured angle of view is stabilized. Therefore, it is possible to generate a stabilized moving image in which the fluctuation of the moving body position is reduced.

(Second embodiment)
Next, a second embodiment of the present invention will be described. The image processing apparatus and the trajectory estimation unit included in the image processing apparatus according to the second embodiment are the same as the image processing apparatus 100 and the trajectory estimation unit 103 according to the first embodiment, and description thereof will be omitted. FIG. 13 is a block diagram showing a configuration example of the trajectory correction unit 104 in the second embodiment. 13, constituent elements having the same functions as those of the constituent elements shown in FIG. 3 are designated by the same reference numerals, and redundant description will be omitted.

The trajectory correction unit 104 according to the second embodiment includes a motion vector size detection unit 301, a moving body region detection unit 302, a moving body position calculation unit 303, and a trajectory optimization unit 304, as well as a moving body size calculation unit 305 and a distance information acquisition unit 306. Have. Although FIG. 13 shows the configuration including the moving body size calculation unit 305 and the distance information acquisition unit 306, it may be configured to include the moving body size calculation unit 305 or the distance information acquisition unit 306.

The second embodiment is different from the first embodiment in that at least one of the size of the moving body and the distance information is further taken into consideration when correcting the imaging trajectory. In the first embodiment, the imaging trajectory is corrected so that the variation in the moving body position between frames is reduced. As a result, in a scene in which the motion of the moving body is noticed, a moving image that is stabilized by reducing the variation of the moving body can be obtained.

Here, as a scene in which the movement of a moving body is noticed, for example, a scene in which a person is walking while chasing from behind, or a scene in which one's face is photographed while walking can be considered. In these scenes, since the size of the moving body is large, or the distance of the moving body is short and the amount of translational movement is relatively large, the movement of the moving body is easily noticed. However, in a scene that does not apply to this, even if the moving amount of the moving body is large, the movement of the moving body is hardly noticed, and it is not necessary to reduce the fluctuation of the moving body. Therefore, in the present embodiment, the degree of attention of the moving body in the scene is determined using at least one of the size of the moving body and the distance information, and the result is reflected in the optimization of the imaging trajectory.

The locus correction processing by the locus correction unit 104 according to the second embodiment will be described. FIG. 14 is a flowchart showing an example of the trajectory correction process in the second embodiment. The processing in steps S1401 to S1404 is the same as the processing in steps S901 to S904 shown in FIG. 9, and thus the description thereof will be omitted.

In step S1405, the moving body size calculation unit 305 of the trajectory correction unit 104 calculates the size of the moving body region detected in step S1402. As the size of the moving body region, for example, the number of pixels forming the moving body region or the area of a rectangle circumscribing the moving body region may be used.

In step S1406, the distance information acquisition unit 306 of the trajectory correction unit 104 acquires the distance (moving object distance) between the camera and the subject at the time of image capturing. For example, the distance measurement information that can be calculated by the autofocus control of the phase difference detection method is used as the moving body distance. Since the area in focus is the area the photographer is paying attention to, if the distance between the areas is short and there is a moving body, it can be determined that the movement of the moving body is of interest.

Here, the order of performing the processes of steps S1403 to S1406 is arbitrary and may be replaced.

In step S1407, the imaging trajectory optimization unit 304 of the trajectory correction unit 104 corrects the imaging trajectory at the time of imaging estimated in step S402 to optimize the imaging trajectory. In the second embodiment, the imaging locus is corrected using at least one of the moving body position calculated in step S1403, the size of the moving body calculated in step S1405, and the distance information acquired in step S1406. The equation for optimizing the imaging trajectory is the same as (Equation 8) to (Equation 13) shown in the first embodiment. Here, the weight λ4 of (Equation 13) is increased in a scene in which the motion of the moving body is noticed, and control is performed so that fluctuations in the moving body position are reduced.

As described above, in a scene where the movement of the moving body is noticed, it is possible that the size of the moving body is large or the distance of the moving body is short. Considering this, FIG. 15 shows an example of control of the weight λ4. In FIG. 15A, the weight λ4 is increased as the size of the moving body increases. Further, in FIG. 15B, the weight λ4 is reduced as the distance of the moving body increases. Although FIG. 15 shows an example in which the weight is changed linearly, the weight may be changed non-linearly.

The weights λ4 determined by the size of the moving body and the distance information are λ4a and λ4b, respectively. The final weight λ4 is obtained by combining the weight λ4a and the weight λ4b. As a method of combining, for example, both may be added. It is preferable to use both the size of the moving body and the distance information because it is possible to accurately determine the scene in which the movement of the moving body is noted, but it is also possible to use either one of the information. In that case, either the weight λ4a or the weight λ4b may be adopted as the final weight λ4.

According to the second embodiment, similarly to the first embodiment, it is possible to generate a stabilized moving image with reduced fluctuation of the moving body position. Further, by optimizing the imaging trajectory by using the weight λ4 determined according to at least one of the size and distance information of the moving object, a stabilized moving image in which fluctuation of the moving object is appropriately reduced according to the scene is generated. It will be possible to generate.

(Other Embodiments of the Present Invention)
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program. It can also be realized by the processing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

FIG. 16 is a block diagram showing an example of the hardware configuration of the image processing apparatus according to this embodiment. As shown in FIG. 16, the image processing apparatus according to this embodiment has a CPU 1601, a RAM 1602, a ROM 1603, an input unit 1604, an output unit 1605, a storage unit 1606, and a communication interface (IF) 1607. The CPU 1601, RAM 1602, ROM 1603, input unit 1604, output unit 1605, storage unit 1606, and communication IF 1607 are communicatively connected via a system bus 1608.

A CPU (Central Processing Unit) 1601 controls each unit connected to the system bus 1608. A RAM (Random Access Memory) 1602 is used as a main storage device of the CPU 1601. A ROM (Read Only Memory) 1603 stores a boot program of the apparatus. The functions of, for example, the trajectory estimation unit 103, the trajectory correction unit 104, and the image generation unit 105 are realized by the CPU 1601 reading the program from the storage unit 1606 and executing the program.

The input unit 1604 receives input from the user and inputs image data. The output unit 1605 outputs image data, processing results in the CPU 1601, and the like. The storage unit 1606 is a non-volatile storage device that stores a control program related to the operation and processing of the device. The communication IF 1607 controls information communication between this device and another device (such as a repeater).

In the device configured as described above, when the device is powered on, the CPU 1601 reads the control program and the like from the storage unit 1606 into the RAM 1602 according to the startup program stored in the ROM 1603. The CPU 1601 realizes the function of the image processing apparatus by executing processing according to the control program read into the RAM 1602. That is, the CPU 1601 of the image processing apparatus executes processing based on a control program or the like, whereby the functional configuration and operation of the image processing apparatus are realized.

It should be noted that each of the above-described embodiments is merely an example of the embodiment in carrying out the present invention, and the technical scope of the present invention should not be limitedly interpreted by these. That is, the present invention can be implemented in various forms without departing from the technical idea or the main features thereof.

100: image processing device 101: image input unit 102: image memory 103: trajectory estimation unit 104: trajectory correction unit 105: image generation unit 201: image matching unit 202: variation amount calculation unit 203: variation amount accumulation unit 301: motion vector Detection unit 302: Moving body region detection unit 303: Moving body position calculation unit 304: Trajectory optimization unit 305: Moving body size calculation unit 306: Distance information acquisition unit

Claims (9)

Trajectory estimation means for estimating an imaging trajectory from the first moving image,
Trajectory correction means for correcting the estimated imaging trajectory,
Image generating means for generating a second moving image corresponding to the corrected image pickup trajectory from the first moving image,
The locus correction means,
The imaging locus is detected so as to reduce the variation of the moving body position in each frame of the second moving image, which is detected from the moving body area based on the motion vector detected from the first moving image, and is calculated from the detected moving body area. An image processing apparatus which corrects The image processing apparatus according to claim 1, wherein the trajectory correction unit detects the moving body region based on a motion vector detected from the first moving image and the estimated imaging trajectory. The trajectory correcting means detects the moving body region based on a motion vector obtained by removing a variation amount calculated based on the imaging trajectory from a motion vector detected from the first moving image. Item 2. The image processing device according to item 2. The image processing apparatus according to claim 1, wherein the trajectory correction unit further corrects the imaging trajectory according to at least one of the size of the moving body region and the distance to the moving body. .. The image processing apparatus according to claim 1, wherein the trajectory correction unit calculates a center of gravity of a rectangle circumscribing the detected moving body region as the moving body position. The image processing apparatus according to any one of claims 1 to 4, wherein the trajectory correction unit calculates an average value of the coordinates of the detected pixels forming the moving body region as the moving body position. The locus correction means corrects the imaging locus by adjusting a camera position and orientation in a three-dimensional space so that the moving body position approaches a reference position on a two-dimensional image. 6. The image processing device according to any one of 6. A trajectory estimation step of estimating an imaging trajectory from the first moving image,
A trajectory correction step of correcting the estimated imaging trajectory,
An image generating step of generating a second moving image corresponding to the corrected image pickup trajectory from the first moving image,
In the trajectory correction step,
The imaging locus is detected so as to reduce the variation of the moving body position in each frame of the second moving image, which is detected from the moving body area based on the motion vector detected from the first moving image, and is calculated from the detected moving body area. An image processing method characterized by: A trajectory estimation step of estimating an imaging trajectory from the first moving image,
A trajectory correction step of correcting the estimated imaging trajectory,
An image generation step of generating a second moving image according to the corrected imaging trajectory from the first moving image,
In the trajectory correction step,
The imaging locus is detected so as to reduce the variation of the moving body position in each frame of the second moving image, which is detected from the moving body area based on the motion vector detected from the first moving image, and is calculated from the detected moving body area. A program for causing a computer to execute the process of correcting the.

Images