JP5251410B2 - Camera work calculation program, imaging apparatus, and camera work calculation method - Google Patents

Description

The present invention relates to a camera work calculation program, an imaging apparatus, and a camera work calculation method.

  Conventionally, camera shake correction processing and camera work optimization processing in moving image shooting are roughly classified into processing performed during moving image shooting and processing performed after shooting.

For example, Patent Document 1 performs processing after shooting a moving image, and detects camera shake or camera movement from movement between fields or frames based on the shot moving image, and performs camera shake correction. In addition, a technique is disclosed in which the trajectory of the camera itself is obtained to estimate the trajectory of the camera work intended by the user, and the motion between fields or frames is corrected so as to be the trajectory of the camera work.
JP 2008-005109 A

  However, the conventional technique such as Patent Document 1 aims at how to smoothly connect each image, which is a frame, so that the subject image in the moving image is stationary. When it is greatly shifted from the position of other images, there is a problem that when playing a video obtained by processing, only that frame is displayed or only part of it is displayed and it is unsightly. .

  In view of the problems of the above-described conventional technology, an object of the present invention is to provide a technology capable of generating a smooth moving image that is easy to see by camera work desired by a user.

In order to solve the above problems, the camera work calculation program of the present invention, the feature quantity extracted for a plurality of images constituting the moving image data generated by continuous imaging an object image, based on the feature amount, a detecting step of detecting a positional deviation amount between the focus cormorants image adjacent among the plurality of images, based on the positional deviation amount between the detected by the detection step image, the first camera work when capturing the moving image data calculating a first camera work calculation step of obtaining a trajectory, the trajectory of the second camera work within a range of positions on the trajectory of the predetermined threshold of the first camera work calculated by the first camera work calculation step a second camera work calculation step, on the trajectory of the second camera work determined by the second camera work calculation step, based on the position information of each of the plurality of images Te, to execute a correction step of correcting the relative position of each of the plurality of images on the computer.

  Further, the present invention further includes a frame shaping step of cutting and shaping surrounding pixels within a predetermined threshold size range for each of the plurality of images whose relative positions have been corrected by the correction step.

Further, in the present invention, positional deviation amount between the images that will be detected in the detection step comprises, one direction component in the image plane, a direction component intersecting the one direction component, and a rotational component.

Further, in the present invention, when the positional deviation amount between the detected Ru image in the detection step is larger than the predetermined value further comprises determining scene determination step interest has changed in the subject image of the moving image data.

Further, in the present invention, the second camera work calculation step is performed such that the sum of the differences between the first camera work locus and the second camera work locus is minimized within a predetermined threshold range. The trajectory of the second camera work is calculated.

In the present invention, the second camera work calculation step performs the sum of the differences for each direction component, the direction component intersecting the one direction component, and the rotation component, and the sum of the differences for each component. The trajectory of the second camera work is calculated so as to be minimized within a predetermined threshold range for each component.
In the present invention, the second camera work calculation step calculates the locus of the second camera work within a predetermined threshold range having a certain correlation with each other between adjacent images.
Further, in the present invention, the constant correlation has a certain size of the area where two images overlap.

The imaging apparatus of the present invention includes an imaging unit that continuously captures subject images and generates moving image data, and a control unit that executes the camera work calculation program of the present invention on the moving image data.

Camerawork calculation method of the present invention, the plurality of images constituting the moving image data generated by continuous imaging an object image and extracts a feature quantity, based on the feature amount, intends case adjacent among the plurality of images a detection step of detecting a positional deviation amount between the images, the first camera work based on the positional deviation amount between the image detected by the detecting step, obtaining the trajectory of the first camera work when capturing the moving image data a calculating step, and the second camera work calculation step of calculating the trajectory of the second camera work from a position on the trajectory of the first camera work calculated by the first camera work calculation step within a predetermined threshold value , on the trajectory of the second camera work determined by the second camera work calculating step, based on the position information of each of the plurality of images, and a correction step of correcting the relative position of each of the plurality of images Obtain.

  According to the present invention, it is possible to generate a smooth moving image that is easy to see depending on the camera work desired by the user.

  FIG. 1 is a conceptual diagram of a computer 10 in which a camera work optimization program according to an embodiment of the present invention is installed.

  A computer 10 shown in FIG. 1 includes a CPU 1, a storage unit 2, an input / output interface (input / output I / F) 3 and a bus 4, and the CPU 1, storage unit 2 and input / output I / F 3 are connected via the bus 4. Connected so that information can be transmitted. The computer 10 is connected to an output device 5 for displaying the progress of image processing and processing results, and an input device 6 for receiving input from the user via an input / output I / F 3. As the output device 5, a general liquid crystal monitor, a printer, or the like can be used. As the input device 6, a keyboard, a mouse, or the like can be appropriately selected and used.

  The CPU 1 reads a camera work optimization program stored in the storage unit 2 based on an instruction from the user received by the input device 6, and performs processing on the moving image data stored in the storage unit 2. The CPU 1 displays the result of the moving image processing on the output device 5 or records it in the storage unit 2. A general central processing unit can be used for the CPU 1.

  The storage unit 2 holds not only the camera work optimization program and moving image data in which the subject image is captured, but also moving image data that is smoothly processed so as to be easily viewed with the camera work desired by the user. The storage unit 2 also stores data such as parameter threshold values necessary for such processing. The moving image data, program, and other data stored in the storage unit 2 can be referred to as appropriate from the CPU 1 via the bus 4. A storage device such as a general hard disk device or a magneto-optical disk device can be selected and used for the storage unit 2. 1 is incorporated in the computer 10, it may be an external storage device. In this case, the storage unit 2 is connected to the computer 10 via the input / output I / F 3.

  Next, the procedure of the camera work optimization process according to the present embodiment will be described with reference to the flowchart of FIG.

  When the user inputs a command of the camera work optimization program using the input device 6 or double-clicks the icon of the program, a program start command is issued to the CPU 1. The CPU 1 receives the command through the input / output I / F 3 and activates the camera work optimization program stored in the storage unit 2. Along with the activation of the program, the CPU 1 reads the parameter threshold for the camera work optimization process and temporarily stores it in a memory (not shown). Then, the user sets a threshold value of a positional deviation amount that is necessary when solving the minimization problem with an inequality constraint condition for obtaining an optimal camerawork trajectory in step S15 described later. As a result, the processing from step S10 in FIG. 2 is performed.

  In the present embodiment, it is assumed that moving image data obtained by capturing a subject image with an electronic camera or the like, which is a processing target, is stored in the storage unit 2 in advance. In the present embodiment, the camera work optimization process is performed based on the 0th frame image of the moving image data.

  Step S10: The CPU 1 accepts designation of a moving image in which the user wants to perform camerawork optimization processing, for example, the scene shown in FIG. Is read and temporarily stored in a memory (not shown).

  Step S11: The CPU 1 obtains a positional shift amount between the images of adjacent frames using the images of a plurality of frames constituting the moving image data read in Step S10. For this purpose, the CPU 1 starts from an image of the first frame (0th frame) and an image of the next first frame, and sequentially obtains a positional shift amount between images of adjacent frames. Therefore, the CPU 1 extracts feature points of the image based on the edge amount that is the feature amount of the image. Specifically, the CPU 1 extracts a plurality of feature points using a KLT (Kanade-Lucas-Tomasi) method and multi-resolution conversion. In general, the KLT method is used to obtain a motion vector of a corresponding part between two images, but the KLT method in the present embodiment is large in the horizontal scanning direction and the vertical scanning direction orthogonal to each other in the image. Use a point with the function of extracting a point with an edge gradient, that is, a point at the corner of the subject image (for example, J. Shin and C. Tomasi, 'Good Features to Track', IEEE Conference on Computer Vision and (See Pattern Recognition (CVPR94) Seattle, June 1994). Furthermore, the accuracy of feature point extraction is increased by using multi-resolution conversion together.

  An example of extracting feature points in this way is shown in FIG. In FIG. 3, in each of the image 20a of the 0th frame and the image 20b of the first frame, feature points (● marks) are extracted at the edge of the mountain, the boundary between the person and the background, and the like. Note that the number of points to be extracted is at least three, and the larger the number of points, the more accurately the amount of positional deviation between images of each frame to be obtained later. In this embodiment, among the extracted feature points, the feature points are extracted in order from the largest value having an edge amount equal to or greater than a predetermined value. Specifically, feature points up to the top 100 were extracted. It should be noted that the number of feature points can be arbitrarily determined according to the shooting situation of the image.

  Step S12: The CPU 1 calculates a motion vector at each feature point by applying an optical flow method to each feature point in the image 20a of the 0th frame and the image 20b of the first frame extracted in step S11. To do. Specifically, the CPU 1 uses the above-described KLT method and, when all feature points in two images of adjacent frames are present with each other, obtains motion vectors for all feature points. On the other hand, when some of the feature points in two images of adjacent frames are framed in or out of the image area (for example, the feature point on the right side of the 0th frame image is out of frame, When the feature point on the left tree in the frame image 20b is in the frame), the CPU 1 obtains a motion vector of only the feature point existing in both images. The arrow shown in the image 20b of the first frame in FIG. 4 indicates the motion vector at each feature point obtained by the above procedure.

  Next, the CPU 1 uses an X component (pixel) in the horizontal scanning direction and the vertical scanning direction by using the feature points for which the motion vector can be obtained from the obtained image 20a of the 0th frame and the image 20b of the first frame. A positional shift amount consisting of a Y component (pixel) and a rotation component θ (degrees) is calculated. In the present embodiment, a method of calculating by using a transformation matrix of the affine transformation composed of the three components, which is one of the methods for calculating the positional deviation amount, is used. That is, the amount of positional deviation between images in each frame is calculated using the equation represented by the following equation (2) having the transformation matrix A represented by the following equation (1).

Formula 1

Formula 2

That is, the CPU 1 coordinates (x, y, 1) of each feature point in the image 20a of the 0th frame corresponding to each motion vector obtained above and the coordinates (x, y, 1) of the feature points in the corresponding image 20b of the first frame. Substituting x ′, y ′, 1) into equation (2), respectively, the position deviation amount (x ₀ , y ₀ , θ), which is a parameter of equation (1), is calculated by the least square method. Then, the CPU 1 records the amount of positional deviation (x ₀ , y ₀ , θ) between the 0th frame image 20 a and the first frame image 20 b in the storage unit 2.

Step S13: The CPU 1 integrates the positional deviation amounts (x ₀ , y ₀ , θ) between the images of the adjacent frames obtained in step S12 for each component as in the following equation (3).

Formula 3

Here, Xm (i), Ym (i), and θm (i) are obtained by integrating the amount of positional deviation (x ₀ , y ₀ , θ) from the 0th frame image to the i-th frame image for each component. Represents a value (i = 0, 1, 2,...). The integrated values (Xm (i), Ym (i), θm (i)) give the camerawork trajectory at the time of actual video shooting, and the position of the image of the i-th frame in the actual camerawork trajectory. Show. Note that the initial values of the integrated values (Xm (0), Ym (0), θm (0)) are preferably set to 0, respectively.

  Step S14: The CPU 1 determines whether or not the processing from step S11 to step S13 has been performed for all the frames of the moving image data. If the CPU 1 determines that all processing has been performed, the process proceeds to step S15 (YES side). On the other hand, if the CPU 1 determines that the process is in progress, the process proceeds to step S11 (NO side). Next, from step S11, the next adjacent first frame image 20b and second frame image are processed. Processing up to step S13 is performed. The CPU 1 performs processing up to the image of the Nth frame (N = 1, 2, 3,...) That is the last frame of the moving image data, and proceeds to step S15.

  Here, FIG. 4 shows (a) the trajectory of the components of Xm and Ym of the actual camera work obtained by the processing from step S11 to step S13 in the image up to the fifth frame among all the frames of the moving image data. (B) An example of the locus of the rotation component θm is shown. In the example of FIG. 4A, since the value of the Xm component has changed greatly, the actual camerawork trajectory is a case of panning in the X direction although there is a certain amount of camera shake. I understand.

  Step S15: The CPU 1 obtains the optimum camerawork locus desired by the user based on the actual camerawork locus obtained in Step S14.

  Here, how to obtain it will be briefly described. Conventional camera work optimization methods include the DP (Douglas-Pecker) method and the B-Spline method used in Patent Document 1 and the like. However, in the processing by these methods, an arbitrary curve, a straight line, an approximate curve, or the like is regarded as an optimal camerawork trajectory. For this reason, for example, when there is an image of a frame whose position is greatly deviated from the image of another adjacent frame, the image is greatly deviated from the arbitrary curve, straight line, approximate curve, or the like obtained as the optimal camerawork trajectory. . As a result, when a moving image obtained by processing is reproduced, the image of the frame that has been greatly misaligned is not completely visible or only partially visible. In other words, a moving image obtained by a conventional processing method is not necessarily an optimal camerawork trajectory and is easy to see and smooth. This is because the conventional processing method uses the above-described curve, straight line, or approximate curve obtained as a mathematically optimal solution as an optimal camerawork trajectory without any limitation.

  On the other hand, in this embodiment, in order to obtain the optimum camerawork trajectory from the actual camerawork trajectory, search is performed within a predetermined limit with a certain correlation between the images of each frame. Ask. Then, by correcting the position of the image of each frame of the moving image so as to be on the locus of the optimal camera work, a smooth moving image that is easy to see is generated. Here, the “constant correlation” means that the size of the region where two images overlap has a certain value or more. In addition, “smooth moving image that is easy to see” means a moving image in which an image of each frame cannot be seen at all or only a part of the image cannot be seen.

Specifically, in the present embodiment, an optimal camerawork trajectory is obtained using an expression represented by the following expression (4). Here, α _i represents any one of the three components of the position (X (i), Y (i), θ (i)) of the image of the i-th frame in the optimal camerawork trajectory, and β _i is This represents one of the three components of the position (Xm (i), Ym (i), θm (i)) of the image of the i-th frame in the actual camerawork trajectory.

Formula 4

The CPU 1 obtains the value of α _i (i = 0 to N) that is the optimal camerawork trajectory that minimizes the parameter E of Equation (4) in each component. For this purpose, the parameter E is partially differentiated by each α _i in each component, and the value of α _i when the partial differential values are all 0 becomes the optimum camerawork trajectory to be obtained. However, this is the same as the conventional method. Therefore, in the present embodiment, when the equation (4) is partially differentiated and solved as described above, the position of the image of each frame (Xm (i), Ym (i), θm (i )) Is provided with a threshold value that is a limit and solved within the range of the threshold value, whereby the position of the image of each frame (X (i), Y (i), θ ( The value of α _i of each component in i)) is obtained. This means solving as a minimization problem with inequality constraints. In this embodiment, the quasi-Newton method is used to solve the minimization problem with an inequality constraint condition. In addition to the quasi-Newton method, a conjugate gradient method, a Nelder mead method, or a multiplier method may be used as another solution method in combination with a solution method for the instant minimization problem with inequality constraints.

  FIG. 5 shows a range of threshold values centered on the position (Xm (i), Ym (i), θm (i)) (● mark) of the image of each frame on the actual camerawork trajectory of FIG. , A region 30 and a region 31 indicated by □. In the present embodiment, the user sets the size of the threshold (ΔXm, ΔYm, Δθm) at the position (Xm (i), Ym (i), θm (i)) of each frame image to (± 10 pixels, ± 10 pixels, ± 5 degrees). In the locus of the rotation component θm in FIG. 5B, the horizontal axis represents the number of frames, so there is no threshold value originally, but the range of the rotation component θm threshold Δθm is easy to see as the region 31 for convenience. It is drawn with a certain width.

As described above, the CPU 1 performs partial differentiation on the parameter E for each component by each α _i and minimizes the value of the partial differentiation (X (i), Y (i), θ (i)). The value of the component α _i is searched within the range of the region 30 and the region 31. As a result, as shown in FIG. 5, an optimal camerawork trajectory obtained by connecting the points indicated by ○ marks is obtained, and the ○ marks indicate the positions of the images (X (i), Y ( i), θ (i)). Note that the mark ◯ indicating the locus of the rotation component θ (i) is always on the line of each frame for the above reason. The CPU 1 records (X (i), Y (i), θ (i)) of each frame image in the optimal camerawork trajectory in the storage unit 2.

  Step S16: The CPU 1 is the position of the image of each frame on the optimal camerawork trajectory obtained in Step S15 (X (i), Y (i), θ (i)) (i = 0 to N). Are used to sequentially correct the relative position of the image of each frame with reference to the image of the 0th frame. FIG. 6 shows an image 50 of the i-th frame and an image 51 obtained by correcting the image 50 based on (X (i), Y (i), θ (i)). The CPU 1 cuts the periphery of the image 51 with a threshold width of ΔXm and ΔYm, generates an image 60, records it in a memory (not shown), and displays the image 60 on the output device 5. In the present embodiment, correction is performed in units of pixels and degrees, but correction may be performed with accuracy in units of subpixels.

  Step S17: The CPU 1 determines whether or not the process of step S16 has been performed for all the frame images. If the CPU 1 determines that all processing has been performed, the process proceeds to step S18 (YES side). On the other hand, if the CPU 1 determines that the process is in progress, the process proceeds to step S16 (NO side), and the process of step S16 is performed on the image of the next (i + 1) th frame. The CPU 1 performs processing until the processing of the image of the Nth frame, which is the last frame of the moving image data, is completed, and the process proceeds to step S18.

  Step S18: The CPU 1 records the moving image data of the optimal camerawork trajectory obtained by the process of step S16 in the storage unit 2 and ends.

  As described above, the present embodiment obtains the actual camerawork trajectory based on the captured moving image data, and solves the minimization problem with the inequality constraint condition of Equation (4), thereby obtaining the optimum camera desired by the user. It can generate a smooth video taken along the work track.

In addition, the position is corrected so that the image of each frame comes on the optimal camerawork trajectory, and the periphery of the image 51 is cut out with the threshold width of ΔXm and ΔYm, and the image 60 is generated. Therefore, it is possible to obtain a smooth video that is easy to view.
≪Supplementary items for the embodiment≫
In the present embodiment, feature point extraction and motion vector detection between images in each frame are performed using the KLT method in combination with multi-resolution conversion, but the present invention is not limited to this. For example, edge extraction for extracting feature points is performed by a general known edge extraction method, and motion vector detection is performed by gradient method (LK method), block matching method, Hesse method, SIFT method, etc. It may be performed using different methods.

  In the present embodiment, the camera work optimization program is applied to moving image data obtained by shooting a subject image as a moving image. However, the present invention is not limited to this, and the present invention is also applied to still images shot continuously. Is possible. However, in the case of still images by continuous shooting, it is only necessary to display them one by one on the output device 5, and it is not necessary to superimpose like a moving image.

  In the present embodiment, the optimal camerawork trajectory is obtained using the 0th frame image of the moving image data as a reference image, but the present invention is not limited to this. For example, when the amount of positional deviation between adjacent frames of video data is larger than a predetermined value (for example, Δθm = ± 30 degrees), the imaging device performs panning or the like to change the subject image to be captured. Therefore, before and after the subject image changes, the camera work optimization process may be performed for each different moving image data. In this case, for example, for moving image data before a large positional shift, the reference image is the 0th frame image, and for moving image data after the positional shift, an image of the frame immediately after the positional shift is generated. A reference image may be used.

  In addition to comparing the amount of positional deviation between adjacent frames of moving image data with a predetermined value, the imaging device performs panning or the like when the magnitude of the motion vector obtained in step S12 is larger than the predetermined value. Thus, it may be determined that the subject image to be photographed has changed. Alternatively, by providing an angle sensor or the like in the imaging device, if the output value of the angle sensor is larger than a predetermined value, the imaging device determines that the subject image to be imaged has changed due to panning or the like. It may be.

  In the present embodiment, the inequality constraint conditional minimization problem according to the equation (4) is calculated for each component of (X (i), Y (i), θ (i)). However, the present invention is not limited to this, and the minimization problem with inequality constraints may be solved by Equation (4) in which all (X (i), Y (i), θ (i)) are combined into one.

  In this embodiment, the magnitudes of the respective threshold values (ΔXm, ΔYm, Δθm) of (Xm (i), Ym (i), θm (i)) are set to (± 10 pixels, ± 10 pixels, ± 5). However, the present invention is not limited to this, and an arbitrary value can be set.

  In this embodiment, the periphery of the image of each frame whose position has been corrected is cut out with the threshold values ΔXm and ΔYm. However, the present invention is not limited to this, and the actual camerawork trajectory and the optimum camerawork When the value of the difference from the locus is smaller than the threshold value ΔXm or ΔYm, the value may be cut off with a value smaller than the threshold value ΔXm or ΔYm.

  In the present embodiment, the positional shift amount is obtained based on motion information between images of adjacent frames. However, the present invention is not limited to this, and may be obtained based on motion information between fields.

  Note that the present invention is also applicable to an imaging apparatus having a control unit that executes each step of the camera work optimization program according to the present invention.

  Note that the present invention is also applicable to a recording medium that stores a camera work optimization program according to the present invention.

  It should be noted that the present invention can be implemented in various other forms without departing from the spirit or main features thereof. For this reason, the above-described embodiment is merely an example in all respects and should not be construed in a limited manner. Further, all modifications and changes belonging to the equivalent scope of the claims are within the scope of the present invention.

1 is a conceptual diagram of a computer 10 installed with a camera work optimization program according to an embodiment of the present invention The flowchart which shows the procedure in the camera work optimization process which concerns on this embodiment Illustration of an example of an image of adjacent frames Actual camerawork trajectory Diagram of the optimal camerawork trajectory obtained based on the actual camerawork trajectory The figure explaining correction of the position of the image of each frame

Explanation of symbols

1 CPU, 2 storage unit, 3 input / output I / F, 4 bus, 5 output device, 6 input device, 10 computer, 20a 0th frame image, 20b 1st frame image

Claims (10)

For a plurality of images constituting a continuous captured moving image data generated by an object image extracting feature amount, based on the feature amount, it detects the positional deviation amount between the focus cormorants image adjacent among the plurality of images Detecting step to
Based on the positional shift amount between the image detected by said detection step, a first camera work calculation step of obtaining a trajectory of the first camera work when imaging the moving picture data,
A second camera work calculation step for calculating a second camera work locus within a predetermined threshold range from a position on the locus of the first camera work obtained by the first camera work calculation step;
Correction step of correcting the relative position of each of the plurality of images based on the position information of each of the plurality of images on the trajectory of the second camera work obtained by the second camera work calculation step. A camera work calculation program for causing a computer to execute and. In the camera work calculation program according to claim 1,
The method further comprises a frame shaping step of cutting and shaping surrounding pixels within the predetermined threshold size for each of the plurality of images whose relative positions have been corrected by the correction step. A camera work calculation program. In the camera work calculation program according to claim 1 or 2,
Positional deviation amount between the pre-Symbol image detected in the detection step, the camera work and having one direction component in the image plane, the direction component intersecting the one direction component, and a rotational component Calculation program. In the camera work calculation program according to any one of claims 1 to 3,
Wherein when positional deviation amount between before Symbol image detected in the detection step is larger than the predetermined value, and further comprising determining scene determination step interest has changed in the subject image of the moving image data Camerawork calculation program. In the camera work calculation program according to any one of claims 1 to 4,
In the second camera work calculation step, the sum of the differences between the trajectory of the first camera work and the trajectory of the second camera work is minimized within the predetermined threshold range . 2. A camera work calculation program for calculating a trajectory of two camera works. In the camera work calculation program according to any one of claims 3 to 5,
The second camera work calculation step performs the sum of the differences for each of the one direction component, the direction component intersecting the one direction component, and the rotation component, and the sum of the differences for each component. , said to minimize within each of said predetermined threshold for each component, camera work calculation program and calculates a trajectory of the second camera work. In the camera work calculation program according to any one of claims 1 to 6 ,
The second camera work calculating step, between the adjacent images, camera work calculation program and calculates a trajectory of the second camera work within a predetermined threshold value having a certain correlation with each other . In the camera work calculation program according to claim 7 ,
The constant correlation, camerawork calculation program size of an area overlapping the two images are characterized Rukoto that Yusuke certain level. An imaging unit that continuously captures subject images and generates moving image data;
A control unit that executes the camera work calculation program according to any one of claims 1 to 8 on the moving image data;
An imaging apparatus comprising: Detection that extracts feature amounts of a plurality of images constituting moving image data generated by continuously capturing subject images, and detects a positional shift amount between adjacent images among the plurality of images based on the feature amounts. Process,
A first camera work calculation step for obtaining a trajectory of the first camera work when the moving image data is imaged based on the positional deviation amount between the images detected by the detection step;
  A second camera work calculation step for calculating a second camera work locus within a predetermined threshold range from a position on the first camera work locus obtained by the first camera work calculation step;
A correction step of correcting the relative position of each of the plurality of images based on the position information of each of the plurality of images on the trajectory of the second camera work obtained by the second camera work calculation step. When
A camera work calculation method comprising:

Images