JP2010011050A - Moving picture processing unit, image recording device integrated with camera, moving picture reproducing device, remote control system, moving picture processing program, storage medium, and moving picture processing system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To eliminate discomfort caused by sway of a moving picture obtained by mobile photographing, such as an image photographed by a VTR integrated with a camera which a photographer who moves by a walk or the like uses, an image from a photographing device attached to a robot, which is displayed on a display of an operator for remote control of the robot, or an image photographed by a photographing device attached to an autonomously moving robot. <P>SOLUTION: Image conversion processing is performed so that a rotational angle in the photographing device becomes smooth (1). Only an image suitable for estimating a parameter used for image conversion is used, and only a precise parameter is used (2). <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a moving image processing apparatus and a moving image processing method for correcting image shaking of a moving image, and in particular, an image taken while a photographer walks and moves using a camera-integrated VTR, or remote operation of a robot Images of moving images, such as images from an imaging device attached to a robot and images taken by an imaging device attached to an autonomously moving robot, displayed on the operator's display The present invention relates to a moving image processing apparatus and a moving image processing method for eliminating discomfort caused by shaking.

  Conventionally, as an anti-vibration technique for preventing image vibration due to camera shake in moving image shooting using a camera-integrated video or the like, an image larger than the output image is input as shown in the conventional example of Patent Document 1. There is a camera shake correction technique in which an image is temporarily stored in a memory and an output image is cut out from the memory for each frame based on the magnitude of a motion vector generated by camera shake.

  Also, in moving shooting in movies and various sports competitions, shooting is performed by mounting a shooting device on a carriage that moves on two rails installed in parallel. In addition, a wire is used for a photographing crane for preventing the photographer's behavior from being transmitted to the photographing device in the moving photographing as in Patent Document 2, and a moving table equipped with the photographing device as in Patent Document 3. There is something to move.

Furthermore, “Recovery of Ego-Motion Using Image Stabilization” (M.Irani etc, CVPR-94, June) is described as a method for reducing the discomfort of moving images taken by moving images using image processing. In this way, time-sequential images are processed to obtain translation / rotation movement parameters of the photographing apparatus, and only the rotation parameters in them are used so as to be in the same orientation as the first photographed image. There is a method of correcting vibration of an image by converting each image.
JP-A-3-280406 JP-A-5-161037 JP 7-325335 A

  Images taken while the photographer moves are subject to image fluctuations due to changes in the position of the photographing device or the photographing direction. In order to eliminate this unpleasant feeling, the methods for preventing image shake during moving shooting as described above have the following problems.

  The method disclosed in Patent Document 1 is based on the temporally previous image as a reference, obtains the movement amount of the subsequent image as a motion vector, and changes the read position of the memory for the motion vector. To make the previous image and the subsequent image look the same. Subsequently, the movement vector between the cut-out image and the image at the next time is obtained, and the image is cut out based on the obtained movement vector. It is effective for taking pictures under extremely small shaking conditions. However, moving shooting is not such a situation, and the object in the image gradually moves up, down, left, and right around the traveling direction of the photographer. In other words, the subject that is initially at the center of the image gradually increases in the image if the photographer moves toward the subject, and gradually moves away from the center of the image if the moving direction differs from the direction of the subject. It will be. In such an image sequence, the same area as that of the first image gradually decreases, so that it is impossible to cut out the same area.

  In addition, the installation of rails and wires or the use of a crane has a problem that it is a large facility.

  Also, in the method based on the paper “Recovery of Ego-Motion Using Image Stabilization”, when correcting a moving image, each image is converted to the orientation of the image taken first, so the direction of travel is not straight but curved. However, there is a problem that the converted image differs from the desired one when the direction of movement is gradually changing, such as when shooting, or when shooting is performed by rotating the shooting device such as panning or tilting. It was.

  Furthermore, this method has a problem that image fluctuation due to translation cannot be corrected because only the rotation component with the translation component completely separated is converted. Therefore, this paper explains that a low-pass filter is applied at the end of processing.

  As described above, conventionally, there is no method for correcting image blur without requiring a large facility in moving shooting. SUMMARY OF THE INVENTION Accordingly, an object of the present invention is to provide a moving image processing apparatus and method for correcting image shake during moving shooting by image processing without requiring a large facility.

In order to achieve the purpose,
With the moving image processing apparatus of the first invention according to the present application,
Parameter extraction means for obtaining a parameter value at each time of an input moving image, and ideal parameter calculation means for statistically processing the parameter value at each time obtained by the parameter extraction means and calculating an ideal parameter value at each time A correction amount is calculated based on the image memory that holds the old and new images, the actual parameter value obtained by the parameter extraction unit and the ideal parameter value calculated by the ideal parameter calculation unit, and based on the correction amount Image conversion means for converting the image held in the image memory, and determination means for determining whether the input signal is used or not. When the determination means determines that the input signal is not used, the image A moving image processing apparatus that converts an old image held in a memory and outputs the converted image.

The moving image processing apparatus of the second invention related to this application is:
In the first invention, when the determining means determines that the input signal is not used and converts the old image held in the image memory, the ideal parameter calculating means predicts the ideal parameter to obtain the predicted ideal parameter, The moving image processing apparatus is characterized in that a correction amount is calculated from the predicted ideal parameter and a parameter obtained with respect to the old image.

The moving image processing apparatus of the third invention according to the present application is:
In the first or second invention, the input signal of the discriminating means is a parameter obtained by the parameter extracting means. The accuracy of the parameter is investigated by the discriminating means, and when the accuracy is low, the parameter is used. It is a moving image processing apparatus characterized by not determining.

The moving image processing apparatus according to the fourth aspect of the present application is:
In a third aspect of the present invention, the parameter extracted by the parameter extracting means is a rotation amount of the image pickup device.

The moving image processing apparatus according to the fifth aspect of the present application is:
In the first or second invention, the input signal of the discriminating means is an image signal, the amount of the low frequency component when the discriminating means investigates the frequency component of the image, or the movement obtained with respect to the image The moving image processing apparatus is characterized in that the image signal is determined not to be used based on a ratio of motion vectors having low reliability when the reliability of the vector is examined.

The sixth invention related to this application is:
A camera-integrated image recording apparatus that processes a captured moving image by the moving image processing described in any one of the first to fifth inventions and records the processed moving image.

The seventh invention related to this application is:
A moving image reproducing apparatus that processes a captured or recorded moving image by the moving image processing described in any one of the first to fifth inventions and reproduces the processed moving image.

The eighth invention according to the present application is
The moving image captured by the imaging device attached to the moving means is subjected to the moving image processing described in any one of the first to fifth inventions, and the processed moving image is transmitted to the network. The remote operation system displays the processed moving image via a network to a remote operator.

The ninth invention according to this application is
A moving image photographed by an imaging device attached to the moving means is transmitted to a network, and a moving image photographed by the moving means obtained via the network is any one of the first to fifth inventions. This is a remote operation system that performs the described moving image processing and displays the processed moving image to an operator who performs remote operation.

The tenth invention related to this application is:
A parameter extraction step for obtaining a parameter value at each time of the moving image; an ideal parameter calculation step for statistically processing the parameter value at each time obtained in the parameter extraction step and calculating an ideal parameter value at each time; An image for calculating a correction amount based on the actual parameter value obtained in the parameter extraction step and the ideal parameter value calculated in the ideal parameter calculation step, and converting an image held in the image memory based on the correction amount In a moving image processing method comprising a conversion step and a determination step for determining whether the input signal is used or not, when the determination step determines that the input signal is not used, the old image held in the image memory is converted. The video processing method is characterized in that the video is output in the same manner.

The eleventh invention related to this application is:
In the tenth invention, when the determination step determines that the input signal is not used and converts the old image held in the image memory, the ideal parameter calculation step predicts the ideal parameter to obtain the predicted ideal parameter, In this moving image processing method, a correction amount is calculated from the predicted ideal parameter and a parameter obtained with respect to the old image.

The twelfth invention related to this application is:
In the tenth or eleventh invention, the input signal of the discrimination step is a parameter obtained in the parameter extraction step. The accuracy of the parameter is investigated in the discrimination step, and when the accuracy is low, the parameter is used. This is a moving image processing method characterized in that it is determined not to be performed.

The thirteenth invention related to this application is
In a twelfth aspect of the invention, the parameter extracted in the parameter extraction step is a rotation amount of the imaging device.

The fourteenth invention related to this application is
In the tenth or eleventh invention, the input signal of the discrimination step is an image signal. In the discrimination step, the amount of the low frequency component when the frequency component of the image is examined, or the movement obtained with respect to the image The moving image processing method is characterized in that the image signal is determined not to be used based on a ratio of motion vectors having low reliability when the reliability of the vector is examined.

The fifteenth aspect of the present application is
A moving image processing program for causing a computer to execute the moving image processing method described in any one of the tenth to fourteenth aspects of the invention, which is a moving image processing program readable by the computer.

  A sixteenth aspect of the present application is a storage medium that stores the moving image processing program described in the fifteenth aspect in a computer-readable form.

According to a seventeenth aspect of the present application,
Image input means, an image memory for holding old and new images, the moving image processing program according to the fifteenth invention, a processor for executing the moving image processing program, and a storage means for storing the moving image processing program Recording means for recording a moving image processed by the moving image processing program, display means for displaying a moving image processed by the moving image processing program, or a network for processing a moving image processed by the moving image processing program It is a moving image processing system comprised from the transmission means to transmit to.

  In order to correct the image shake of an image shot by changing the position or shooting direction of the shooting device due to the movement of the photographer, the parameter extraction unit first sets the parameter value at each time from the image shot at each time. Ask for. This parameter is the rotation amount of the imaging device. The ideal parameter calculation means obtains an ideal rotation amount at each time using past parameters so that the rotation amount of the photographing apparatus changes smoothly at each time. Then, the image stored in the image memory is converted based on the correction amount calculated based on the actual rotation amount and the ideal rotation amount by the image conversion means. The discrimination means discriminates whether the input signal is used or not. This input signal is one parameter, and the other is an input image. When the input signal is used as a parameter, the discrimination means investigates the accuracy of the parameter and determines whether or not to use the parameter. When the input signal is an input image, the discriminating unit investigates the frequency component of the image and the reliability of the motion vector related to the image, and uses the input image based on the amount of the low frequency component and the reliable motion vector ratio. Determine whether or not. When the discriminating means discriminates that the input signal is used as it is, the new image held in the image memory is converted and output. When it is discriminated that the input signal is not used, the old image held in the image memory is converted. Output.

  In addition, when the determination unit determines that the input signal is not used, the ideal parameter calculation unit obtains a predicted ideal parameter by predicting the ideal parameter from the stored information, and calculates the predicted ideal parameter and the old image. The old image in the image memory is converted from the parameters and output.

  As described in the above embodiment, according to the moving image processing apparatus and method of the present invention, image conversion processing is performed so that the rotation angle of the photographing apparatus is smoothed without requiring a large facility. Even if the movement is such that the traveling direction changes, there is an effect that the image shake at the time of moving photographing can be corrected.

  Further, at that time, using only an image suitable for estimating a parameter used for image conversion or using only a highly accurate parameter has an effect of improving the accuracy of image shake correction by image conversion.

  In addition, this processing is used in a camera-integrated recording apparatus to record an image processed so as not to shake the captured image, a device for reproducing the captured image or the recorded image, a robot, etc. The moving means can be used in a system that remotely operates via a network, and an image obtained by processing a moving image photographed by the moving means so as not to be shaken can be displayed to the operator, thereby eliminating the discomfort of the observer. There is an effect. In addition, an image input by the image input means is executed by a processor of the program, and an image processed so as not to be shaken is recorded or displayed or transmitted to a network. There is also an effect that processing can be performed.

  Next, details of the present invention will be described in accordance with the description of the embodiments.

  Embodiments of the present invention will be described below with reference to the drawings.

  FIG. 1 is a diagram showing the configuration of this embodiment.

  In FIG. 1, 100 is an image memory, 101 is an image memory control means, 110 is a motion vector extraction means, 120 is a parameter extraction means, 130 is a parameter determination means, 140 is an ideal parameter calculation means, and 150 is The image conversion means, 160 an interpolation means, 170 a recording means, 180 a display means, and 190 a transmission means.

  The functions of the components shown in FIG. 1 will be described below.

  The image memory 100 is a memory for holding a digitized image, and the image memory control means 101 controls the writing of the image into the image memory 100, the reading from the image memory 100, and the setting of the address. Do. The motion vector extraction unit 110 performs matching between the image held in the image memory 100 and the next image, and extracts a motion vector indicating movement of a point in the image. The parameter extraction unit 120 obtains parameters such as rotation between temporally continuous images from the motion vector obtained by the motion vector extraction unit 110. The parameter determination unit 130 determines the accuracy of the parameter obtained by the parameter extraction unit 120. If the accuracy is high, the parameter determination unit 130 outputs the parameter to the ideal parameter calculation unit 140. If the accuracy is low, the parameter determination unit 130 does not use the parameter. Is notified to the ideal parameter calculation means 140. The ideal parameter calculation unit 140 stores a plurality of parameters obtained by the parameter extraction unit 120 and, if a parameter is received from the parameter determination unit 130, calculates the ideal parameter using the parameter. Further, if a signal indicating that the parameter is not used is received from the parameter determining unit 130, an ideal parameter is calculated using the held parameter. Based on the parameters obtained by the parameter extraction means 120 and the ideal parameters calculated by the ideal parameter calculation means 140, the image conversion means 150 can select an image to be recorded on the recording means 170 from the image held in the image memory 100, or The coordinates of the image memory 100 for creating an image to be displayed on the display unit 180 or a converted image to be transmitted from the transmission unit 190 are calculated and output to the image memory control unit 101. The interpolation unit 160 interpolates the converted image read from the image memory 100 and outputs it. The recording unit 170 records the image output from the interpolation unit 160. The display unit 180 displays the image output from the interpolation unit 160. The transmission unit 190 transmits the image output from the interpolation unit 160 to the Internet or the like.

  Subsequently, the overall operation of the moving image processing apparatus and method will be described with reference to FIGS.

  The digital input image is branched and input to the image memory 100 and the motion vector extraction means 110. In the path to be input to the image memory 100, the input image is held in the image memory 100 under the control of the image memory control unit 101. It is also input to the motion vector extracting means 110 and used for calculation of motion vector extraction.

  The image memory 100 holds the digital input image, reads the image held at the previous time, and outputs it to the motion vector extraction means 110. That is, the image memory 100 holds two images, an old image and a new image.

  Subsequently, the motion vector extraction unit 110 performs matching between the input digital input image and the image at the previous time input from the image memory 100 to obtain a motion vector in the image. The selection of the point for obtaining the motion vector may be a point on a strong edge in the image, which may be automatically selected from a plurality of points separated by a certain amount, or may be selected by a human. The extracted motion vector and its reliability are output to the parameter extracting means 120. The reliability of the motion vector may be, for example, the residual value in the case of template matching of the residual method. In this case, the reliability becomes higher as it is closer to 0. In the correlation method, a correlation value may be used. In this case, the closer to 1, the higher the reliability.

  The parameter extraction unit 120 obtains an image rotation parameter between temporally consecutive images from the input motion vector. There are several ways to determine this amount of rotation. For example, the method called the 8-point algorithm described in Koichiro Deguchi “Basics of Robot Vision”, and the “Recovery of Ego-Motion” shown in the previous example. You can use any of the three-image methods described in the “Using Image Stabilization” paper. Here, a method using affine transformation will be described.

The horizontal axis in the image is the X axis, the vertical axis is the Y axis, the direction perpendicular to the screen is the Z axis, and the rotation angles about the XYZ axes are (ψ, φ, θ). If the corresponding point coordinates of the subsequent image in time are (Xa, Ya) and the coordinates of the previous image are (Xb, Yb), the subsequent image is rotated by θ with respect to the previous image, (dx, dy). ) Let's say that the translation, m times magnification,
Xa = (cosθ · Xb + sinθ · Yb−dx) · m
= A · Xb + B · Yb + C (1)
Ya = (-sinθ · Xb + cosθ · Yb-dy) · m
= -B · Xb + A · Yb + D (2)
(However, A = m · cos θ, B = m · sin θ, C = −m · dx, D = −m · dy)
There is a relationship. Substituting the coordinates of the motion vectors of a plurality of points obtained by the motion vector extracting means 110 into this equation, the coefficients of A, B, C, and D are obtained by the least square method, and from there, the rotation amount θ and the parallel movement amount dx, dy, and enlargement amount m are obtained. Further, since the image is generally changed by the rotation component rather than the translation component of the photographer's movement, it is assumed that the parallel movement amounts dx and dy are caused by the rotation φ and the rotation ψ, respectively, and the rotation angle ψ , Φ are obtained as φ = arctan (dx / f) and ψ = arctan (dy / f). Here, f is a focal length of the imaging system, and is obtained from an automatic focusing means (not shown). At this time, some of the input motion vectors are mis-extracted, and using these mis-extracted results reduces the accuracy of the required parameters. A motion vector can be determined. Each parameter calculated in this way is output to the parameter discrimination means 130.

  The parameter discriminating unit 130 discriminates the accuracy of the input parameter, and when it is discriminated that the accuracy is high, the parameter is outputted to the ideal parameter calculating unit 140, and conversely, when it is discriminated that the accuracy is low and cannot be used, The parameter is discarded, and the result indicating that the parameter cannot be used is output to the ideal parameter calculation means 140. Here, the operation of discriminating the input parameters will be described with reference to the flowchart of FIG. In step S201, the motion vector of each point obtained by the motion vector extraction means 110 is input. In step S202, for all input motion vectors, the coordinates of the start point of the motion vector (the point (Xb, Yb) of the previous image) are substituted into formulas (1) and (2) and obtained by the parameter extraction means 120. The coordinates (Xc, Yc) after conversion with the specified parameters are obtained. In step S203, the difference between the converted coordinate (Xc, Yc) and the end point coordinate (Xa, Ya) of the motion vector is obtained for all the motion vectors, and the sum of squares of the lengths (S = Σ ( (Xa-Xc) * (Xa-Xc) + (Ya-Yc) * (Ya-Yc))). In step S204, the difference obtained in step S203 is divided by the number of motion vectors to obtain an error (Es) at one point. In step S205, the error Es is compared with the threshold value ε, and if the error Es is smaller, the parameter is output to the ideal parameter calculation means 140, assuming that the accuracy of the parameter is high in step S206. If the error Es is larger, it is discarded in step S207 because the accuracy of the parameter is low, and the result that the parameter cannot be used is output to the ideal parameter calculation means 140. At the same time, a signal indicating whether or not the parameter is usable is output to the image conversion unit 150 and the image memory control unit 101.

  The ideal parameter calculation unit 140 adds the rotation parameters input via the parameter determination unit 130 in order to determine the actual rotation angle. This is because the rotation angles ψ, φ, and θ obtained by the parameter extraction means 120 are relative rotation angles between the images (rotation angles with reference to the image read in advance). That is, the amount of rotation of the device is obtained. In other words, with the first input image as a reference, if the rotation angles around the X, Y, and Z axes of the reference image are α, β, and γ, the rotation angles for the nth image are respectively , Α (n) = α (n−1) + ψ (n), β (n) = β (n−1) + φ (n), γ (n) = γ (n−1) + θ (n) . Since the first image is used as a reference, α (0) = β (0) = γ (0) = 0, and the rotation angle between the first image and the next image is ψ (1), φ. (1) and θ (1). This change in rotation angle is schematically shown in FIG. In FIG. 3, the horizontal axis is the image number, the vertical axis is the rotation angle of the photographing apparatus when the first image is used as a reference, (a) is the rotation angle α, (b) is the rotation angle β, and (c). Is the rotation angle γ. Then, an ideal rotation angle at which the rotation angle changes smoothly next is obtained by calculation. Specifically, as shown in FIGS. 3 (a), 3 (b), and 3 (c), a curve as indicated by a dotted line is represented by a quadratic minimum 2 with respect to an actual rotation angle indicated by a black dot. Obtained by multiplication. As a general operation, the ideal rotation angle is output to the image conversion means 150.

  When a reliable parameter is input from the parameter discriminating means 130, rotation angles ψ, φ, θ are newly input, so as shown in FIGS. 4 (a), 4 (b), and 4 (c). In addition, the graph relating to the rotation angles α, β, γ is updated. In FIG. 4, white circles indicate new rotation angles α, β, and γ updated from newly input ψ, φ, and θ. Then, the ideal rotation curve is newly calculated including the new rotation angle after deleting the oldest data. Then, the newly obtained ideal rotation angle is output to the image conversion means 150. That is, in FIG. 4, the rotation angle indicated by the circle with the mesh is output. On the other hand, when a signal indicating that the parameter cannot be used is input from the parameter determining unit 130, the ideal curve obtained previously is extrapolated to obtain a predicted ideal parameter. This situation is shown in FIGS. 5 (a), 5 (b), and 5 (c). In FIG. 5, the past rotation angle indicated by the black circle is the same as that shown in FIG. 3, and the ideal curve indicated by the dotted line is the same as that obtained from the past rotation angle. However, since a new rotation angle is not input this time, the length of one image is longer than that of FIG. 3 in order to obtain the ideal rotation angle. That is, this time, the parameter determining unit 130 determines that the accuracy of the parameter obtained by the parameter extracting unit 120 is low, and the parameter is discarded. Therefore, in order to create a corrected image at that time, the ideal curve is removed. The ideal parameter is obtained by insertion, and the ideal parameter, the previous image, and the parameter obtained from the previous image are used. That is, the ideal parameter calculation unit 140 outputs the predicted ideal parameter indicated by the white circle of the mesh on the extrapolated ideal curve shown in FIG.

  In the image conversion means 150, if there is a signal that the parameter is used from the parameter determination means 130, the difference between the parameter from the parameter detection means 120 and the ideal parameter from the ideal parameter calculation means 140 is obtained, and the rotation angle of the difference is used. Coordinates for transforming the image are calculated, output to the image memory control unit 101, read out from the new image held in the image memory 100, and output to the interpolation unit 160. On the other hand, if there is a signal that the parameter cannot be used from the parameter discriminating unit 130, the difference between the previous parameter from the parameter detecting unit 120 and the predicted ideal parameter from the ideal parameter calculating unit 140 is obtained, and the rotation angle of the difference is obtained. The coordinates for converting the image are calculated and output to the image memory control means 101, the old image held in the image memory 100 is read out, and output to the interpolation means 160. That is, when the accuracy of the parameter obtained from the new image is low, the old image is converted to become the predicted ideal parameter. This is shown in FIG. FIG. 6 shows the rotation angle α, but FIG. 6A shows the case where the parameter discriminating unit 130 discriminates that the accuracy of the parameter is high, and a black circle is actually inputted from the parameter extracting unit 120. The white circles shown by the parameter values and meshes are the ideal parameters input from the ideal parameter calculation means 140, and calculation is performed so as to convert at the difference rotation angle dα. On the other hand, FIG. 6B shows a case where the parameter discriminating unit 130 discriminates that the accuracy of the parameter is low. The black circle is the parameter value input from the parameter extracting unit 120 and the white circle shown by the mesh for the previous image. Is the predicted ideal parameter input from the ideal parameter calculation means 140 and is calculated so as to be converted by the difference dα. Incidentally, the dotted line is an ideal curve, and white circles indicate the ideal parameters at the previous time, and the difference dα ′ is the difference used for the conversion at the previous time.

  The image memory control unit 101 reads out either a new image or an old image among the images held in the image memory 100 based on a signal from the parameter determination unit 130. Also, when the next image comes, if the accuracy of the previous parameter is high, the next image is held in the area where the old image was held as a new image, and the previous new image is set as the old image. . Also, if the accuracy of the previous parameter is low, the image next to the area where the new image was held is held and set as a new image. At this time, the old image remains as it is.

  The overall operation when the parameter determination unit 130 determines that the accuracy of the parameter is low will be described again using the flowchart shown in FIG. In step S701 in FIG. 7, the ideal parameter calculation unit 140 obtains a predicted ideal parameter. In step S702, the image conversion means 150 obtains a correction amount from the previous parameter and the predicted ideal parameter. In step S703, the image memory control unit 101 reads the old image and holds the next image in the area where the new image is held. In step S703, it is possible to write the next image in the area where the new image is held while reading the old image.

  The interpolation unit 160 interpolates the pixel values of the coordinates calculated by the image conversion unit 150 based on the pixel values of the surrounding points. Then, the interpolated result is output. The output interpolation image is recorded by the recording unit 170, displayed by the display unit 180, or transmitted to the transmission unit 190 Internet or the like.

  As described above, in this embodiment, there is an effect that the image shake at the time of moving shooting can be corrected by performing the rotation conversion of the image and correcting the rotation of the imaging apparatus. In addition, the accuracy of the obtained rotation parameter is investigated, and when the parameter is used with high accuracy, the accuracy of the parameter is low due to erroneous extraction of the motion vector, translational movement of the imaging device, etc. There is an effect that the accuracy of correction is improved by using the predicted parameter. In addition, when the accuracy of the parameter is low, since the parameter is not used in the actual rotation angle calculation, there is also an effect that it is possible to prevent the influence on the subsequent image processing.

  In the first embodiment described above, whether or not to use the parameter is determined from the accuracy of the parameter obtained from the movement vector. On the other hand, in this embodiment, it is determined whether or not to use the input image. Specifically, it is determined whether or not the image is blurred. In other words, the shot image may be blurred due to the relationship between the instantaneous movement in the moving shooting and the exposure time of the image sensor. In such an image, an erroneous extraction of the movement vector is likely to occur, and even if the image is corrected and output, the image becomes difficult to see with many low frequencies. Therefore, in this embodiment, such a blurred image is discriminated and is not used for creating a corrected image.

  FIG. 8 is a diagram showing the configuration of this embodiment.

  In FIG. 8, the same numbers as those in FIG. In FIG. 8, 100 is an image memory, 801 is an image memory control means, 110 is a motion vector extraction means, 815 is an image discrimination means, 120 is a parameter extraction means, 840 is an ideal parameter calculation means, and 850 is an ideal parameter calculation means. The image conversion means, 160 an interpolation means, 170 a recording means, 180 a display means, and 190 a transmission means.

  In the following, the functions of the image discriminating means 815, the ideal parameter calculating means 840, the image converting means 850, and the image memory control means 801, which are means different from those in the first embodiment, in the components shown in FIG. 8 will be described. .

  The image discriminating means 815 investigates the frequency of the input image, and determines that the image is not used as a blurred image if there are many low frequencies. Also, it is determined that the image is not used even when there are many motion vectors with low reliability among the motion vectors input from the motion vector extraction unit 110. The ideal parameter calculation unit 840 holds a predetermined number of parameters obtained by the parameter extraction unit 120, calculates an ideal parameter from them, and outputs the ideal parameter to the image conversion unit 850. When a signal not to use an image is input from the image discrimination unit 815, the ideal parameter is predicted and the predicted ideal parameter is output to the image conversion unit 850. In the image conversion means 850, an image or display means for recording in the recording means 170 from the image held in the image memory 100 based on the parameters from the parameter calculation means 120 and the ideal parameters or the predicted ideal parameters from the ideal parameter calculation means 840. The coordinates of the image memory 100 for creating an image to be displayed on 180 or a converted image to be transmitted from the transmission means 190 are calculated and output to the image memory control means 801. The image memory control means 801 changes the control according to a signal indicating whether or not to use the image when controlling the writing of the image and the reading of the image from the image memory 100 and the setting of the address.

  Next, the overall operation of the moving image processing apparatus and method will be described with reference to FIGS.

  The digital input image is branched and input to the image memory 100, the motion vector extraction unit 110, and the image determination unit 815. In the path for inputting to the image memory 100, the input image is held in the image memory 100 under the control of the image memory control unit 801. Further, it is also input to the motion vector extracting means 110 and used for extracting a motion vector. Further, it is also input to the image discriminating means 815 to discriminate image blurring.

  The image memory 100 holds the digital input image, reads the image held at the previous time, and outputs it to the motion vector extraction means 110. That is, the image memory 100 holds two images, an old image and a new image.

  Subsequently, the motion vector extraction unit 110 performs matching between the input digital input image and the image at the previous time input from the image memory 100 to obtain a motion vector in the image. The selection of the point for obtaining the motion vector is the same as in the first embodiment. The extracted motion vector and its reliability are output to the image discrimination means 815.

  The image discriminating means 815 investigates the frequency of the input image, and determines that the image is not used as a blurred image if there are many low frequencies. Also, it is determined that the image is not used even when there are many motion vectors with low reliability among the motion vectors input from the motion vector extraction unit 110. The frequency detection method of the image can be determined by using, for example, FFT, that the sum of high frequency components higher than a certain frequency is lower than a certain threshold. As a simpler method for discriminating an image with a lot of low frequencies, the sum of differences in pixel values from the vicinity of a certain representative point can be used. This method will be described with reference to the flowchart of FIG.

  In the flowchart of FIG. 9, in step S901, variables are initialized. In step S902, an arbitrary number of representative points are determined from the image. This may be the same as the point where the motion vector is obtained. In step S903, one of the representative points selected in step S902 is selected, and the pixel value of the representative point is input to the variable I. In step S904, one point is selected from the points near the representative point selected in step S903, and the pixel value of that point is input to the variable R. In step S905, the absolute value of the difference between variable I and variable R is obtained and input to variable D. In step S906, the variable D is added to the variable S. In step S907, it is determined whether the process has been completed for all points near the representative point. If not, the process returns to step S904. In step S908, it is determined whether all the representative points selected in step S902 have been completed. If not, the process returns to step S903. In step S909, the variable S is compared with the threshold value Th. If the variable S is larger than the threshold value Th, it is determined that the image is normal in step S910. On the contrary, if the value of the variable S is smaller than the threshold value Th, it is determined that there are many low frequencies, and an image having many low frequencies is determined in step S911. Then, it is determined that the image is not used.

  Further, the reliability of the motion vector input from the motion vector extraction unit 110 is investigated. As described in the first embodiment, the parameter extraction unit 120 does not use a motion vector with low reliability so as to improve the accuracy of the parameter, but the image determination unit 815 has low reliability. When the ratio of motion vectors is large, for example, in the case of motion vectors whose reliability is low at 70% or more, the image is determined as an image in which the motion vector is difficult to detect. Then, it is determined that the image is not used.

  If the image discriminating unit 815 determines that an image is to be used, the motion vector is output to the parameter extracting unit 120, and the parameter extracting unit 120 calculates a rotation parameter between temporally continuous images from the input motion vector. calculate. Since this calculation is the same as that of the first embodiment, description thereof is omitted. The calculated rotation parameter is output to the ideal parameter calculation means 840. Then, as in the first embodiment, the ideal parameter is calculated by the ideal parameter calculator 840 and is output to the image converter 850. The image conversion means 850 calculates the conversion coordinates in the same manner as in the first embodiment and outputs it to the image memory control means 801. The image memory control unit 801 reads a new image held in the image memory 100 based on the information input from the image conversion unit 850. When the next image arrives, the new image is written into the area of the old image, and the new image so far becomes the old image. The interpolation unit 160 interpolates and outputs the converted image read from the image memory 100. The output interpolation image is recorded by the recording unit 170, displayed by the display unit 180, or transmitted by the transmission unit 190. Or sent to the Internet.

  On the other hand, the case where the image determining unit 715 determines that the image is not used such as a blurred image will be described using the flowchart shown in FIG. If it is determined that the input image is not used, the result is output to the ideal parameter calculation means 840, the image conversion means 850, and the image memory control means 801. This is step S1001 in FIG. Subsequently, the ideal parameter calculation means 840 obtains a predicted ideal parameter and outputs it to the image conversion means 850 in the same manner as when parameters are not used in the first embodiment. This is step S1002. Then, the image conversion unit 850 calculates a correction amount using the predicted ideal parameter and the previous parameter, calculates coordinates for the converted image, and outputs the calculated coordinate to the image memory control unit 801. This is step S1003. Then, the image memory control means 801 reads the old image based on the information from the image conversion means 850, and if the next image comes, the next image is held in the area where the new image is held, and the new image and To do. This is step S1006. The interpolation unit 160 interpolates and outputs the converted image read from the image memory 100. The output interpolation image is recorded by the recording unit 170, displayed by the display unit 180, or transmitted by the transmission unit 190. Or sent to the Internet.

  As described above, the present embodiment has an effect that the accuracy of image shake correction can be improved by discriminating an input image whose parameter estimation is difficult.

  It should be noted that the moving image processing shown in the first and second embodiments is used in a camera-integrated recording apparatus to record an image processed so as not to shake the captured image, and the captured image Used in a device that plays back recorded images, plays back images that have been processed so as not to shake the image, and is used in a system that remotely controls a moving means such as a robot via a network. Sent to the network using a system that remotely processed the moving means such as a robot through the network. , Displaying an image obtained by processing a moving image photographed by the moving means so as not to be shaken to an operator, and making a moving image processing program, and inputting an image The image input by the means can be executed by the processor with the program, and the image processed so as not to shake the image can be recorded or displayed or transmitted to the network.

The figure which shows the structure of Example 1. Flow chart showing parameter discrimination operation Diagram showing rotation angle and ideal curve Diagram showing update of ideal curve Diagram showing predicted ideal parameters Diagram showing correction amount Flow chart showing operation when parameter accuracy is low The figure which shows the structure of Example 2. Flow chart showing operation of discrimination of low frequency image Flow chart showing the operation when the image is determined to be a low frequency image

Explanation of symbols

DESCRIPTION OF SYMBOLS 100 Image memory 101,801 Image memory control means 110 Motion vector extraction means 120 Parameter extraction means 130 Parameter discrimination means 140,840 Ideal parameter calculation means 150,850 Image conversion means 160 Interpolation means 170 Recording means 180 Display means 190 Transmission means 815 Image Discriminating means

Claims (17)

Parameter extraction means for obtaining a parameter value at each time of an input moving image, and ideal parameter calculation means for statistically processing the parameter value at each time obtained by the parameter extraction means and calculating an ideal parameter value at each time A correction amount is calculated based on the image memory that holds the old and new images, the actual parameter value obtained by the parameter extraction unit and the ideal parameter value calculated by the ideal parameter calculation unit, and based on the correction amount In the moving image processing apparatus comprising image conversion means for converting the image held in the image memory and determination means for determining use / non-use of the input signal,
A moving image processing apparatus according to claim 1, wherein when the determining means determines that the input signal is not used, the old image stored in the image memory is converted and output.   When the determination means determines that the input signal is not used and converts the old image held in the image memory, the ideal parameter calculation means predicts an ideal parameter to obtain a predicted ideal parameter, and the predicted ideal parameter The moving image processing apparatus according to claim 1, wherein the correction amount is calculated from a parameter obtained with respect to the old image.   The input signal of the discrimination means is a parameter obtained by a parameter extraction means, the accuracy of the parameter is investigated by the discrimination means, and when the accuracy is low, it is determined that the parameter is not used. The moving image processing apparatus according to claim 1 or 2.   The moving image processing apparatus according to claim 3, wherein the parameter extracted by the parameter extraction unit is a rotation amount of the imaging apparatus.   The input signal of the discriminating means is an image signal, and when the discriminating means investigates the amount of the low frequency component when investigating the frequency component of the image or the reliability of the motion vector obtained for the image The moving image processing apparatus according to claim 1, wherein the image signal is determined not to be used based on a ratio of a motion vector having a low reliability.   6. A camera-integrated image recording apparatus, wherein the moving image processing according to claim 1 is performed on a captured moving image, and the moving image subjected to the moving image processing is recorded.   6. A moving image reproducing apparatus that performs moving image processing according to any one of claims 1 to 5 on a captured or recorded moving image, and reproduces the moving image subjected to the moving image processing.   The moving image according to any one of claims 1 to 5, for a moving image captured by an imaging device attached to the moving means in a remote operation system for remotely operating the moving means such as a robot via a network. A remote operation system that performs processing, transmits a moving image subjected to moving image processing to a network, and displays the processed moving image via a network to an operator who performs remote operation.   In a remote control system that remotely controls a moving means such as a robot via a network, a moving image shot by an imaging device attached to the moving means is transmitted to the network, and is shot by the moving means obtained via the network. 6. A remote operation system that performs the moving image processing according to claim 1 on a moving image and displays the processed moving image to an operator who performs a remote operation.   A parameter extraction step for obtaining a parameter value at each time of the moving image; an ideal parameter calculation step for statistically processing the parameter value at each time obtained in the parameter extraction step and calculating an ideal parameter value at each time; An image for calculating a correction amount based on the actual parameter value obtained in the parameter extraction step and the ideal parameter value calculated in the ideal parameter calculation step, and converting an image held in the image memory based on the correction amount In the moving image processing method comprising the conversion step and the determination step for determining whether the input signal is used or not, when the determination step determines that the input signal is not used, the old image held in the image memory is A moving image processing method characterized by converting and outputting.   When the determination step determines that the input signal is not used and converts the old image held in the image memory, the ideal parameter calculation step predicts the ideal parameter to obtain the predicted ideal parameter, and the predicted ideal parameter The moving image processing method according to claim 10, wherein the correction amount is calculated from a parameter obtained with respect to the old image.   The input signal of the determination step is a parameter obtained in the parameter extraction step, the accuracy of the parameter is investigated in the determination step, and when the accuracy is low, it is determined that the parameter is not used. The moving image processing method according to claim 10 or 11.   The moving image processing method according to claim 12, wherein the parameter extracted in the parameter extraction step is a rotation amount of the imaging apparatus.   The input signal in the determination step is an image signal, and in the determination step, the amount of the low frequency component when investigating the frequency component of the image or the reliability of the motion vector obtained for the image is investigated. The moving image processing method according to claim 10 or 11, wherein the image signal is determined not to be used based on a ratio of a motion vector having a low reliability.   A moving image processing program for causing a computer to execute the moving image processing method according to claim 10, wherein the moving image processing program is readable by the computer.   A storage medium storing the moving image processing program according to claim 15 in a form readable by a computer.   An image input unit, an image memory that holds old and new images, the moving image processing program according to claim 15, a processor that executes the moving image processing program, and a storage unit that stores the moving image processing program Recording means for recording a moving image processed by the moving image processing program, display means for displaying a moving image processed by the moving image processing program, or a network for processing a moving image processed by the moving image processing program A moving image processing system comprising: a transmitting means for transmitting to a moving image.

Images