JP2012080411A - Imaging apparatus and control method therefor - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a technique capable of achieving both reduction of a blur in imaging and suppression of a random feeling according to the quality of a motion of an imaging apparatus.SOLUTION: An imaging apparatus comprises: an imaging unit that images an object and outputs a first video signal not containing a blur of the object and a second video signal containing a blur of the object; a motion detection unit that detects motion of the imaging apparatus itself in imaging; and an output control unit that switches between the first video signal and the second video signal or combines them according to the quality of the detected motion, thereby generating an output video signal. The output control unit outputs the first video signal in a case where the velocity or acceleration of the motion is constant, and outputs the second video signal in a case where the velocity or acceleration of the motion is not constant. Or, in the case where the velocity or acceleration of the motion is not constant, a weight of the second video signal is increased than in the case where the velocity or acceleration of the motion is constant.

Description

  The present invention relates to an imaging apparatus and a control method thereof.

  If the imaging device (video camera) itself moves during imaging, motion blur (also referred to as motion blur or motion blur) may occur on an object in the video. It is known that such blur (blur) can be improved by shortening the imaging time of the image sensor (faster shutter speed). Patent Document 1 discloses that when motion of the video camera itself is detected, an image with less blur is obtained by increasing the shutter speed until the aperture is opened or close to the open position.

JP-A-4-246833

As described above, by shortening the imaging time, a clear video signal with less motion blur can be obtained.
However, it has been found by the inventors that the following problems occur when such a video signal is displayed on an impulse display. When the movement of the imaging device is constant speed or constant acceleration, the target object in the video also moves at approximately constant speed or constant acceleration. With such a video, it is easy for the observer to follow the target in the display video, so that a high-quality video display is possible. On the other hand, when the movement of the imaging device is random, the movement (direction / speed) of the object in the video is also random, making it difficult to follow. When such difficult movements of follow-up are displayed with less motion blur, the continuity of the movement of the object cannot be felt visually, and the object appears to appear and disappear at random positions. There is. This sense of interference is referred to as randomness in this specification. In the conventional method, such a random feeling cannot be avoided.

  The present invention has been made in view of the above circumstances, and an object thereof is to further improve the display quality when a moving image is displayed. More specifically, an object of the present invention is to provide a technique capable of achieving both reduction of blur at the time of imaging and suppression of randomness according to the quality of motion of the imaging device.

  According to a first aspect of the present invention, an imaging unit that captures an image of an object and outputs a first video signal that does not include the blur of the object and a second video signal that includes the blur of the object; A motion detection unit that detects the motion of the imaging device itself, and generates an output video signal by switching between the first video signal and the second video signal according to the quality of the motion detected by the motion detection unit An output control unit configured to output the first video signal when the detected motion is a constant velocity or a constant acceleration motion, and the detected motion is a constant velocity. Alternatively, there is provided an imaging device that outputs the second video signal when the motion is not at constant acceleration.

According to a second aspect of the present invention, there is provided an imaging unit that captures an image of an object and outputs a first video signal that does not include blur of the object and a second video signal that includes blur of the object; An output video signal obtained by weighting and combining the first video signal and the second video signal in accordance with the motion quality detected by the motion detection unit and a motion detection unit that detects the motion of the imaging device itself An output control unit that generates the output control unit when the detected motion is not a constant velocity or a constant acceleration motion, compared to a case where the detected motion is a constant velocity or a constant acceleration motion. An image pickup apparatus that increases the weight of the video signal 2 is provided.

  According to a third aspect of the present invention, there is provided a camera capable of changing an imaging time, a motion detection unit that detects a motion of the imaging device itself at the time of imaging, and imaging when the detected motion is a uniform velocity or a uniform acceleration. There is provided an imaging apparatus including a control unit that controls the imaging time of the camera so that the imaging time becomes long when the time is short and the detected motion is not a constant velocity or uniform acceleration.

  According to a fourth aspect of the present invention, an object is imaged, a first video signal that does not include blur of the object and a second video signal that includes blur of the object are acquired, and imaging at the time of imaging Detecting the movement of the device itself with an acceleration sensor, and generating an output video signal by switching between the first video signal and the second video signal in accordance with the quality of the detected movement. And the first video signal is output when the detected motion is a constant velocity or constant acceleration motion, and the second video signal is output when the detected motion is not a constant velocity or constant acceleration motion. Provided is a method for controlling an imaging apparatus that outputs a video signal.

  According to a fifth aspect of the present invention, an object is imaged, a first video signal that does not include blur of the object and a second video signal that includes blur of the object are acquired, and imaging at the time of imaging Detecting the movement of the device itself with an acceleration sensor, and generating an output video signal by weighting and combining the first video signal and the second video signal according to the detected motion quality; When the detected motion is not equal velocity or equal acceleration motion, the control of the imaging device increases the weight of the second video signal compared to the case where the detected motion is equal velocity or equal acceleration motion. Provide a method.

  According to a sixth aspect of the present invention, there is provided a method for controlling an image pickup apparatus having a camera capable of changing an image pickup time, the step of detecting the movement of the image pickup apparatus itself at the time of image pickup by an acceleration sensor, and the detected movement being a constant velocity. Or the step of controlling the imaging time of the camera so that the imaging time is short if the motion is equal acceleration, and the imaging time is longer if the detected motion is not a constant velocity or constant acceleration motion. A method for controlling an imaging apparatus is provided.

  According to the present invention, it is possible to further improve the display quality when displaying a moving image. Further, according to the present invention, it is possible to achieve both reduction of blur during imaging and suppression of randomness according to the quality of motion of the imaging device.

1 is a block diagram of an imaging apparatus according to a first embodiment of the present invention. The block diagram of the imaging device of the 2nd Embodiment of this invention. The block diagram of the imaging device of the 3rd Embodiment of this invention. The block diagram of the imaging device of the 4th Embodiment of this invention. The figure for demonstrating the coordinate system of an imaging device, and the blurring at the time of imaging. The figure for demonstrating the relationship between the motion of an imaging device, and the motion of a target object. The figure for demonstrating the evaluation of a uniform velocity motion and the evaluation of a uniform acceleration motion. The figure which shows the structure of the imaging time control part which evaluates equal speed. The graph which shows an example of the weight of the video signals V1 and V2 with respect to the shift coefficient K. The figure which shows the structure of the imaging time control part which evaluates equal acceleration. The graph which shows an example of the weight of the video signals V1 and V2 with respect to the shift coefficient L. The graph which shows an example of the imaging time with respect to the deviation coefficient K. 6 is a graph showing an example of an imaging time with respect to a deviation coefficient L. The figure which shows the example by which two types of high-speed and low-speed image sensors were mounted in 1 chip | tip. The figure which shows the structure of the imaging part using a double shutter camera. The timing diagram of the structure using a double shutter camera.

The present invention relates to a technique capable of reducing blurring at the time of imaging when displaying a video signal using an impulse display and suppressing occurrence of randomness in a video portion that is difficult to follow. The present invention can be preferably applied to, for example, a portable imaging device such as a home video camera and a video system including such an imaging device and a display device. The impulse display is typically a CRT, a field emission display (FED) that performs line sequential driving, a surface-conduction electron-emitter display (SED), or the like.
However, in the present invention, an LCD (liquid crystal display) that improves hold blur (motion blur) by inserting a black frame or shortening the light emission time of the backlight is also included in the category of the impulse display.

(Bokeh during imaging)
First, blur at the time of imaging will be described. The blur at the time of imaging occurs when an object that is a subject moves within the imaging time of the imaging device, and is also called motion blur. Similarly, even when the object is stationary, the imaging device (video camera device) itself has moved, so that when the object moves relatively with respect to the viewing angle of the imaging device, blurring at the time of imaging also occurs. appear. The present invention provides a method for satisfactorily improving blur at the time of imaging due to the movement of the latter imaging apparatus.
In order to reduce blur at the time of imaging, there is a method in which the speed of the electronic shutter of the imaging device is controlled and an object is imaged in an imaging time shorter than one frame time. For example, when an object is imaged with an imaging time of 1/1000 second, each frame of the video signal becomes a sharp image without blurring.

  Blur (motion blur) at the time of imaging blurs the outline of the subject and reduces the contrast of color and brightness. Therefore, when comparing the spatial frequency components of multiple types of images obtained by imaging the same object at different imaging times (shutter speeds), a video signal with a shorter imaging time has more high frequency components, and a video signal with a longer imaging time has higher frequency components. Less. Therefore, a video signal with a short imaging time can be called a “video signal with a high frequency component”, and a video signal with a long imaging time can be called a “video signal with a low high frequency component”. When attention is paid to changes in the pixel value (or luminance or color) in the time axis direction, a video signal with a short imaging time is referred to as a “video signal with a good time response”, and a video signal with a long imaging time is referred to as a “time response”. It can also be called “bad video signal”. In addition, when focusing on the presence or absence of motion blur of the object in the video, the video signal with a short imaging time is referred to as “video signal that does not include blurring of the object”, and the video signal with a long imaging time is referred to as “ It can also be referred to as a “video signal including”.

(Motion of the imaging device and blur at the time of imaging)
Next, the movement of the imaging device itself and blurring during imaging will be described.
FIG. 5A is a diagram schematically illustrating the coordinates of the imaging device and the object that is the subject for explaining the movement of the imaging device itself. In FIG. 5A, reference numeral 90 denotes a subject, and reference numeral 100 denotes a coordinate axis of the imaging apparatus.
The movement of the image pickup apparatus at the time of image pickup includes a parallel movement in the X, Y, and Z directions shown in FIG. 5A and a movement in the rotation direction with the X, Y, and Z axes as rotation axes. Here, the Z axis is taken in the optical axis direction of the image pickup apparatus, and the Y axis is taken in the vertical direction of the image pickup apparatus and the X axis is taken in the horizontal direction in a plane perpendicular to the Z axis.

When the imaging device moves in parallel in the X direction or rotates around the Y axis, an image in which the object moves in the X direction is obtained, and when the imaging device moves in parallel in the Y direction or rotates around the X axis. An image in which the object moves in the Y direction is obtained. However, the moving direction of the object is opposite to the moving direction of the imaging device.
When the imaging apparatus moves in parallel in the Z direction, an image is obtained in which the object is out of focus or the size of the object in the image changes. Further, when the imaging device rotates around the Z axis, an image in which the object rotates and moves around the Z axis in the reverse direction is obtained.
Therefore, when the imaging time of the imaging device is long, motion blur (blurring) in the direction opposite to the motion of the imaging device occurs on the target in the video.

  FIGS. 5B and 5C schematically show examples of video signals picked up by rotating the image pickup apparatus around the Y axis. FIG. 5B is an example of a video signal with a short imaging time, 101 is an image of one frame, and 90a indicates an object. When the imaging time is short, there is almost no blur of the object 90a. FIG. 5C is an example of a video signal having a long imaging time, 101 is an image of one frame, and 90b indicates an object. When the imaging time is long, blurring in the Y direction occurs on the object 90b due to the movement of the imaging device.

(Hold blur)
Next, hold blur will be described. Hold blur occurs when a moving object on the screen is tracked. Tracking vision refers to observing a moving object while keeping the line of sight following the movement of the object. While the blur at the time of imaging described above is included in the video signal, the hold blur occurs due to the visual action of the observer.
Here, the moving object means an object whose position in the video (on the screen) is changing, and it is irrelevant whether or not the object is actually moving. As described above, even when the object is stationary, an image of the moving object can be obtained if the object and the field of view of the image capturing apparatus move relative to each other by moving the image capturing apparatus.

In impulse-type displays such as CRT, line-sequential FED and SED (Surface-conduction Electron-emitter Display), the display time (light emission time) in each frame (or field) is very short. Therefore, no blur occurs even when the moving object is viewed.
On the other hand, in a hold-type display such as an LCD, the light emission intensity is maintained for one frame. Therefore, when the moving object is viewed, the object spreads in the moving direction and is imaged on the retina. The This is observed as a hold blur. The hold blur always occurs in the hold-type display when the moving object is tracked. In order to avoid this hold blur, it is preferable to display the object like a impulse display by controlling the backlight emission time to be short when displaying a moving object on the hold display. Recently, LCDs are approaching the characteristics of impulse-type displays due to techniques for shortening the backlight emission time of LCDs and techniques for inserting black frames between video frames. The image processing method of the present invention is also effective for an LCD that performs such control. In this specification, such an LCD is also considered as a category of an impulse display.

(A sense of interference with an impulse display)
Next, a feeling of interference when displaying on an impulse display will be described.
As described above, the impulse display does not cause a sense of hindrance such as hold blur. However, in the impulse-type display, when an image without blur at the time of imaging is displayed, a sense of interference occurs under certain conditions.
That is, when an object that is not blurred at the time of imaging is observed on an impulse display, the object that can be followed can be clearly seen without occurrence of hold blur. On the other hand, in the case of an object whose movement is difficult to follow, the movement of the observer's line of sight and the movement of the object on the video are shifted, and the position on the retina is shifted.
In fact, when an image of an object that performs such a movement is displayed on an impulse display, the observer cannot visually perceive the continuity of the movement of the object, and the object appears and disappears at random positions. Looks unnatural. This disturbing feeling is more unnatural than the blur at the time of imaging, and the sense of discomfort is great.
As an example of an image that is difficult to follow, for example, an image captured by a photographer holding an image pickup device by hand and walking or running. In this case, in addition to the movement intended by the photographer, fine vibrations and random movement are added to the imaging apparatus. The vibration and random movement of the imaging apparatus itself appear as vibration (in the reverse direction) and random movement of the entire image screen. Since such a movement is difficult to follow, the observer cannot feel the continuity for each frame, and a sense of disturbance called random feeling occurs.

  As a result of studies by the present inventors, it has been found that when a random motion is applied to the imaging apparatus, it is possible to remove the random feeling by setting the imaging time to be long and intentionally adding the blur at the time of imaging. It has also been found that randomness can be reduced in the same manner by adding blur corresponding to blur at the time of imaging to a video signal captured with a short imaging time using an electronic shutter. Note that the blur at the time of imaging appears along the moving direction of the object, so that there is less unnaturalness than the random feeling.

(Motion of the imaging device and motion of the imaged object)
The movement of the imaging apparatus and the movement of the imaged object by the movement will be described.
As described above, the motion of the object imaged in the opposite direction to the motion of the imaging device is reversed.

First, the parallel movement of the imaging apparatus in the X direction will be described. Due to the parallel movement of the imaging device in the X direction, the object on the image moves in the opposite direction of the X direction. The same applies to the Y direction.
FIG. 6A shows the parallel movement of the imaging apparatus in the X direction. In FIG. 6A, 110 is a straight line indicating the X direction of the imaging apparatus, 111 is a plane on which an object is placed, and is a plane perpendicular to the Z direction of the imaging apparatus. The imaging device and the object are separated by a distance S.
112a is a dotted line indicating the angle of view of the imaging device at a certain time, 112b is a one-dot chain line indicating the angle of view of the imaging device after elapse of one frame time, for example, and the angle of view is θ. Further, the speed in the X direction of the imaging apparatus at this time is assumed to be vx. This speed vx is defined by, for example, a distance traveled by the imaging apparatus in the X direction in one frame time.
At this time, the speed Vx of the imaged object is considered by normalizing the length of the display screen in the X direction as 1.
Vx = -vx / [2 ・ S ・ tan (θ / 2)] ・ ・ ・ ・ Equation 1)
It becomes.
Similarly, with respect to the acceleration in the X direction of the imaging apparatus, if the acceleration (change in speed per frame) of the imaging apparatus is ax, the acceleration Ax of the captured object is the length of the display screen in the X direction. Is normalized to 1, and
Ax = -ax / [2 ・ S ・ tan (θ / 2)] ・ ・ ・ ・ Equation 2)
It becomes.

Next, the relationship between the angular velocity around the Y axis of the imaging device and the velocity in the X direction of the imaged object will be described with reference to FIG. The rotation around the X axis can be considered in the same way.
In FIG. 6B, 110 is a straight line indicating the X direction of the imaging apparatus, 111 is a plane on which an object is placed, and is a plane perpendicular to the Z direction of the imaging apparatus. The imaging device and the object are separated by a distance S.
113a is a dotted line indicating the angle of view of the imaging device at a certain time, 113b is a one-dot chain line indicating the angle of view of the imaging device after elapse of one frame time, for example, and the angle of view is θ. The angular velocity around the Y axis at this time is represented by vωx. This angular velocity vωx is determined by the imaging device Y during one frame time.
Consider the angle rotated around the axis.
At this time, the velocity Vωx of the imaged object is considered by normalizing the length of the display screen in the X direction as 1.
Vωx = -vωx / θ (3)
It becomes.
Similarly, consider the relationship between the angular acceleration around the Y axis of the imaging device and the acceleration of the imaged object. Assuming that the angular acceleration around the Y axis is aωx, the acceleration Aωx of the imaged object is considered by normalizing the length of the display screen in the X direction as 1.
Aωx = -aωx / θ (4)
It becomes.

  As described above, the parallel movement (velocity and acceleration) of the imaging apparatus in the X and Y directions and the movement in the rotational direction around the X and Y axes (angular velocity and angular acceleration) are related to the X of the captured object. , It is proportional to the parallel movement (speed, acceleration) in the Y direction.

  When the imaging apparatus moves in parallel in the Z direction, the object is out of focus or the size of the object changes. When the imaging device rotates about the Z axis, the object rotates in the direction opposite to the rotation direction of the imaging device. Since these movements are generally difficult to follow, it is not necessary to evaluate the quality (constant speed, constant acceleration) of these movements, and it is determined that they cannot be followed according to the presence (or size) of these movements. Good.

In general, when the distance S is large, the motion of the captured object is small with respect to the parallel motion of the imaging device in the X and Y directions. For this reason, parallel movements in the X and Y directions of the imaging apparatus described later have little influence on image quality without evaluation.
Further, with regard to the parallel movement of the image pickup apparatus in the Z direction, since the distance S is generally much larger than the distance that the photographer can move, the focus shift and the change in the size of the object are extremely large. small. Therefore, the parallel movement in the Z direction has little influence on the image quality without evaluation.
Therefore, when the process is simplified, it is not necessary to evaluate the parallel movement of the imaging apparatus in the X, Y, and Z directions. In other words, when it is necessary to evaluate parallel movement in the X, Y, and Z directions, it is a case where an object having a small distance S is imaged, that is, an image is picked up by an imaging device using a macro lens.

<First Embodiment>
The first embodiment of the present invention evaluates the quality of motion (constant velocity, uniform acceleration) of the imaging apparatus, and the length of the imaging time of the output video signal according to the evaluation result, that is, the ease of following vision. To change. Specifically, a first video signal with a short imaging time (video signal that does not include blur of the object) and a second video signal with a long imaging time (video signal that includes the blur of the object) are switched or combined as appropriate. Then, an output video signal is generated. By outputting such an output video signal to the display, a sharp image without blurring is realized for an object that is easy to follow, and randomness is suppressed for an object that is difficult to follow. Realize natural image quality.
In the following description, the simplified expression “the movement of the imaging device that is easy / difficult to follow” is used. This is because “the movement of the object in the video caused by the movement of the imaging device follows. It means “the movement of the imaging device that makes the movement easy / difficult to see”.

(Conditions for follow-up viewing)
Prior to the description of the first embodiment, conditions for following-viewing an object displayed on a display device will be described. Then, the condition of the quality of motion (speed, angular velocity, acceleration, angular acceleration) of the imaging device that can be followed is shown by the relational expression between the motion of the object and the motion of the imaging device.
When the inventor observed the movement of an object that can be followed, the human eye can follow the object moving at a constant speed such as a telop or the object moving at a constant acceleration. I understood.
For this reason, for an object that moves at a constant velocity or a constant acceleration, a video signal with a short imaging time (a video signal with good time responsiveness) is output, and an image without blurring at the time of imaging is displayed. This prevents blurring of the object that can be followed. Since it is difficult to follow the movement of other objects, switch to a video signal with a long imaging time or combine a video signal with a short imaging time and a long video signal so that randomness does not occur. Display video that includes (blur). As described above, the blur (motion blur) at the time of imaging is less unnatural than the random feeling, so that the overall display quality of the entire video is improved.

(Evaluation of constant speed)
First, an example in which the constant velocity of the captured object is evaluated will be described.
FIG. 7A shows a graph for explaining the evaluation of constant velocity motion. In FIG. 7A, the vertical axis represents time and the horizontal axis represents the X direction. T _n-2 , T _n−1 , T _n , and T _{n + 1} indicate the time for each frame. Although the horizontal axis will be described as the X direction, it is preferable to determine both the X and Y axes. In FIG. 7 (a) 401a, 401b, 401c, 401d each time _{T n-2, T n-} 1, T n, a line of sight that tracks the motion when _{T n + 1} shown schematically. Reference numerals 402a, 402b, 402c, and 402d are moving objects that are moving at approximately equal speed. The observer cannot adjust the line of sight to the movement of the object for each frame, and follows the average movement of the object and moves the line of sight at a constant speed. That is, a deviation (ΔX) from the line of sight 401c occurs for an object that deviates from the constant velocity, such as the object 402c. This shift is blurred on the retina. Due to this blur, a random feeling is generated in the impulse display.

  In this specification, the degree of blurring, that is, the ratio of how much the object is deviated with respect to the line of sight following at a constant speed is defined as “deviation coefficient: K”. If this deviation coefficient K is a small value, randomness is unlikely to occur, so that a video signal with a short imaging time can be displayed on the impulse display with good display quality.

と定義する。
ずれ係数Ｋは図７（ａ）に示したように、対象物の位置と視線の位置の差（ΔＸ）を１フレームあたりの視線の移動距離（Ｖａｖｅ）で割った値で定義する。 When the velocity of the object in the mth frame is Vm and the speed of the observer's line of sight is Vave, the deviation coefficient K in the nth frame, which is the current time, is

It is defined as
As shown in FIG. 7A, the deviation coefficient K is defined by a value obtained by dividing the difference (ΔX) between the position of the object and the position of the line of sight by the movement distance (Vave) of the line of sight per frame.

If the shift coefficient K is 0, for example, the position of the line of sight and the position of the object are not shifted, so that a video signal with a short imaging time is displayed to prevent blurring of the object that can be followed. In this case, there is no disturbing feeling such as a random feeling. On the other hand, when the deviation coefficient K is 0.5 or more, the object is displaced by a distance that is half of the moving distance in one frame period when the follow-up view is performed, and the random feeling starts to become noticeable.
Therefore, as the value of the deviation coefficient K increases, the video signal V1 with a short imaging time is switched to the video signal V2 with a long imaging time, or the synthesis rate (weight) of the video signal V2 with respect to the video signal V1 is changed. By increasing the value, the object is intentionally blurred at the time of imaging. This suppresses the generation of randomness in video signals that are difficult to follow.

と、変形できる。現時刻nより前までは追従視できている（すなわち、視線の位置と対象
物の位置がずれていない）と仮定する。式で示すと、

となる。
式６）に式７）を代入し
K=|Vn-Vave|/|Vave| ・・・・式８）
が求まる。
式５）あるいは、式８）により、ずれ係数Ｋを求め、追従視可能かを判断すると好適である。 In order to simplify the definition formula of the deviation coefficient K, it is also preferable to obtain the deviation coefficient by modifying the following expression. That is, Equation 5) is

And can be transformed. It is assumed that a follow-up view is possible before the current time n (that is, the position of the line of sight is not shifted from the position of the object). In terms of the formula:

It becomes.
Substituting Equation 7) into Equation 6)
K = | Vn-Vave | / | Vave |
Is obtained.
It is preferable that the deviation coefficient K is obtained by Equation 5) or Equation 8) to determine whether follow-up viewing is possible.

となる。
ここで、ｖｘａｖｅは撮像装置のＸ方向の速度の平均を示す。 Next, the deviation coefficient K is obtained from the velocity vx and angular velocity vωx of the imaging device. Needless to say, the Y direction can be similarly derived.
First, the relationship between the speed of the imaging apparatus and the deviation coefficient K is shown. Assuming that the speed of the imaging device in the mth frame is vxm, Equation 1) is substituted into Equation 5) and Equation 8),

It becomes.
Here, vxave indicates the average speed in the X direction of the imaging apparatus.

となる。
ここで、ｖωｘａｖｅは撮像装置のＹ軸まわりの角速度の平均を示す。
以上の式９）〜式１２）によって、撮像装置の動き（速度、角速度）からずれ係数Ｋを導出できる。 Next, the relationship between the angular velocity of the image pickup apparatus and the deviation coefficient K is shown. Assuming that the angular velocity vωxm of the imaging device in the m-th frame, Equation 3) is substituted into Equation 5) and Equation 8),

It becomes.
Here, vωxave represents an average of angular velocities around the Y axis of the imaging apparatus.
The deviation coefficient K can be derived from the movements (velocity, angular velocity) of the imaging apparatus by the above equations 9) to 12).

と、求めることができる。
式１３）、式１４）の開始時刻は、例えば、シーンが変わったときを基点として、過去から計算すればよい。 Next, a method for deriving the average vxave of the velocity in the X direction of the imaging device and the average vωxave of the angular velocity around the Y axis of the imaging device will be described.
Since the average velocity vxave and the average angular velocity vωxave are the average values of the velocity and angular velocity before the current time n,

And can be asked.
The start times of Equation 13) and Equation 14) may be calculated from the past, for example, when the scene changes.

When the imaging device is stationary, vxave and vωxave are 0, and the denominators of Equations 11) and 12) are 0. Since it can be estimated that follow-up vision is possible when the imaging apparatus is stationary, in this case, calculation of Equations 11) and 12) is not performed, and a small value (for example, K = 0) is output as the value of K. To do.
Since these calculations only need to be performed for each frame, they can be processed by software such as a microcontroller.

On the other hand, the average velocity vxave and the average angular velocity vωxave can be calculated by the cyclic filters shown in Equations 15) to 18), which are averages obtained by reducing the past weights. This calculation is not only simple, but also closely matches the actual observer's tracking.
Time n-th average speed Vxave _n when the frame and the average of the angular velocity and Buiomegaxave _n, equation for obtaining the Vxave _n,
vxave _n = S1 · vx _n-1 + S2 · vxave _n-1 ··· Equation 15)
However,
S1 + S2 = 1 ・ ・ ・ ・ Equation 16)
And
The equation for calculating vωxave _n is
vωxave _n = S1 · vωx _n-1 + S2 · vωxave _n-1 Equation 17)
However,
S1 + S2 = 1 ・ ・ ・ ・ Equation 18)
It becomes.
S1 and S2 can change the weight of the line-of-sight speed one frame before and the speed of the imaging apparatus one frame before. Usually, S1 and S2 should be set so that S2 is larger than S1.

  Furthermore, in order to obtain the average velocity vxave and the average angular velocity vωxave more easily, the average of the velocity vx and the average of the angular velocity vωx of a plurality of frames immediately before the current time may be used. For further simplification, the velocity vx and angular velocity vωx of the immediately preceding frame (one frame before) may be used as an average as they are. This calculation method is simpler, and the calculation has a great advantage that the amount of calculation can be reduced although the error is somewhat increased.

(Evaluation of equal acceleration)
Next, an example in which the equal acceleration of the imaged object is evaluated will be described.
FIG. 7B shows a graph for explaining the evaluation of the uniform acceleration motion.
In FIG. 7B, the vertical axis represents time and the horizontal axis represents the velocity in the X direction. T _n-2 , T _n−1 , T _n , and T _{n + 1} indicate the time for each frame. Although the horizontal axis will be described as the speed in the X direction, it is preferable to judge by the speed of both the X and Y axes. In FIG. 7 (b) 401a, 401b, 401c, 401d each time _{T n-2, T n-} 1, T n, a line of sight that tracks the motion when _{T n + 1} shown schematically. Reference numerals 402a, 402b, 402c, and 402d are moving objects, and are approximately moving at an equal acceleration. The observer cannot adjust the line of sight to the movement of the object for each frame, and follows the average movement of the object and moves the line of sight with equal acceleration. That is, for an object that deviates from the constant acceleration, such as the object 402c, a speed deviation (ΔV) from the line of sight 401c occurs. This speed difference (ΔV) is blurred on the retina. Due to this blur, a random feeling is generated in the impulse display.

  In this specification, the degree of blurring, that is, the ratio of how much the acceleration of the object is deviated from the line of sight following with constant acceleration is defined as “deviation coefficient: L”. If the deviation coefficient L is a small value, randomness is unlikely to occur, so that a video signal with a short imaging time can be displayed on the impulse display with good display quality.

When An is the acceleration of the object in the nth frame and Aave is the average acceleration of the object (that is, the average acceleration observed by the observer), the deviation coefficient L is defined as follows.
L = | An-Aave | / | Aave |
That is, the deviation coefficient L is a value obtained by dividing the difference between the acceleration of the object at the current time and the average acceleration of the observer's line of sight by the average acceleration of the observer's line of sight. If this ratio is 0, since the observer's line of sight and the movement of the object are the same, a video signal with a short imaging time is displayed to prevent blurring of the object that can be followed. In this case, there is no disturbing feeling such as a random feeling. On the other hand, when the deviation coefficient L is 0.5 or more, the distance and the object corresponding to the half of the speed changing in one frame period when following is viewed, the object is displaced, and the random feeling starts to become noticeable.
Therefore, as the deviation coefficient L increases, the video signal V1 with a short imaging time is switched to the video signal V2 with a long imaging time, or the synthesis rate (weight) of the video signal V2 with respect to the video signal V1 is changed. By increasing the value, the object is intentionally blurred at the time of imaging. This suppresses the generation of randomness in video signals that are difficult to follow.

Next, a method for obtaining the deviation coefficient L from the acceleration ax and angular acceleration aωx of the imaging apparatus will be described. Needless to say, the Y direction can be similarly derived.
First, the relationship between the acceleration of the imaging device and the deviation coefficient L is shown. Assuming that the acceleration of the imaging device in the m-th frame is ax _m , Equation 2) is substituted into Equation 19),
L = | ax _n -axave | / | axave |
It becomes.
Here, “axave” represents an average of accelerations in the X direction of the imaging apparatus.
Next, the relationship between the angular acceleration of the imaging device and the deviation coefficient L is shown. Assuming that the angular acceleration aωx _m of the imaging device in the mth frame, Equation 4) is substituted into Equation 19).
L = | aωx _n -aωxave | / | aωxave |
It becomes.
Here, aωxave represents the average angular acceleration around the Y axis of the imaging apparatus.

と、求めることができる。
式２２）、式２３）の開始時刻は、例えば、シーンが変わったときを基点として、過去
から計算すればよい。 Next, a method for deriving the average acceleration axave in the X direction of the imaging apparatus and the average angular acceleration aωxave around the Y axis of the imaging apparatus will be described.
The average acceleration axave and the average angular acceleration aωxave are average values of acceleration and angular acceleration before the current time n.

And can be asked.
The start times of Equation 22) and Equation 23) may be calculated from the past, for example, when the scene changes.

When the imaging device is moving at a constant speed (constant angular speed movement), axave and aωxave are 0, and the denominators of Expressions 22) and 23) are 0. Since it can be estimated that follow-up vision is possible when the imaging device is moving at a constant velocity or a constant angular velocity, in this case, the calculation of Equation 22) and Equation 23) is not performed, and a small value (for example, L) L = 0) is output.
Since these calculations only need to be performed for each frame, they can be processed by software such as a microcontroller.

On the other hand, the average acceleration axave and the average angular acceleration aωxave can be calculated by the cyclic filters shown in equations 24) to 27), which are averages obtained by reducing the past weights. This calculation is not only simple, but also closely matches the actual observer's tracking.
Axave _n an average of the acceleration at time n-th frame, when the average of the angular acceleration and Eiomegaxave _n, equation for obtaining the Axave _n,
axave _n = S1 · ax _n-1 + S2 · axave _n-1 ··· Equation 24)
However,
S1 + S2 = 1 ・ ・ ・ ・ Equation 25)
And
The equation for calculating aωxave _n is
aωxave _n = S1 · aωx _n-1 + S2 · aωxave _n-1 ··· Equation 26)
However,
S1 + S2 = 1 ・ ・ ・ ・ Formula 27)
It becomes.
By S1 and S2, it is possible to change the weight of the sight line acceleration (angular acceleration) one frame before and the acceleration (angular acceleration) of the imaging device one frame before. Usually, S1 and S2 should be set so that S2 is larger than S1.

  Furthermore, in order to obtain the average acceleration axave and the average angular acceleration aωxave more easily, the average of the accelerations ax and the average of the angular accelerations aωx immediately before the current time may be used. For further simplification, the acceleration ax and angular acceleration aωx of the immediately preceding frame (one frame before) may be used as an average as they are. This calculation method is simpler, and the calculation has a great advantage that the amount of calculation can be reduced although the error is somewhat increased.

  As described above, the deviation coefficient K and the deviation coefficient L, which are evaluation values indicating the ease of follow-up vision, can be obtained from the movement (speed, angular velocity, acceleration, angular acceleration) of the imaging device.

(Device configuration)
FIG. 1 shows a block diagram of the imaging apparatus according to the first embodiment of the present invention.
In FIG. 1, reference numeral 1 denotes an acceleration sensor as a motion detection unit that detects the motion of the imaging apparatus. An imaging time control unit 2 outputs a control signal CS1 for controlling the length of imaging time of the output video signal based on the output of the acceleration sensor 1. Reference numeral 4 denotes a switch (output control unit) that switches or combines the video V1 having a short imaging time and the video V2 having a long imaging time in accordance with the control signal CS1. 91a is a high-speed shutter camera with a short imaging time, 91b is a low-speed shutter camera with a long imaging time (for example, 1/60 second of one frame time), and 92 is light from the subject 90 to the two cameras 91a and 91b. It is a half mirror that leads.

In the configuration of FIG. 1, the high-speed shutter camera 91 a and the low-speed shutter camera 91 b capture an image of a target object 90 that is a subject through a half mirror 92. The positions of the high-speed shutter camera 91a, the low-speed shutter camera 91b, and the half mirror 92 are adjusted so that the images of the high-speed shutter camera 91a and the low-speed shutter camera 91b completely overlap. In the first embodiment, the imaging unit is configured by two cameras 91a and 9b and a half mirror.
The images of the high-speed shutter camera 91a and the low-speed shutter camera 91b are synchronized, and two images with different imaging times (shutter speeds) can be obtained at the same timing. Video outputs of the high-speed shutter camera 91a and the low-speed shutter camera 91b that image the object 90 are input to the input terminal of the switch 4 as video signals V1 and V2, respectively.

  The acceleration sensor 1 is a three-axis acceleration sensor that detects accelerations in six directions (X, Y, Z directions, rotation directions with the X, Y, and Z axes as rotational axes) of the imaging apparatus, and further detects the detected acceleration. Is integrated to calculate the speed. Then, acceleration and speed are output with one frame time as a unit time. There are six output speeds: X-direction speed: vx, Y-direction speed: vy, Z-direction speed: vz, X-axis angular speed: vωy, Y-axis angular speed: vωx, and Z-axis angular speed: vωz. The output acceleration includes six types of X-direction acceleration: ax, Y-direction acceleration: ay, Z-direction acceleration: az, X-axis angular acceleration: aωy, Y-axis angular acceleration: aωx, and Z-axis angular acceleration: aωz. .

The imaging time control unit 2 outputs a control signal CS1 for switching between the video signal V1 from the high-speed shutter camera 91a and the video signal V2 from the low-speed shutter camera 91b based on the acceleration and speed that are output from the acceleration sensor 1.
The switch 4 switches or combines the video signals V1 and V2 according to the control signal CS1, and outputs an output video signal from the output terminal 10a. The output video signal is input to a display device such as an impulse display.

(Imaging time controller that evaluates constant speed)
FIG. 8 shows a configuration of the imaging time control unit 2 when the constant speed is evaluated.
In FIG. 8, reference numeral 201 denotes an input terminal to which speed and angular velocity are input from the acceleration sensor 1. 202 is a tracking visual velocity calculation unit, 203 is a K calculation unit for obtaining a deviation coefficient K, 204 is a conversion table, 205 is a minimum value selection unit, 206 is a low-pass filter, 207 is a z-speed determination unit, and 208 is an output terminal.

  The velocity input from the acceleration sensor 1 (X-direction velocity: vx, Y-direction velocity: vy, Z-direction velocity: vz, X-axis angular velocity: vωy, Y-axis angular velocity: vωx, Z-axis angular velocity: vωz) Input to the calculation unit 202, the K calculation unit 203, and the z speed determination unit 207. The tracking visual speed calculation unit 202 calculates the X-direction speed: vx, the Y-direction speed: vy, the X-axis angular speed: vωy, and the Y-axis angular speed: vωx according to Expression 13), Expression 14), Expression 15), Expression 17). Determine the speed of tracking vision. Further, as described above, it may be obtained from the average of the speeds of the imaging devices of a plurality of frames immediately before the current time. In the above formula, the X direction is described, but the Y direction can be similarly obtained.

Next, the K calculation unit 203 calculates each direction (X, Y direction) from the input speed at the current time and the tracking visual speed calculated by the tracking visual speed calculation unit 202 according to Formula 10) and Formula 12). , X and Y axis rotation directions).
The obtained deviation coefficient K in each direction is converted into the weight of the video signal V1 in the conversion table 204 by the characteristics shown in FIGS. 9A, 9B, and 9C, for example. As can be seen from the characteristics of FIGS. 9 (a), (b), and (c), if the deviation coefficient K is small, there is little deviation between the line of sight and the object, and tracking is easy. Select or dominate V1. On the other hand, when the deviation coefficient K becomes large and the tracking vision becomes difficult, the video signal V2 having a long imaging time is selected or made dominant.

The conversion table 204 in FIG. 9A regards the motion as equal speed when the deviation coefficient K is smaller than the threshold value 0.5, outputs the video signal V1, and the deviation coefficient K is greater than or equal to the threshold value 0.5. In this case, it is assumed that the movement is not performed at a constant speed, and the video signal V2 is output. The video signal V1
It is also preferable to change the synthesis rate (weight) of the video signals V1 and V2 instead of selectively switching between and V2. Also at this time, for example, when the deviation coefficient K is 0.5 or more (that is, when the movement is not at a constant speed), the video is compared with when the deviation coefficient is smaller than 0.5 (that is, when the movement is at a constant speed). If the weight of the signal V2 is increased, the same effect as described above can be obtained. Preferably, in the case of uniform motion, the weight of V1 is set larger than the weight of V2 (for example, the weight of V1 is 0.6 to 1.0 and the weight of V2 is 0.4 to 0.0. ) If the motion is not uniform, the weight of V2 is made larger than the weight of V1 (for example, the weight of V1 is 0.0 to 0.4 and the weight of V2 is 1.0 to 0.6). Good. Thereby, suppression of blurring (blur) and suppression of generation of random feeling can be controlled more reliably.

  Furthermore, it is preferable that the weight (combination rate) is continuously changed according to the magnitude of the deviation coefficient K so that the weight of V2 increases as the deviation coefficient increases. The conversion tables 204 in FIGS. 9B and 9C are such that the video signals V1 and V2 are weighted according to the magnitude of the deviation coefficient K (combination rate) so that the switching between the video signals V1 and V2 does not become unnatural. Is an example in which an output video signal is generated by weighted synthesis. 9B and 9C, the weight of V1 is changed in the range of 1.0 to 0.0. However, the upper limit of the weight is made smaller than 1.0, or the lower limit of the weight is set to 0. 0. It may be larger than zero. For example, the weight of V1 is changed in the range of 1.0 to 0.1 (the weight of V2 is 0.0 to 0.9), or the weight of V1 is changed to 0.9 to 0.0 (the weight of V2 is 0). 0.1 to 1.0), or the weight of V1 may be changed within the range of 0.9 to 0.1 (the weight of V2 is 0.1 to 0.9). The characteristics shown in FIGS. 9A to 9C are examples, and it is preferable to perform subjective evaluation to determine appropriate conversion characteristics.

In addition, it is good to provide the speed judgment part of the X and Y directions shown with the dotted line 209 in FIG. 8 independently for every direction.
In the above example, the case where the deviation coefficient K is smaller than 0.5 is evaluated as “equal speed movement”, and the case where the deviation coefficient K is 0.5 or more is evaluated as “not equal speed movement”. The threshold for determining whether or not the movement is at a constant speed is not limited to this. Further, it may be evaluated whether or not the movement is at a constant speed by an index other than the deviation coefficient K.

  The Z-direction velocity: vz and the Z-axis angular velocity: vωz, which are the outputs of the acceleration sensor 1, are input to the z-speed determining unit 207. The z speed determination unit 207 outputs the weight of the video signal V1 based on the magnitudes of the input vz and vωz. The movement in the Z direction and the movement around the Z axis are difficult to follow as described above. Therefore, as an example, when the magnitudes of vz and vωz exceed a predetermined threshold, 0 may be output as the weight of V1. Further, if the magnitudes of vz and vωz are equal to or smaller than the threshold values, the weight of V1 can be determined by the movement in the other direction (that is, the weight of V1 output from the conversion table 204 is used). A sufficiently large value (for example, 1) is output as the weight.

The weight of V1 in each direction (each direction and the rotation direction of each axis), which is output from the conversion table 204 and the z speed determination unit 207, is input to the minimum value selection unit 205. The minimum value selection unit 205 selects and outputs the minimum value from these. That is, if there is at least one direction that is difficult to follow in any direction (each direction, the rotation direction of each axis), it is difficult to follow the object in the video, and randomness may occur. Therefore, the weight of the video signal V1 is reduced.
The low-pass filter 206 performs a low-pass filter process in the time direction on the weight of V1 that is the output of the minimum value selection unit 205. The low-pass filter has an effect of moderating the temporal change in the weight of V1, that is, the change in the synthesis rate of the video signals V1 and V2, and reducing the uncomfortable feeling caused by switching the video signal.

The weight of V1 calculated by the imaging time control unit 2 as described above is input to the switch 4 as the control signal CS1. The switch 4 receives the control signal CS1 (V1 weight), subtracts the V1 weight from 1 to calculate the V2 weight, and multiplies the input video signals V1 and V2 by the V1 weight and the V2 weight, respectively. Further, the multiplication results are added and output.
Here, it is desirable that the input video signals V1 and V2 of the switch 4 have data proportional to the luminance. It is preferable to perform inverse gamma conversion on a video signal that has undergone gamma conversion to obtain data in proportion to the luminance, because the luminance of the video signal output from the switch 4 does not change.
The minimum value selection unit 205 selects the smallest value from the V1 weights in each direction, but the average of the V1 weights in each direction, the average of the weights of a predetermined number of V1s selected in order from the smallest, etc. It is also preferred to select another value. Further, the low-pass filter 206 may be omitted, and the output of the minimum value selection unit 205 may be directly input to the switch 4 as the control signal CS1.

(Imaging time controller that evaluates equal acceleration)
FIG. 10 shows a configuration of the imaging time control unit 2 in the case of evaluating uniform acceleration.
In FIG. 10, reference numeral 211 denotes an input terminal to which acceleration and angular acceleration are input from the acceleration sensor 1. 212 is a tracking visual acceleration calculation unit, 213 is an L calculation unit for obtaining a deviation coefficient L, 214 is a conversion table, 215 is a minimum value selection unit, 216 is a low-pass filter, 217 is a z acceleration determination unit, and 218 is an output terminal.

  The accelerations input from the acceleration sensor 1 (X direction acceleration: ax, Y direction acceleration: ay, Z direction acceleration: az, X axis angular acceleration: aωy, Y axis angular acceleration: aωx, Z axis angular acceleration: aωz) are: The tracking visual acceleration calculation unit 212, the L calculation unit 213, and the z acceleration determination unit 217 are input. The tracking visual acceleration calculation unit 212 calculates the X-direction acceleration: ax, the Y-direction acceleration: ay, the X-axis angular acceleration: aωy, and the Y-axis angular acceleration: aωx, from Equation 22), Equation 23), Equation 24), Equation 26. ) To find the acceleration of follow-up vision. Further, as described above, it may be obtained from the average of the accelerations of the imaging devices of a plurality of frames immediately before the current time. In the above formula, the X direction is described, but the Y direction can be similarly obtained.

Next, the L calculation unit 2013 calculates each direction (X, Y direction) from the input acceleration at the current time and the tracking visual acceleration calculated by the tracking visual acceleration calculation unit 212 according to Formula 20) and Formula 21). , X and Y axis rotation directions).
The obtained deviation coefficient L in each direction is converted into the weight of the video signal V1 in the conversion table 214 by the characteristics shown in FIGS. 11A, 11B, and 11C, for example. As can be seen from the characteristics of FIGS. 11 (a), (b), and (c), if the deviation coefficient L is small, there is little deviation between the line of sight and the object, and tracking is easy. Select or dominate V1. On the other hand, when the deviation coefficient L becomes large and the follow-up view becomes difficult, the video signal V2 having a long imaging time is selected or made dominant. 11A to 11C are the same as those described with reference to FIGS. 9A to 9C. The characteristics shown in FIGS. 11A to 11C are also examples, and it is preferable to perform subjective evaluation and determine appropriate conversion characteristics.
Here, the case where the deviation coefficient L is smaller than 0.5 may be evaluated as “equal acceleration movement”, and the case where the deviation coefficient L is 0.5 or more may be evaluated as “no constant acceleration movement”. However, the threshold for determining whether or not the movement is equal acceleration is not limited to this. Further, whether or not there is a constant acceleration movement may be evaluated by an index other than the deviation coefficient L.

The Z direction acceleration: az and the Z-axis angular acceleration: aωz, which are the outputs of the acceleration sensor 1, are input to the z acceleration determination unit 217. The z acceleration determination unit 217 outputs the weight of the video signal V1 based on the input az and aωz. The movement in the Z direction and the movement around the Z axis are difficult to follow as described above. Therefore, as an example, when the magnitudes of az and aωz exceed a predetermined threshold, 0 may be output as the weight of V1. If the magnitudes of az and aωz are equal to or smaller than the threshold values, a sufficiently large value (for example, 1) is output as the weight of V1 so that the weight of V1 can be determined by movement in other directions.

  The configurations and operations of the minimum value selection unit 215 and the low-pass filter 216 are the same as those of the minimum value selection unit 205 and the low-pass filter 206 in FIG.

  Although the method for evaluating the speed and the method for evaluating the acceleration have been described above, both the speed and the acceleration can be evaluated. In that case, the speed determination unit 209 and the z speed determination unit 207 in the X and Y directions shown in FIG. 8 and the acceleration determination unit 219 and the z acceleration determination unit 217 in the X and Y directions shown in FIG. And acceleration are evaluated independently, and the weight of V1 is calculated for each. Then, the control signal CS1 may be generated by selecting the minimum value from the obtained V1 weights.

In addition, it is preferable not to evaluate all of the 12 types of directions but to omit the evaluation of some directions and simplify the processing.
For example, as described above, the velocity and acceleration in the X-axis direction, the Y-axis direction, and the Z-axis direction: vx, vy, vz, ax, ay, and az, Since the movement is generally small, it may be omitted. Furthermore, since it is difficult for the photographer to move the imaging device at a constant acceleration and a constant angular acceleration, the possibility that such shooting is performed is extremely low. Therefore, the process may be simplified by omitting the evaluation of the equal acceleration and the equal angular acceleration. In addition, since it is rare for the photographer to intentionally perform the Z-axis rotational motion to capture an image, the evaluation of the speed and acceleration in the Z-axis rotational direction may be omitted to simplify the processing.
That is, at least the angular velocities in the X-axis direction and the Y-axis direction: vωx, vωy are evaluated, and by selecting a video signal with good time response and a video signal with poor time response, the effect of the present invention is achieved. Can be realized with minimal processing.
In the description of the first embodiment of the present invention, the description has been given with respect to the hardware configuration. However, as long as the function can be realized, the processing may be performed by software.

(Advantages of the first embodiment)
According to the imaging apparatus of the first embodiment, the ease of following-up viewing of an image obtained by imaging is estimated (determined) by evaluating the quality of motion (constant velocity, uniform acceleration) of the imaging apparatus itself. . If the movement is easy to follow, the video signal V1 with a short imaging time without motion blur is output. If the motion is not easy to follow, the video signal V2 with a long imaging time including motion blur is output. Or a combined signal of V1 and V2. By outputting such an output video signal to an impulse display, a sharp image without blur is achieved for images that are easy to follow, and randomness is suppressed for images that are difficult to follow. Natural image quality can be achieved.
In the present embodiment, the movement of the object in the video is estimated based on the movement of the imaging device detected by the acceleration sensor. This method can realize real-time processing more easily than a method of analyzing each frame image input from the camera to obtain the motion of the object (for example, motion vector analysis). Further, since an image analysis circuit and a frame memory are not required, there is an advantage that the apparatus configuration can be simplified and the cost can be reduced.

<Second Embodiment>
Next, a second embodiment of the present invention will be described. In the first embodiment, the video signals V1 and V2 are acquired by two cameras having different shutter speeds, and the video signals V1 and V2 are appropriately switched to generate an output video signal. On the other hand, in the second embodiment, the output generated in the first embodiment is controlled by controlling the shutter speed (imaging time) of the camera based on the evaluation result of the motion quality of the imaging device detected by the acceleration sensor. An imaging apparatus capable of directly acquiring a video equivalent to a video signal is shown.

FIG. 2 shows a block diagram of an imaging apparatus according to the second embodiment of the present invention.
In FIG. 2, 1 is an acceleration sensor that detects the movement of the imaging device, 3 is an imaging time calculation unit that calculates an appropriate imaging time based on the output of the acceleration sensor 1 and outputs a control signal CS2 that indicates the imaging time. Control unit). An output terminal 10c outputs a video signal that is an output of the imaging apparatus. Reference numeral 90 is a subject, and 91c is a variable shutter camera capable of changing an imaging time (shutter speed) by a control signal CS2. In the second embodiment, the variable shutter camera 91c corresponds to the imaging unit.

In the imaging apparatus according to the second embodiment, an example in which constant speed is evaluated as the quality of motion of the imaging apparatus will be described.
When evaluating the uniform speed, the configuration of the imaging time calculation unit 3 is the same as that of FIG. The difference from the first embodiment is that the optimum imaging time for the shift coefficient K is stored in the conversion table 204. Examples of optimum imaging time for the deviation coefficient K are shown in FIGS. 12 (a), 12 (b), and 12 (c). FIGS. 12A, 12 </ b> B, and 12 </ b> C are examples of conversion of the conversion table 204 that outputs a preferable imaging time for the shift coefficient K. FIG. The horizontal axis indicates the deviation coefficient K, and the vertical axis indicates the imaging time. The imaging time is standardized in the range of 0 to 1, and when it is 1, the maximum imaging time is obtained. Note that the characteristics shown in the figure are merely examples, and it is preferable to actually perform subjective evaluation to determine the optimum characteristics.

FIG. 12A is an example in which the length of the imaging time is simply switched depending on whether or not the deviation coefficient K is greater than or equal to a threshold value (0.5 in the illustrated example). That is, when the deviation coefficient K is smaller than the threshold value, it is easy to follow up. Therefore, the imaging time is shortened to capture an image with no blur (blur). Since it is difficult, blur is intentionally generated by extending the shooting time. Thereby, a video signal equivalent to the control of FIG. 9A in the first embodiment can be obtained.
FIGS. 12B and 12C are examples in which the imaging time is continuously changed according to the shift coefficient K in order to reduce the change in the image and the uncomfortable feeling when the imaging time is switched. By using such a table, it is possible to obtain a video signal equivalent to the control of FIGS. 9B and 9C in the first embodiment.

  In the imaging apparatus according to the second embodiment, when the constant acceleration is evaluated as the motion quality of the imaging apparatus, the configuration of the imaging time calculation unit 3 is the same as that illustrated in FIG. Then, the conversion table 214 of FIG. 10 is provided with a table having the characteristics as shown in FIGS. 13A, 13B, and 13C, and the imaging time of each block is controlled according to the deviation coefficient L. It is possible to obtain a video signal equivalent to the case where the uniform velocity evaluation is performed.

  The configurations and operations of the minimum value selection units 204 and 214, the low-pass filters 206 and 216, the z speed determination unit 207, and the z acceleration determination unit 217 are the same as those described in the first embodiment, and thus description thereof is omitted. .

  Also in the imaging apparatus of the second embodiment described above, a video signal that can realize high-quality display in an impulse display can be obtained as in the first embodiment. In addition, the second embodiment with one camera has a large cost merit and is a useful configuration as compared with the first embodiment in which two cameras are required and geometric optical adjustment is necessary. .

<Third Embodiment>
Next, a third embodiment of the present invention will be described. In the first embodiment, the video signal V1 having a high frequency component and the video signal V2 having a low high frequency component are acquired by two cameras having different shutter speeds. On the other hand, in the third embodiment, a video signal V1 having a high frequency component is generated by performing high-frequency emphasis processing on the video signal V2 obtained by the low-speed shutter camera. The point that the output video signal is generated by appropriately switching or synthesizing the video signals V1 and V2 depending on the quality of the motion of the imaging device is the same as that of the first embodiment.

FIG. 3 shows a block diagram of an imaging apparatus according to the third embodiment of the present invention.
The configuration different from that of the first embodiment (FIG. 1) is that a moving direction high-frequency emphasis filter 20 is provided instead of the high-speed shutter camera, and a frame delay unit 21 that delays the video signal of the low-speed shutter camera 91b. Is a point provided. Other than that, the configuration is the same as that of the first embodiment. In the third embodiment, the low-speed shutter camera 91b and the moving direction high-frequency emphasis filter 20 correspond to the imaging unit.

The motion direction high-frequency emphasis filter (blur reduction unit) 20 performs high-frequency emphasis in the motion direction on the video signal of the low-speed shutter camera 91b. As a result, blurring at the time of imaging included in the video signal of the low-speed shutter camera 91b is canceled, and a video signal having many high-frequency components equivalent to the video signal obtained by the high-speed shutter camera is obtained. For example, when the movement is fast (the amount of movement is large), the blur at the time of imaging becomes large, and therefore, a treatment for raising the high range from a low spatial frequency may be performed. The direction and amount of movement may be determined from the output of the acceleration sensor 1. The frame delay unit 21 delays the video signal output from the low-speed shutter camera 91b by the same time as the processing time of the motion direction high-frequency emphasis filter 20. Thus, the video signals V1 and V2 are input to the switch 4 at the same timing.
As described above, the imaging apparatus according to the third embodiment is different from the first embodiment in that a video signal obtained from a low-speed shutter camera is subjected to high-frequency emphasis as the video signal V1. Since the configurations and operations of the acceleration sensor 1, the imaging time control unit 2, and the switch 4 are the same as those in the first embodiment, the description thereof is omitted.

  Also in the imaging apparatus of the third embodiment described above, a video signal that can realize high-quality display in an impulse display can be obtained as in the first embodiment. In addition, the third embodiment with one camera, which requires two cameras and requires geometric optical adjustment, has a large cost merit and is a useful configuration. .

<Fourth Embodiment>
Next, the 4th Embodiment of this invention is shown. In the first embodiment, the video signal V1 having a high frequency component and the video signal V2 having a low high frequency component are acquired by two cameras having different shutter speeds. On the other hand, in the fourth embodiment, a low-pass filter is applied to the video signal V1 obtained by the high-speed shutter camera to generate a video signal V2 with a small high-frequency component. The point that the output video signal is generated by appropriately switching or synthesizing the video signals V1 and V2 depending on the quality of the motion of the imaging device is the same as that of the first embodiment.

FIG. 4 shows a block diagram of an imaging apparatus according to the fourth embodiment of the present invention.
The configuration different from the first embodiment (FIG. 1) is that a moving direction low-pass filter 30 is provided instead of the low-speed shutter camera, and a frame delay unit 31 that delays the video signal of the high-speed shutter camera 91a is provided. This is the point. The high-speed shutter camera 91a is a camera with a short imaging time, and has an imaging time (1/100 to 1/2000 seconds), for example, a fraction of one frame time or a few hundredths. Other than that, the configuration is the same as that of the first embodiment. In the fourth embodiment, the high-speed shutter camera 91a and the movement direction low-pass filter 30 correspond to the imaging unit.

The motion direction low-pass filter (blur addition unit) 30 performs a motion direction low-pass filter process on the video signal of the high-speed shutter camera 91a. As a result, blur such as blur at the time of imaging is added to the sharp video signal of the high-speed shutter camera 91a, and a video signal with less high-frequency components equivalent to the video signal obtained by the low-speed shutter camera is obtained. It is done. For example, when the movement is fast (the amount of movement is large), the blur at the time of imaging becomes large, and therefore, a treatment for reducing the high range from a low spatial frequency may be performed. The direction and amount of movement may be determined from the output of the acceleration sensor 1. The frame delay unit 31 delays the video signal output from the high-speed shutter camera 91a by the same time as the processing time of the motion direction low-pass filter 30. Thus, the video signals V1 and V2 are input to the switch 4 at the same timing.
As described above, the imaging apparatus according to the fourth embodiment is different from the first embodiment in that a video signal obtained from a high-speed shutter camera is subjected to low-pass filter processing and used as the video signal V2. Since the configurations and operations of the acceleration sensor 1, the imaging time control unit 2, and the switch 4 are the same as those in the first embodiment, the description thereof is omitted.

  Also in the imaging apparatus of the fourth embodiment described above, a video signal that can realize high-quality display in an impulse display can be obtained as in the first embodiment. In addition, the fourth embodiment with one camera, which requires two cameras and requires geometric optical adjustment, has a large cost merit and is a useful configuration. .

<Other embodiments>
The imaging unit configured by the high-speed shutter camera 91a, the low-speed shutter camera 91b, and the half mirror 92 illustrated in FIG. 1 may be preferably realized by a semiconductor imaging device configured by a CCD or a CMOS illustrated in FIG. FIG. 14 shows a semiconductor image sensor in which an image sensor that operates with a high-speed shutter and an image sensor that operates with a low-speed shutter are mounted on a chip of one semiconductor image sensor.

In FIG. 14, 900 is a line schematically showing the outer shape of the chip (die) of the semiconductor image sensor, and 901 is a broken line showing a part of one light receiving element of the semiconductor image sensor. A light receiving portion 902 is realized by a photodiode for a high-speed shutter of the semiconductor image sensor, and a light receiving portion 903 is realized by a photodiode for a low-speed shutter of the semiconductor image sensor. In addition, since the received video data transfer circuit, timing circuit, video data output circuit, and the like do not directly affect the present invention, they are omitted in FIG. In FIG. 14, the light received by the light receiving portion 902 is photoelectrically converted in a short imaging time. On the other hand, the light received by the light receiving portion 903 is photoelectrically converted in a long imaging time.
If the area ratio between the light receiving portion 902 and the light receiving portion 903 is designed to be proportional to the ratio of the reciprocal of the imaging time, it is preferable because the photoelectrically converted voltages are substantially the same. The shift between the two video signals may be performed by a gain adjustment circuit (not shown).

  This semiconductor image pickup device has an optical system similar to a normal semiconductor image pickup device, and a single light receiving element 901 can simultaneously obtain a video signal having a short imaging time and a video signal having a long imaging time. Since the optical system may be the same as that of a normal camera as compared to the first embodiment, two video signals having different imaging time lengths can be obtained simultaneously with a very simple configuration. By applying these video signals to the first embodiment, the imaging part can be simplified.

  Further, the image pickup unit including the high-speed shutter camera 91a, the low-speed shutter camera 91b, and the half mirror 92 illustrated in FIG. 1 may be preferably realized with the configuration illustrated in FIG. FIG. 15 shows a configuration using a double shutter camera that time-divides one frame into two subframes and sequentially outputs a video signal of a high-speed shutter and a low-speed shutter.

In FIG. 15, reference numeral 91g denotes a double shutter camera that divides one frame into two subfields (one is a video signal with a short imaging time and the other is a video signal with a long imaging time) and outputs these subfields in a time-sharing manner. is there. A serial-parallel conversion unit 600 receives the output VS1 of the double shutter camera 91g and outputs two signals of a video signal VS2 having a short imaging time and a video signal VS3 having a short imaging time at the same time. Reference numeral 601 denotes a frame memory, reference numeral 602 denotes a timing generation circuit, reference numeral 603 denotes an output terminal that outputs a video signal VS2 having a short imaging time, and reference numeral 604 denotes an output terminal that outputs a video signal VS3 having a long imaging time.

  FIG. 16 shows a timing chart of a configuration using the double shutter camera 91g. In the timing chart of FIG. 16, the horizontal axis is a time axis in units of frames. The operation will be described below with reference to FIGS. 15 and 16.

For example, the double shutter camera 91g outputs a video signal (SS _n ) having a short imaging time and a video signal (SL _n ) having a long imaging time at the n-th frame time in a time division manner. Although the signal levels of the video signal SS _n and the video signal SL _n are different because the imaging time is different, it is preferable to adjust the two video signals to the same level by a gain adjustment circuit (not shown).
The serial-parallel converter 600 writes a video signal (SS _n ) with a short imaging time and a video signal (SL _n ) with a long imaging time into the frame memory 601 at the n-th frame time.
Next, at the (n + 1) th frame time, the double shutter camera 91g outputs the video signal (SS _{n + 1} ) having a short imaging time and the video signal (SL _{n + 1} ) having a long imaging time in a time division manner. At the (n + 1) th frame time, the serial-parallel converter 600 writes a video signal (SS _{n + 1} ) with a short imaging time and a video signal (SL _{n + 1} ) with a long imaging time into the frame memory 601. At the same time, the serial-parallel converter 600 converts the video signal (SS _n ) with a short imaging time and the video signal (SL _n ) with a long imaging time written in the previous frame time into one frame time at the (n + 1) th frame time from the frame memory 601. Read in parallel. A video signal (SS _n ) with a short imaging time is output to the output terminal 603, and a video signal (SL _n ) with a long imaging time is output to the output terminal 604.
The timing generation circuit 602 generates timing necessary for these operations.

  In this way, two types of video signals, that is, a video signal with a short imaging time and a video signal with a long imaging time can be obtained from one camera. Since the configuration using the double shutter camera 91g may be the same optical system as that of a normal camera, both a short video signal and a long video signal can be obtained simultaneously with a single light receiving element with a very simple configuration. . By applying this configuration to the first embodiment, the configuration of the imaging part can be simplified.

In recent years, a so-called camera shake correction method has been realized by detecting the movement of the imaging apparatus using an acceleration sensor and moving the lens optical system or the imaging element itself in a direction that cancels the movement of the imaging apparatus. In addition, a method of correcting camera shake by capturing an image larger than a necessary imaging range and changing the clipping range according to the movement of the imaging device is realized. These methods are generally called camera shake correction.
When such camera shake correction is performed, the amount of motion of the object in the video is an amount obtained by subtracting the amount of correction due to camera shake correction from the amount of motion due to the motion of the imaging device itself. Therefore, when the present invention is applied to an imaging apparatus that performs camera shake correction, it is preferable to evaluate the quality of motion after camera shake correction.

Furthermore, the present inventor has shown that when the movement of the imaging device is very fast, even if the following conditions such as constant speed and constant acceleration are possible, the video signal with longer imaging time and blurring at the time of imaging is better. I found that there was little discomfort. That is, when the movement of the image pickup apparatus is too fast, it is no longer possible to follow the image, and it has been found that an image including blur (motion blur) at the time of image pickup looks more natural.
As described above, since it is difficult to realize such a fast movement in the movement of the imaging device in the X and Y directions, in reality, as such a fast movement, the X and Y of the imaging device shown in Expression 3) can be used. This is due to the speed of the object due to rotation around the axis.
For example, it is difficult to follow up with a movement that moves the display screen in the X direction from 1 second to 2 seconds or less. If the number of frames per second is N and the length of the display screen in the X direction is normalized to 1, tracking becomes difficult if Vx in Equation 3) exceeds 1 / N to 1 / 2N per frame. ,
It was preferable to output a video signal with a longer imaging time.
In other words, if 1 / N per frame is exceeded, for example, it is difficult to follow-up.
| vωx / θ | ≧ (1 / N) ・ ・ ・ ・ Equation 28)
In the first, third, and fourth embodiments, it is preferable to set the weight of V1 of the switch 4 to 0, and in the second embodiment, to increase the imaging time (one frame time).
The above-described conditions for the follow-up viewing are an example when a high-vision video signal is viewed at a standard viewing distance, and vary depending on the angle of view of the display device. For this reason, it is preferable to perform subjective evaluation according to the format to be imaged and determine the conditions for determination.

  1: acceleration sensor, 2: imaging time control unit, 3: imaging time calculation unit, 4: switch, 91a: high-speed shutter camera, 91b: low-speed shutter camera, 91c: variable shutter camera

Claims (15)

An imaging unit that images a target and outputs a first video signal that does not include blur of the target and a second video signal that includes blur of the target;
A motion detector that detects the motion of the imaging device itself during imaging;
An output control unit that generates an output video signal by switching between the first video signal and the second video signal according to the quality of motion detected by the motion detection unit;
Have
The output control unit outputs the first video signal when the detected motion is a constant velocity or a constant acceleration motion, and when the detected motion is not a constant velocity or a constant acceleration motion, An imaging apparatus that outputs a second video signal. When the optical axis direction of the imaging device is the Z axis, the vertical direction is the Y axis, and the horizontal direction is the X axis,
The motion detection unit detects an angular velocity around the X axis and an angular velocity around the Y axis,
The output control unit outputs the first video signal when the angular velocity around the X axis and the angular velocity around the Y axis are equal, and outputs at least one of the angular velocity around the X axis and the angular velocity around the Y axis. The imaging apparatus according to claim 1, wherein the second video signal is output when one of the speeds is not constant. The motion detection unit detects at least one of a velocity in the Z direction and an angular velocity around the Z axis;
When at least one of the speed in the Z direction and the angular velocity around the Z axis exceeds a threshold value, the output control unit does not depend on whether the angular velocity around the X axis and the angular velocity around the Y axis are equal. The imaging apparatus according to claim 2, wherein the second video signal is output. An imaging unit that images a target and outputs a first video signal that does not include blur of the target and a second video signal that includes blur of the target;
A motion detector that detects the motion of the imaging device itself during imaging;
An output control unit that generates an output video signal by weighting and combining the first video signal and the second video signal according to the quality of motion detected by the motion detection unit;
Have
The output control unit increases the weight of the second video signal when the detected motion is not constant velocity or constant acceleration compared to the case where the detected motion is constant velocity or constant acceleration. An imaging device. The output control unit increases the weight of the first video signal to be greater than the weight of the second video signal when the detected motion is a constant velocity or constant acceleration motion. 5. The imaging apparatus according to claim 4, wherein the weight of the second video signal is set to be larger than the weight of the first video signal when the motion is not at a constant velocity or a constant acceleration. The output control unit
Calculate the difference between the detected motion and the constant velocity or constant acceleration,
6. The image pickup apparatus according to claim 4, wherein the weight of the second video signal is increased as the deviation becomes larger. When the optical axis direction of the imaging device is the Z axis, the vertical direction is the Y axis, and the horizontal direction is the X axis,
The motion detection unit detects an angular velocity around the X axis and an angular velocity around the Y axis,
When the angular velocity around the X axis and the angular velocity around the Y axis are both equal, the output control unit makes the weight of the first video signal larger than the weight of the second video signal, 6. The weight of the second video signal is made larger than the weight of the first video signal when at least one of the angular velocity around and the angular velocity around the Y axis is not equal. The imaging device described. The motion detection unit detects at least one of a velocity in the Z direction and an angular velocity around the Z axis;
When at least one of the speed in the Z direction and the angular velocity around the Z axis exceeds a threshold value, the output control unit does not depend on whether the angular velocity around the X axis and the angular velocity around the Y axis are equal. The imaging apparatus according to claim 7, wherein a weight of the second video signal is larger than a weight of the first video signal. The imaging unit has two cameras with different imaging times,
The first video signal is a video signal captured by a camera with a shorter imaging time,
The imaging apparatus according to claim 1, wherein the second video signal is a video signal captured by a camera having a longer imaging time. The imaging unit includes a camera with an imaging time shorter than one frame time, and a blur adding unit that adds blur to a video signal,
The first video signal is a video signal captured by the camera;
The said 2nd video signal is a video signal by which blur was added by the said blur addition part with respect to the video signal imaged with the said camera, The any one of Claims 1-8 characterized by the above-mentioned. Imaging device. The imaging unit includes a camera having an imaging time equal to one frame time, and a blur reduction unit that reduces blurring of the video signal.
The first video signal is a video signal in which blur is reduced by the blur reduction unit with respect to a video signal captured by the camera,
The imaging apparatus according to claim 1, wherein the second video signal is a video signal captured by the camera. A camera that can change the imaging time;
A motion detector that detects the motion of the imaging device itself during imaging;
The imaging time of the camera is set so that the imaging time is short when the detected motion is a constant velocity or uniform acceleration motion, and the imaging time is long when the detected motion is not a uniform velocity or uniform acceleration motion. A control unit to control;
An imaging device comprising: Imaging a target object and obtaining a first video signal that does not include blur of the target object and a second video signal that includes blur of the target object;
Detecting the movement of the imaging device itself during imaging with an acceleration sensor;
Generating an output video signal by switching between the first video signal and the second video signal according to the quality of motion detected;
Have
The first video signal is output when the detected motion is a constant velocity or constant acceleration motion, and the second video signal is output when the detected motion is not a constant velocity or constant acceleration motion. A method for controlling an imaging apparatus, characterized by being output. Imaging a target object and obtaining a first video signal that does not include blur of the target object and a second video signal that includes blur of the target object;
Detecting the movement of the imaging device itself during imaging with an acceleration sensor;
Generating an output video signal by weighting and combining the first video signal and the second video signal according to the detected motion quality;
Have
When the detected motion is not constant velocity or uniform acceleration, the weight of the second video signal is increased as compared with the case where the detected motion is constant velocity or uniform acceleration. Method. A method for controlling an imaging apparatus having a camera capable of changing an imaging time,
Detecting the movement of the imaging device itself during imaging with an acceleration sensor;
The imaging time of the camera is set so that the imaging time is short when the detected motion is a constant velocity or uniform acceleration motion, and the imaging time is long when the detected motion is not a uniform velocity or uniform acceleration motion. Controlling step;
A method for controlling an imaging apparatus, comprising:

Images