JP2008103928A - Imaging apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an imaging apparatus capable of efficiently encoding a moving image in accordance with an exposure time. <P>SOLUTION: A digital camera 1 is provided with an imaging element 3, an exposure time adjusting part 8 for adjusting the exposure time of the imaging element 3, and an encoding part 6 for detecting a motion vector in an image obtained from the imaging element 3, performing motion compensation processing using the motion vector and encoding a moving image. A search range changing part 11 changes the width of a search range of the motion vector on the basis of the exposure time of the imaging element 3. For example, the search range changing part 11 narrows the search range of the motion vector as the exposure time of the imaging element 3 is longer. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to an imaging apparatus that performs motion compensation processing using a motion vector and encodes a moving image.

  In recent years, video signals have been digitized and high-efficiency encoding of moving images has been performed. As an example of such an encoding algorithm, inter-frame predictive encoding that encodes a difference between frames in a video signal is known. This inter-frame predictive coding is a coding method that reduces the amount of information using the correlation between adjacent frame images. Recently, an encoding method has been used in which motion compensation is added to this encoding method to improve the encoding efficiency of an image including a fast-moving subject. As an example of such an encoding method, motion compensation interframe predictive encoding is known. The above encoding method is adopted in encoding standards such as MPEG (Moving Picture Expert Group) and H261.

2. Description of the Related Art Conventionally, there has been known an imaging apparatus that records an image captured by a video camera in a recording unit after MPEG encoding (see, for example, Patent Document 1). In this conventional imaging apparatus, an image output from a video camera is MPEG-encoded by an encoding unit. In the encoding unit, motion compensation processing for searching for a motion vector in a predetermined range is performed.
Japanese Patent Laid-Open No. 10-276359 (page 2-7, FIG. 2)

  However, in the conventional imaging apparatus, the encoding (motion compensation interframe predictive encoding) processing performed by the encoding unit and the operation of the imaging element are controlled independently of each other. For example, in a conventional imaging apparatus, a search range for detecting a motion vector in motion compensation processing is set in advance regardless of the exposure time of the imaging element. Therefore, when the exposure time of the image sensor is increased in order to increase the sensitivity of the imaging device, that is, when the frame rate is reduced, the compression effect in the time axis direction due to motion compensation is reduced as follows, There is a problem that inefficient moving image encoding is performed.

  For example, when a fast-moving subject is photographed, if the frame rate is low, the subject may move greatly up to the next frame and be out of the motion vector search range. In such a case, there is a problem that the motion compensation process cannot be performed, and the arithmetic process performed for the motion compensation process may be wasted.

  In addition, when a fast-moving subject is photographed, the contour of the subject may become blurred if the exposure time of the image sensor is long. In the process of detecting a motion vector, the current processing target image is compared with a past reference image. Therefore, if the contour of the subject is blurred, there is a problem that it becomes difficult to compare the current processing target image with the past reference image, and the motion vector detection accuracy is lowered.

  Also, if the frame rate is low, the time difference (time interval) between the current frame and the next frame is large, so that the shape of the subject may change during that time. Even in such a case, there is a problem that it becomes difficult to compare the current processing target image with the past reference image, and the motion vector detection accuracy is lowered.

  The present invention has been made to solve the above-described conventional problems, and an object thereof is to provide an imaging apparatus that can efficiently encode a moving image according to an exposure time.

  An image pickup apparatus according to the present invention includes an image pickup unit, an exposure time adjustment unit that adjusts an exposure time of the image pickup unit, a motion vector in an image obtained from the image pickup unit, and a motion compensation process using the motion vector And a search range changing means for changing the width of the search range of the motion vector based on the exposure time of the imaging means. ing.

  With this configuration, when the exposure time changes, the motion vector search range is changed according to the exposure time, and a moving image is encoded. Therefore, for example, even when the exposure time of the image sensor is increased in order to increase the sensitivity of the imaging device, motion vector detection is performed in a search range corresponding to the exposure time, and efficient moving image encoding is performed. It can be performed.

  In the imaging apparatus of the present invention, the search range changing unit has a configuration in which the search range of the motion vector is narrowed as the exposure time of the imaging unit is longer.

  With this configuration, the longer the exposure time, the narrower the motion vector search range. For example, when a fast-moving subject is photographed, if the exposure time is long, the subject may move greatly up to the next frame and be out of the motion vector search range. In such a case, the motion vector search range is narrowed according to the length of the exposure time. As a result, it is not necessary to perform calculation processing for motion vector detection that is highly likely to be wasted, and the amount of calculation can be reduced. Therefore, it is possible to perform other calculation processes by the amount of calculation amount reduction. For example, it is possible to perform arithmetic processing for improving motion vector detection accuracy, such as processing for clarifying the contour of the subject or processing for correcting a change in the shape of the subject.

  In the imaging apparatus of the present invention, the search range changing unit has a configuration that changes the size of the search range of the motion vector in inverse proportion to the length of the exposure time of the imaging unit.

  With this configuration, the motion vector search range becomes narrower as the exposure time becomes longer, and the motion vector search range becomes wider as the exposure time becomes shorter, in inverse proportion to the length of the exposure time. For example, when a fast-moving subject is photographed, if the exposure time is long, the subject may move greatly up to the next frame and be out of the motion vector search range. In such a case, the motion vector search range is narrowed in inverse proportion to the length of the exposure time. As a result, it is not necessary to perform calculation processing for motion vector detection that is highly likely to be wasted, and the amount of calculation can be reduced. Therefore, it is possible to perform other calculation processes by the amount of calculation amount reduction. For example, it is possible to perform arithmetic processing for improving motion vector detection accuracy, such as processing for clarifying the contour of the subject or processing for correcting a change in the shape of the subject.

  The imaging apparatus of the present invention further includes an exposure time determining unit that determines whether an exposure time of the imaging unit is longer than a predetermined threshold exposure time, and the search range changing unit includes the exposure time of the imaging unit. Is longer than the threshold exposure time, the motion vector search range is narrowed.

  With this configuration, when the exposure time is longer than a predetermined threshold exposure time, the motion vector search range is narrowed. For example, when a fast-moving subject is photographed, if the exposure time is longer than a predetermined threshold exposure time, the subject may move greatly to the next frame and be out of the motion vector search range. In such a case, the motion vector search range is narrowed. As a result, it is not necessary to perform calculation processing for motion vector detection that is highly likely to be wasted, and the amount of calculation can be reduced. Therefore, it is possible to perform other calculation processes by the amount of calculation amount reduction. For example, it is possible to perform arithmetic processing for improving motion vector detection accuracy, such as processing for clarifying the contour of the subject or processing for correcting a change in the shape of the subject.

  The present invention has an effect that it is possible to efficiently encode a moving image according to the exposure time by providing a search range changing means for changing the width of the search range of the motion vector based on the exposure time. The imaging device which has can be provided.

  Hereinafter, an imaging device according to an embodiment of the present invention will be described with reference to the drawings. In the present embodiment, an image pickup apparatus or the like that converts and outputs a sensor output signal captured by an image pickup sensor into a digital video signal, particularly for encoding an encoded video signal in order to compress the information amount of the captured video signal. The case where it is the imaging device etc. which output is illustrated. Here, a case of a digital camera having an MPEG encoding function is illustrated.

  A digital camera according to an embodiment of the present invention is shown in FIG. FIG. 1 is a block diagram showing the configuration of the digital camera 1. As shown in FIG. 1, the digital camera 1 includes a lens 2 that collects incident light and an imaging element 3 that photoelectrically converts the incident light collected on the imaging surface into an electrical signal. The imaging device 3 is a sensor such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor), and corresponds to the imaging means of the present invention.

  The digital camera 1 also includes an AD conversion unit 4 that AD converts an analog signal output from the image sensor 3 into a digital signal, and a signal processing unit 5 that performs predetermined signal processing on the digital signal. In the signal processing unit 5, for example, the digital signal is subjected to processing such as gamma correction, white balance correction, and contour correction to output a digital video signal composed of a luminance signal and a color difference signal, or brightness information and color information from the imaging signal. And the like are extracted for control.

  The digital camera 1 also includes an encoding unit 6 that encodes a video signal (digital video signal) obtained from the image sensor 3. The encoding unit 6 detects a motion vector in an image obtained from the image sensor 3 when encoding a moving image, and performs a motion compensation process using the motion vector. Specifically, the current processing target image and a past reference image are compared, and a motion vector in a predetermined search range is detected. This encoding unit 6 corresponds to the encoding means of the present invention.

  As shown in FIG. 1, the digital camera 1 includes a control unit 7 that controls operations of the image sensor 3, the signal processing unit 5, and the encoding unit 6. The control unit 7 controls the shutter speed of the image sensor 3 to adjust the exposure time, and determines whether the exposure time of the image sensor 3 is longer than a predetermined threshold exposure time. An exposure time determination unit 9, a processing control unit 10 that controls the signal processing unit 5 based on predetermined control information (such as brightness information extracted for control), and a wide search range for detecting motion vectors A search range changing unit 11 for changing the height is provided. Here, the exposure time adjustment unit 8 corresponds to the exposure time adjustment unit of the present invention, and the exposure time determination unit 9 corresponds to the exposure time determination unit of the present invention. The search range changing unit 11 corresponds to search range changing means of the present invention.

  The operation of the digital camera 1 configured as described above will be described with reference to the drawings.

  Here, first, the operation of the image sensor 3 will be described with reference to FIGS. FIG. 2 is a timing chart for explaining the normal operation of the image sensor 3. In FIG. 2, the vertical synchronization signal indicates the period of one frame of the video signal. The period from one ON signal to the next ON signal is one frame, and in this figure, four frames from the first frame to the fourth frame are illustrated.

  In FIG. 2, the accumulated charge amount represents an image of charge accumulation of the image sensor 3 (for example, CCD). As shown in FIG. 2, the image pickup device 3 (CCD) normally accumulates electric charges corresponding to the amount of received light for each frame and outputs it as an output signal (sensor output). For example, in the current frame, charges that have been photoelectrically converted by receiving light on the light receiving surface are accumulated, and then in the next frame, an output signal corresponding to the accumulated charges is read for each pixel. In the example shown in FIG. 2, the charge accumulation is completed at the timing A at the end of the first frame, and the output signal is output from the timing B. In FIG. 2, the waveform of the output signal (video signal) read after the second frame is schematically shown. In this figure, the height of the output signal indicates the brightness of the light.

  FIG. 3 is a timing chart for explaining the operation when the level of light input to the image sensor 3 is low, such as when the illumination is dark. In the example shown in FIG. 3, the level of the input light to the image sensor 3 is half that in the case of FIG. Therefore, the charge accumulation amount at timing A is also half that of FIG. 2, and the height of the output signal is also half that of FIG. In such a case, for example, when the illumination is dark, it is difficult for the human eye to recognize the image. Therefore, in such a case, a technique is used in which the exposure time (charge accumulation time) is lengthened to improve the sensitivity of the image sensor 3. Control for adjusting the exposure time is performed by the exposure time adjustment unit 8 of the control unit 7.

  Here, with reference to FIG. 4 to FIG. 7, an operation when the exposure time adjustment unit 8 adjusts the exposure time of the image sensor 3 will be described with some examples.

  FIG. 4 is a timing chart for explaining the operation when the exposure time of the image sensor 3 is increased. In the example shown in FIG. 4, the exposure time of the image sensor 3 is twice that of FIG. That is, in the example of FIG. 2, the accumulation of electric charges is completed after the end of the first frame, whereas in the example of FIG. 4, accumulation of electrification is performed until the second frame is completed. Therefore, in the example shown in FIG. 4, the charge accumulation amount of the image sensor 3 is doubled compared to the case of FIG. 2, and the level of the output signal is also doubled. In this way, the sensitivity of the image sensor 3 is doubled. In order to further improve the sensitivity of the image sensor 3, the exposure time of the image sensor 3 is further increased. Control for adjusting the exposure time in this way (for example, control for increasing the exposure time by two frames) is performed by the exposure time adjustment unit 8 of the control unit 7.

  When the exposure time is increased as described above, the output signal (sensor output) is not output in the frame in which charge is being accumulated. For example, in the example shown in FIG. 4, the output signal (sensor output) is not output in the fourth frame.

  As a method for compensating for the presence of a frame in which an output signal is not output, there is a method of performing frame interpolation by accumulating the output signal in a frame memory or the like. For example, in the example shown in FIG. 5, there is a method in which the output signal of the third frame is stored in the frame memory, and the stored output signal is read again in the fourth frame.

  As another method for compensating for the presence of a frame in which no output signal is output, there is a method of using an accumulated frame signal indicating that a frame is accumulating charges. When the accumulated frame signal is sent together with the output signal (sensor output) to a subsequent processing block (block for performing subsequent processing, not shown), the subsequent processing block refers to the accumulated frame signal and outputs an output signal (video signal). Process according to the presence or absence of. It can be said that this accumulated frame signal is a signal indicating a period of invalidity (a frame in which the output signal is invalid). For example, in the example shown in FIG. 6, the accumulation frame signal is OFF in the third frame, and only the output signal is sent. On the other hand, in the fourth frame, the accumulation frame signal is ON, and the accumulation frame signal is sent together with the output signal. As a result, the post-processing block performs processing as the output signal “present” in the third frame and performs processing as the output signal “none” in the fourth frame.

  FIG. 7 is a timing chart for explaining the operation when the exposure time of the image sensor 3 is adjusted using the shutter pulse. The example in which the exposure time of the image sensor 3 (CCD) is adjusted by accumulating charge signals over a plurality of frames (two frames in the example of FIG. 4) has already been described. Here, an example in which the exposure time is adjusted within one frame will be described. In the example of FIG. 7, a shutter pulse is used as a signal for determining an exposure time (charge accumulation time) within one frame. For example, FIG. 7 shows a state where the shutter pulse is generated four times in the second frame. In addition, an example in which the shutter pulse is generated 8 times in the third frame and the shutter pulse is generated 12 times in the fourth frame is shown. When this shutter pulse is generated, the image sensor 3 (CCD) is controlled so as to discard the accumulated charge. That is, no charge is accumulated while the shutter pulse is generated.

  In the example illustrated in FIG. 7, the exposure time of the second frame is from timing C at which the shutter pulse ends to timing D at which the second frame ends. The exposure time for the third frame is from timing E when the shutter pulse ends to timing F when the third frame ends. The exposure time of the fourth frame is from timing G when the shutter pulse ends to timing H when the fourth frame ends. In this way, the exposure time of the image sensor 3 is adjusted for each frame. Control for adjusting the exposure time (for example, control for shortening the exposure time within one frame) is performed by the exposure time adjustment unit 8 of the control unit 7.

  In this case, if the amount of incident light is the same for all frames, the level of the output signal decreases as the exposure time of the image sensor 3 decreases. Such adjustment of the exposure time is effective when the amount of incident light is large (the subject is bright) and the output signal level is saturated. In other words, even when the amount of incident light is large (the subject is bright) and the output signal level is saturated, the output signal level is suppressed below the saturation level by shortening the exposure time of the image sensor 3. be able to. As a result, a video signal that can be recognized by the human eye can be obtained.

  Next, an operation when the encoding unit 6 detects a motion vector will be described with reference to FIGS.

In this embodiment, the motion vector is detected by the block matching method.
FIG. 8 is an explanatory diagram of a block size (16 pixels × 16 lines) as a unit of motion vector detection. In the block matching method, a target image to be encoded (also referred to as a target image to be encoded) is divided into blocks of 16 pixels × 16 lines as shown in FIG. Then, a motion vector is detected (searched) for each block. That is, the current block to be processed is compared with a block in the past reference image (a block having the same size as the block to be processed currently). Then, the block in the reference image that most closely matches the block of the current processing target image is found, and the relative position of the block with respect to the block of the current processing target image is detected as a motion vector.

  For example, in the comparison between the block of the current image to be processed and the block of the reference image, the sum of absolute differences of pixels at positions corresponding to each other in the block is used as the evaluation value. Therefore, in the example shown in FIG. 8, it is necessary to perform 16 × 16 = 256 times of difference processing and the like for each comparison operation, and this is performed for the number of times of search (for the number of comparison operations).

  FIG. 9 is an explanatory diagram of a motion vector search range (48 pixels × 48 lines). In FIG. 9, the rectangular portion in the middle of the diagonal line is the current pixel block to be processed, and the rectangular portion enclosed by the dotted line is the past reference pixel block. In this example, it is necessary to search for a motion vector in a search range of 48 pixels × 48 lines, and the number of searches is 48 × 48 = 2304.

  The search range changing unit 11 changes the size of the search range of the motion vector based on the exposure time of the image sensor 3. In the present embodiment, the motion vector search range is narrowed as the exposure time of the image sensor 3 increases. In this case, the motion vector search range is narrowed in inverse proportion to the length of the exposure time of the image sensor 3. For example, when the exposure time of the image sensor 3 is doubled, the motion vector search range is doubled. On the contrary, when the exposure time of the image sensor 3 is 0.5 times, the motion vector search range is doubled.

FIG. 10 is an explanatory diagram of a motion vector search range (32 pixels × 32 lines). In the example of FIG. 10, the motion vector search range is 2/3 times that in the case of FIG. That is, this example is an example in which the exposure time of the image sensor 3 is 1.5 times that in the case of FIG. In this case, the number of searches is 32 × 32 = 1024 times, the amount of computation is reduced to about 0.44 times (32 ^{2/48 2).}

  In the present embodiment, the search range changing unit 11 changes the search range of the motion vector as described above when the exposure time T of the image sensor 3 is longer than a predetermined threshold exposure time Ts. FIG. 11 is a flowchart for explaining the operation when the search range is changed.

  As shown in FIG. 11, first, the exposure time adjustment unit 8 adjusts the exposure time of the image sensor 3 in accordance with the photographing condition (brightness of illumination) (S1). Next, the exposure time determination unit 9 determines whether or not the exposure time T set by the exposure time adjustment unit 8 is longer than a predetermined threshold exposure time Ts (S2). When the exposure time T of the image sensor 3 is longer than the threshold exposure time Ts, the search range changing unit 11 changes the search range of the motion vector (S3). Then, a motion vector is detected in the changed search range, and a motion compensation process using the motion vector is performed by the encoding unit 6 (S4).

  On the other hand, when the exposure time T of the image sensor 3 is equal to or shorter than the threshold exposure time Ts, the motion vector search range is not changed. Then, the encoding unit 6 detects a motion vector within the search range, and performs a motion compensation process using the motion vector (S4).

  According to the digital camera 1 of the embodiment of the present invention, by providing the search range changing unit 11 that changes the width of the search range of the motion vector based on the exposure time of the image pickup device 3, It is possible to efficiently encode moving images according to the exposure time.

  That is, in the present embodiment, when the exposure time of the image sensor 3 changes according to the shooting conditions (brightness of illumination), the width of the motion vector search range is changed according to the exposure time, The moving image is encoded. Therefore, for example, when the exposure time of the image sensor 3 is increased in order to increase the sensitivity of the imaging device, such as when the brightness of the illumination is not sufficient, motion vector detection is performed in the search range according to the exposure time, Efficient video encoding can be performed.

  For example, in the present embodiment, the motion vector search range becomes narrower as the exposure time of the image sensor 3 becomes longer. For example, when a fast-moving subject is photographed, if the exposure time of the image sensor 3 is long, the subject may move greatly up to the next frame and deviate from the motion vector search range. In such a case, the motion vector search range is narrowed according to the length of the exposure time of the image sensor 3. As a result, it is not necessary to perform calculation processing for motion vector detection that is highly likely to be wasted, and the amount of calculation can be reduced. Therefore, it is possible to perform other calculation processes by the amount of calculation amount reduction. For example, it is possible to perform arithmetic processing for improving motion vector detection accuracy, such as processing for clarifying the contour of the subject or processing for correcting a change in the shape of the subject. In particular, when the arithmetic processing is realized by software, it is important to efficiently use the processing capability in this way.

Specifically, the motion vector search range becomes narrower as the exposure time becomes longer, and the motion vector search range becomes wider as the exposure time becomes shorter, in inverse proportion to the length of the exposure time. For example, as shown in FIG. 10, when the exposure time of the image sensor 3 becomes 1.5 times, the motion vector search range is made 2/3 times. When a fast-moving subject is photographed, if the exposure time of the image sensor 3 is long, the subject may move greatly up to the next frame and be out of the motion vector search range. In such a case, the motion vector search range is narrowed in inverse proportion to the length of the exposure time of the image sensor 3. As a result, it is not necessary to perform calculation processing for motion vector detection that is highly likely to be wasted, and the calculation amount can be reduced. In this case, as compared with the case of FIG. 9, the calculation amount is reduced to about 0.44 times (32 ^{2/48 2).}

  In the present embodiment, when the exposure time T is longer than the predetermined threshold exposure time Ts, the motion vector search range is narrowed. For example, when a fast-moving subject is photographed, if the exposure time T is longer than a predetermined threshold exposure time Ts, the subject may move greatly to the next frame and be out of the motion vector search range. In such a case, the motion vector search range is narrowed. This also eliminates the need for performing motion vector detection computation processing that is highly likely to be wasted, thereby reducing the amount of computation.

  In this way, the amount of code can be reduced and the amount of calculation can be reduced. This makes it possible to efficiently encode a moving image from both the code amount and the calculation amount. Note that a subject that does not move (background, etc.) can be compressed without any problem.

  The embodiments of the present invention have been described above by way of example, but the scope of the present invention is not limited to these embodiments, and can be changed or modified according to the purpose within the scope of the claims. is there.

  As described above, the image pickup apparatus according to the present invention has an effect of being able to efficiently encode a moving image according to the exposure time of the image pickup device, and is a digital camera having an MPEG encoding function. Useful as.

1 is a block diagram of a digital camera according to an embodiment of the present invention. Timing chart for explaining operation of image pickup device (normal time) according to an embodiment of the present invention Timing chart for explaining operation of image pickup device (when input light level is low) in the embodiment of the present invention Timing chart for explaining operation of image pickup device (when exposure time is doubled) in the embodiment of the present invention Timing chart for explaining the operation of the image sensor (when performing frame interpolation output) in the embodiment of the present invention Timing chart for explaining operation of image pickup device (when using accumulated frame signal) in the embodiment of the present invention Timing chart for explaining operation of image pickup device (when shutter pulse is used) in the embodiment of the present invention Explanatory drawing of block size (16 pixels × 16 lines) as a unit of motion vector detection in the embodiment of the present invention Explanatory drawing of the search range (48 pixels x 48 lines) of the motion vector in the embodiment of the present invention Explanatory drawing of the search range (32 pixels x 32 lines) of the motion vector in the embodiment of the present invention FIG. 3 is a flowchart for explaining the operation of the digital camera in the embodiment of the present invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Digital camera 3 Image pick-up element 6 Encoding part 7 Control part 8 Exposure time adjustment part 9 Exposure time determination part 11 Search range change part

Claims (4)

Imaging means;
Exposure time adjusting means for adjusting the exposure time of the imaging means;
Encoding means for detecting a motion vector in an image obtained from the imaging means and performing a motion compensation process using the motion vector to encode a moving image;
Search range changing means for changing the width of the search range of the motion vector based on the exposure time of the imaging means;
An imaging apparatus comprising: The search range changing means includes
The imaging apparatus according to claim 1, wherein the longer the exposure time of the imaging unit, the narrower the search range of the motion vector. The search range changing means includes
The imaging apparatus according to claim 1, wherein the motion vector search range is changed in inverse proportion to a length of an exposure time of the imaging unit. Exposure time determination means for determining whether an exposure time of the imaging means is longer than a predetermined threshold exposure time;
The search range changing means includes
The imaging apparatus according to claim 1, wherein when the exposure time of the imaging means is longer than the threshold exposure time, the search range of the motion vector is narrowed.

Images