JP4724639B2 - Imaging device - Google Patents

Description

  The present invention relates to an imaging apparatus, and more specifically to an imaging apparatus that selects or synthesizes a plurality of captured images and performs compression encoding.

  In recent years, an imaging apparatus that digitally encodes a moving image and records it on a recording medium such as a disk-shaped recording medium or a semiconductor memory has been proposed. As a compression encoding method of a moving image, DCT (Discrete Cosine Transform) conversion, MPEG (Moving Pictures Experts Group) based on motion compensation and variable length coding, MPEG-4, H.264, etc. H.264 is also known. These use block coding in which one screen is divided into a plurality of blocks each composed of a plurality of pixels, and transform coding is performed on a block basis.

  For example, in the MPEG method, one image is divided into pixel blocks of n pixels × n pixels (n is, for example, 8 or 16), and DCT conversion is performed in block units on the motion-predicted difference image. Next, the DCT coefficient after conversion is divided by a certain divisor (quantization step), and the remainder is rounded and quantized to reduce the amount of information.

  In the case of a moving image, the feature that the images before and after are very similar is used. That is, moving image information is compressed by a motion compensation method that uses the presence / absence of a change between the previous image and the current image and the amount of movement in pixel blocks.

  Furthermore, there is a tendency that the DCT coefficient and the movement amount have a clear difference in appearance probability, and the average information generation amount can be reduced by using variable length coding that assigns a short code to a code having a high appearance probability. .

  In the MPEG system, compression encoding is executed in a unit called GOP (Group of Pictures) in which a certain number of images are collected. Within 1 GOP, there are an I picture, a P picture, and a B picture, each of which has a different encoding method. An I picture is an intra-frame coded image, and all macroblocks are intra-coded. A P picture is an inter-frame predictive encoded image, and intra-frame encoding and inter-frame encoding are selected for each macroblock. In inter-frame encoding, a difference value from the immediately preceding frame is encoded. The B picture is a bi-directional predictive encoded image, and a difference value from the predicted value from the immediately preceding and subsequent I pictures or P pictures is encoded.

  In the decoding process, the I picture and P picture are images used for prediction, while the B picture is an image reproduced using the I picture and P picture. For this reason, the overall signal-to-noise ratio is improved by keeping the picture quality of the I picture and P picture high and keeping the picture quality of the B picture low.

  When a moving image signal is compressed using variable length coding, the amount of data after compression varies depending on the moving image signal. The recording rate is kept constant by temporarily storing the compressed data in a buffer and reading it at a constant rate. Depending on the relationship between the write rate and read rate of the buffer, underflow of the buffer may occur, and the compression rate is adjusted by controlling the quantization step. As a result, when the compression rate of the I picture image that is the reference for prediction reference changes in a scene change or a scene in which the luminance changes rapidly, image quality deterioration becomes conspicuous.

  In particular, when the illumination required for photographing cannot be obtained indoors or the like, if slow shutter photographing is employed in which a bright image is photographed by slowing down the shutter speed, the difference between the images tends to increase. Further, in slow shutter photography, an image is updated only at 1 / n (n: natural number) at the time of normal photography, so that a recorded image is difficult to be smooth.

  To deal with such a problem, Patent Document 1 discloses that the data input amount of the compressed image data is reduced by slowing down the image input speed to the encoding processing unit and reconstructing the GOP picture configuration during slow shutter photography. The technology to do is described.

Patent Document 2 describes a technique for recording a smooth image by combining a plurality of image signals and generating a single image during slow shutter photography.
JP 2001-025011 A JP 2000-307964 A

  With the technique described in Patent Document 1, it is possible to prevent deterioration in image quality due to fluctuations in the compression ratio, but the recorded image remains 1 / n at the time of normal shooting, and a smooth image cannot be recorded. In addition, since the GOP picture configuration is reorganized, there is a problem that complicated processing is required in the encoding processing unit.

  With the technique described in Patent Document 2, a smooth image can be recorded, but it is not possible to prevent deterioration in image quality due to fluctuations in the compression rate during encoding. In addition, since the I picture image that serves as a reference for prediction reference is a composite image, the overall image quality is degraded.

  An object of this invention is to show the imaging device which solves such a problem.

Imaging device according to the present invention includes an imaging means to output the shooting image signals, an image memory for storing the frames included in the photographed image signal, sequentially updates the frame to be stored according to a first update interval An image memory for storing frames that are temporally continuous with frames stored in the first image memory and sequentially updating the frames stored in accordance with the first update interval A second image memory; and an encoding unit that compresses and encodes an image signal input at a second update interval shorter than the first update interval by intraframe encoding or interframe predictive encoding; One of the two consecutive frames stored in the first image memory and the second image memory is selected, and the image signal of the selected frame is set in accordance with the second update interval. Wherein the image selecting means to supply to the encoding means, and the timing of inter-frame predictive coding is performed by said encoding means, based on the first update interval, selecting for controlling the selection of said image selecting means Te characterized by comprising a control hand stage.

Imaging device according to the present invention includes an imaging means to output the shooting image signals, an image memory for storing the frames included in the photographed image signal, sequentially updates the frame to be stored according to a first update interval An image memory for storing frames that are temporally continuous with frames stored in the first image memory and sequentially updating the frames stored in accordance with the first update interval A second image memory; and an encoding unit that compresses and encodes an image signal input at a second update interval shorter than the first update interval by intraframe encoding or interframe predictive encoding; Two consecutive frames of image signals stored in the first image memory and the second image memory are synthesized, and the synthesized image signal is supplied to the encoding means according to the second update interval. That an image combining unit, and the timing of inter-frame predictive coding is performed by said encoding means, based on the first update interval, and image combining control means to control the mixing ratio of the image synthesizing unit It is characterized by having

  According to the present invention, it is possible to reduce the amount of encoded data without reconstructing the encoding structure by controlling the selection or synthesis of the low-speed captured image according to the image update timing and the encoding structure information. .

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 is a block diagram showing a schematic configuration of an imaging apparatus according to an embodiment of the present invention. An optical image of a subject enters the image sensor 14 through the lens 10 and the diaphragm 12, and the image sensor 14 converts the optical image into an electric signal and outputs an image signal. A CDS (Correlated Double Sampling) circuit 16 removes noise of the image signal output from the image sensor 14 by correlated double sampling, and an AGC (Auto Gain Control) circuit 18 detects the level of the output image signal of the CDS circuit 16. To automatically control. The A / D converter 20 converts the analog image signal output from the AGC circuit 18 into a digital image signal. This digital image signal is stored in the image memory 22.

  The CPU 38 controls the charge accumulation time of the image sensor 14 by the shutter speed control device 36. Thereby, the exposure time of the image sensor 14 is controlled. This is a so-called electronic shutter function, and the shutter speed is changed. The CPU 38 controls the write timing of the image memory 22 so that only necessary signals are written in accordance with the charge accumulation time of the image sensor 14.

  The image memory 24 stores the image data read from the image memory 22 in accordance with the update timing of the image memory 22. As a result, two consecutive frames of image signals are stored in the image memories 22 and 24.

  The image synthesizing device 26 synthesizes the stored image in the image memory 22 and the stored image in the image memory 24 with the synthesis coefficient specified by the image synthesis control device 40, and outputs the synthesized image to the camera signal processing device 28. When the stored image in the image memory 22 is A, the stored image in the image memory 24 is B, the composite coefficient for the image A is x, and the composite coefficient for the image B is y, the image composition device 26 outputs a composite image obtained by xA + yB. To do. Usually, y = 1−x.

  When x = 1, the image composition device 26 selects a stored image in the image memory 22, and when y = 1, the image composition device 26 selects a stored image in the image memory 24. That is, when x is 0 or 1, the image composition device 26 functions as a selector that selects one of the images stored in the image memory 22 and the image memory 24. In this embodiment, this is particularly called selective synthesis.

The camera signal processing unit 28, a well-known camera signal processing of the exposure control and color balance adjustment in the composite image signal from the image synthesizer 2 6 performed, and outputs the processing result to the encoder 30.

  The encoder 30 compresses and encodes the image signal from the camera signal processing device 28 in accordance with preset GOP structure information, and outputs the compressed image data to the buffer 32. In the case of MPEG2, the encoder 30 has a known circuit configuration such as a block divider, a DCT circuit, a quantizer, a variable length encoder, and a motion compensation circuit. The encoder 30 changes the quantization step according to the buffer remaining amount information from the buffer 32 when the buffer remaining amount is low.

  The buffer 32 reads the compressed image data written from the encoder 30 to the recording medium 34 at a constant rate.

  FIG. 2 shows an operation flowchart of the image composition device 26. Here, the image composition control device 40 controls the image composition ratio with binary values, and the image composition device 26 functions as a selector for selecting one of the image in the image memory 22 and the image in the image memory 24. FIG. 3 shows an operation for adaptively controlling the composition ratio in binary according to the GOP structure information and the image update timing. For reference, FIG. 4 shows the operation when the composition ratio is not controlled.

In the flow shown in FIG. 2, first, it is determined whether or not the synthesis target image, that is, the image stored in the image memory 22 or the image memory 24 is an image encoded as an I picture (S1). When an I picture is included (S1), 1 is set to the synthesis coefficient x and 0 is set to the synthesis coefficient y (S2). When none of the images stored in the image memories 22 and 24 is an I picture (S1), an image update timing is set between the image stored in the image memory 22 or the image memory 24 and the immediately preceding predicted reference image. It is checked whether it is contained (S3). When the image update timing has not been entered (S3), 1 is set to the synthesis coefficient x and 0 is set to the synthesis coefficient y (S2). That is, the image composition device 26 selects an image in the image memory 22. When the image update timing has been entered (S3), 0 is set to the synthesis coefficient x and 1 is set to the synthesis coefficient y (S4). That is, the image synthesizing apparatus 2 6 selects the image in the image memory 24.

  The effect of controlling the synthesis ratio shown in FIG. 2 will be described with reference to FIGS. FIG. 3A shows the image update timing, which is also the imaging timing of the electronic shutter by the shutter speed control device 36. FIG. 3B shows a stored image in the image memory 22, and FIG. 3C shows a stored image in the image memory 24. FIG. 3D shows the encoding timing in the encoder 30, and the image synthesis device 26 generates a synthesized image in accordance with the encoding timing. FIG. 3E shows which picture the image synthesized by the image synthesizing device 26 is encoded.

  FIG. 3F shows the composite coefficient x of the stored image in the image memory 22, and FIG. 3G shows the composite coefficient y of the stored image in the memory 24. FIG. 3 (h) is a synthesized image synthesized by the image synthesizing device 26 according to FIGS. 3 (b), (c), (f) and (g). FIG. 3 (i) is prediction reference information indicating an image to which the synthesized image is predicted and referenced at the time of encoding, and an image indicated by an arrow is referred to.

  FIG. 3J shows the amount of encoded data. In the I picture, since all macroblocks are intraframe-encoded, the amount of encoded data increases. When there is no difference value between the P picture and the picture to be predicted and referenced, all macroblocks are inter-frame encoded and the amount of encoded data is small. When there is a difference value, the number of encoded data increases because the number of macroblocks to be intraframe encoded increases. Since the B picture can use the prediction reference pictures before and after, the amount of encoded data is basically small. As shown in FIG. 3, when the image update cycle is longer than the cycle in which the I picture or P picture appears, the difference value from the predicted reference image is small, and the amount of encoded data is small.

  FIGS. 4A to 4J show the same contents as FIGS. 3A to 3J. However, in FIG. 4, the synthesis coefficient x is always 1 and the synthesis coefficient y is always 0. That is, FIG. 4 corresponds to the case where the image stored in the memory 22 is input to the camera signal processing device 28 as it is.

In the case of FIG. 4, an image with a large amount of information at the time of encoding continues in a short period of pictures P _{4 to} P ₁₀ . As a result, when the remaining amount of the buffer 32 is small when the picture I ₇ or P ₁₀ is encoded, the quantization step is controlled, and the compression rate varies. When the compression rate of an image serving as a reference for prediction such as an I picture increases, deterioration during decoding affects all images in the GOP.

In contrast, in the present embodiment, as shown in FIG. 3, as in the picture B ₃ to P ₄ and B ₉ to P _10, the prediction reference image and the image update timing between the current picture (Fig. 3 ( When a)) is entered, it is possible to select the stored image of the second memory 24 that is a predicted reference image and supply the selected image to the encoder 30, thereby reducing the remaining amount of the buffer 32 when encoding an I picture. Preventing and reducing the possibility of readjustment of the quantization step.

  As described above, in this embodiment, one of two temporally continuous images is selected from the image update timing and the GOP structure information and supplied to the encoder, thereby improving the encoding efficiency. As a result, the amount of encoded data can be reduced without reconstructing the GOP structure.

  Next, a second embodiment of the present invention will be described. FIG. 5 shows a flowchart for adaptively determining the synthesis coefficient of two images according to the GOP structure information, and FIG. 6 shows an operation example of this embodiment. FIG. 7 shows an operation example in the case of synthesizing an image with a fixed synthesis coefficient value without referring to the GOP structure information for two frames after the image update timing.

  In the flow shown in FIG. 5, first, it is determined whether the synthesis target image, that is, the image stored in the memory 22 or the memory 24 is an image encoded as an I picture (S11).

  When the I picture is not included (S11), the images in the memories 22 and 24 are linearly synthesized with the synthesis coefficient x = 1/3 and the synthesis coefficient y = 2/3 with respect to the first frame at the image update timing. For the second frame, the images in the image memories 22 and 24 are linearly synthesized with the synthesis coefficient x = 2/3 and the synthesis coefficient y = 1/3 (S12).

  If an I picture is included (S11), it is determined whether the first picture immediately after the image update is an I picture (S13). When the first picture is an I picture (S13), the image before the image update is selected for the I picture, and the synthesis coefficient x = 1/3 and the synthesis coefficient y = 2/3 for the first frame thereafter. The images in the image memories 22 and 24 are linearly synthesized, and the images in the image memories 22 and 24 are linearly synthesized with the synthesis coefficient x = 2/3 and the synthesis coefficient y = 1/3 for the second frame ( S14). As in this example, linear synthesis of the images in the image memories 22 and 24 while changing the synthesis coefficients x and y for each picture for a plurality of temporally continuous pictures is referred to as gradient synthesis. To.

  When the first picture immediately after the image update is not an I picture (S13), the synthesis coefficients x and y are respectively set to ½ with respect to the image before the I picture, and the images in the image memories 22 and 24 are linearized. Synthesize (S15).

  6 (a) to (i) and FIGS. 7 (a) to (i) show the same contents as FIG. 3 (a) to (i). The synthesis coefficient x shown in FIG. 6F and the synthesis coefficient y shown in FIG. 6G are determined according to the flow shown in FIG. On the other hand, the synthesis coefficient x shown in FIG. 7 (f) and the synthesis coefficient y shown in FIG. 6 (g) are constant.

When the gradient composition is simply applied to the two frames after the image update timing, as illustrated in FIG. 7, the two pictures after the image update timing are I _{3 to} B ₄ and B _{8 to} I ₉ , and the composite image is It becomes FIG.7 (h). Since I ₃ and I ₉ encoded as an I picture are combined images from a plurality of images, the image quality of the entire GOP is degraded.

On the other hand, as shown in FIG. 6, in the present embodiment, the synthesis coefficients x and y and the synthesis range (start and end) are controlled with reference to the GOP configuration information, so that the image quality deterioration can be eliminated or reduced. . Specifically, when the first picture after the image update timing is an I picture as in I _{3 to} B ₄ , the picture before the image update is adopted (x = 0, y = 1), and the next picture after the I picture is used. For the two pictures from the picture, the images in the image memories 22 and 24 are linearly synthesized. In addition, when the first picture after the image update timing is not an I picture as in B _{8 to} I ₉ , as shown in FIGS. The images in the memories 22 and 24 are linearly synthesized.

By controlling the synthesis coefficients x and y and the synthesis range in this way, the pictures I ₃ and I ₉ are composed of only one image in the image memories 22 and 24. In addition, the pictures B _{4 to} B ₅ and B ₈ are images obtained by linearly synthesizing images before and after the image update timing.

  As described above, in this embodiment, by controlling the composite coefficient of two images that are temporally continuous from the image update timing and the GOP structure information, a smooth image can be obtained while maintaining high image quality of the I picture image. Can record.

  FIG. 8 shows a schematic block diagram of the third embodiment of the present invention. Constituent elements having the same functions as those of the embodiment shown in FIG. A different part from Example 1 is demonstrated. The image composition control device 60 refers to not only the GOP structure information from the encoder 30 but also the remaining amount information from the buffer 32 and determines the composition counts x and y of the image composition device 26.

  FIG. 9 shows an operation example of the embodiment shown in FIG. FIGS. 9A to 9I show the same contents as FIGS. 3A to 3I. FIG. 9J shows an index indicating the amount of encoded data when encoding is performed in a normal quantization step. Here, in the case of inter-frame coding when the index of the coding data amount in the case of an I picture for intra-frame coding of all macroblocks is set to a maximum value of 7 and there is no difference value from the predicted reference picture The index of the encoded data amount is set to 1 as the minimum value. FIG. 9K shows an index of the remaining capacity of the buffer 32. When the remaining capacity index is 0, an underflow occurs. If the data amount (index) to be output to the recording medium is 3, the remaining amount (index) of the buffer 32 increases or decreases depending on the difference value between the encoded data amount and the data amount to be output to the recording medium.

In FIG. 9, when the remaining amount of the buffer is large such as B _{3 to} P ₄ and I _{13 to} B ₁₅ , smoothness is achieved by using gradient synthesis using non-zero synthesis coefficients x and y over a plurality of pictures. Record an image. When the remaining amount of the buffer is small as in B _{8 to} P ₁₀ , the data amount at the time of encoding is reduced by using the selective combination for selecting one of the images.

  As described above, in this embodiment, when the buffer 32 for equalizing the recording rate is likely to underflow, the encoded data amount is reduced by performing binary control of image synthesis, and the buffer 32 is underflowed. If not, a smoother image is recorded by using gradient synthesis or equality synthesis.

  FIG. 10 shows a schematic block diagram of a fourth embodiment of the present invention. Constituent elements having the same functions as those of the embodiment shown in FIG. A different part from Example 1 is demonstrated.

  The exposure evaluation value calculation device 70 divides the image signal output from the A / D converter 20 into a plurality of areas in the screen, calculates an exposure evaluation value based on luminance for each area, and the image composition control device 72. Output to. The image composition control device 72 determines the composition coefficients x and y of the image composition device 26 according to the GOP structure information from the encoder 30 and the amount of change in the exposure evaluation value before and after the image update timing.

  FIG. 11 shows an operation example of the embodiment shown in FIG. FIGS. 11A to 11J show the same contents as FIGS. 9A to 9J. FIG. 11K shows the amount of change in the exposure evaluation value before and after the image update timing. In many cases, the amount of change in the exposure evaluation value is proportional to the difference value between images.

In FIG. 11, when the difference value of the exposure evaluation values is small as in B _{3 to} P ₄ and I _{13 to} B ₁₅ , a smooth image is recorded by applying gradient synthesis. Also, if a large difference value as B ₈ to P _10, reducing the encoded data amount by using the selective combining.

  As described above, in the present embodiment, when the change in the exposure evaluation value is large, the selective combination is applied, and when the change in the exposure evaluation value is small, the gradient combination is applied. Smooth image recording while reducing the amount.

  Although MPEG has been described as an example of a moving image compression encoding method, it is obvious that the present invention can be applied to the case of using another encoding method in which the amount of encoded data varies dynamically depending on the screen. is there.

It is a schematic block diagram of Example 1 of the present invention. FIG. 2 is an operation flowchart of the image composition device 26. It is an operation example of the present embodiment in which the composition ratio is adaptively binary-controlled according to GOP structure information and image update timing. It is an example of an image composition operation when the composition ratio is not controlled. 10 is a flowchart of an operation of adaptively determining a synthesis coefficient of two images according to GOP structure information in the second embodiment. 10 is an operation example of the second embodiment. It is an operation example in the case of synthesizing an image with a fixed synthesis coefficient value without referring to GOP structure information for two frames after the image update timing. FIG. 10 is a schematic configuration block diagram of Embodiment 3. 10 is an operation example of the third embodiment. FIG. 10 is a schematic block diagram of a fourth embodiment. 10 is an operation example of the fourth embodiment.

Explanation of symbols

10: Lens 12: Aperture 14: Image sensor 16: CDS circuit 18: AGC circuit 20: A / D converter 22, 24: Image memory 26: Image composition device 28: Camera signal processing device 30: Encoder 32: Buffer 34: Recording medium 36: Shutter speed control device 38: CPU
40: Image composition control device 60: Image composition control device 70: Exposure evaluation value calculation device 72: Image composition control device

Claims (2)

An imaging means to output the shadow image signals Ta,
An image memory for storing frames included in the captured image signal, wherein the first image memory sequentially updates the frames stored in accordance with a first update interval ;
An image memory for storing frames that are temporally continuous with the frames stored in the first image memory, wherein the second image memory sequentially updates the frames stored in accordance with the first update interval;
Encoding means for compressing and encoding an image signal input at a second update interval shorter than the first update interval by intraframe encoding or interframe predictive encoding;
One of the two consecutive frames stored in the first image memory and the second image memory is selected, and the image signal of the selected frame is sent to the encoding unit according to the second update interval. Image selection means to supply;
And when the inter-frame predictive coding is performed by said encoding means, the first based on the update interval, imaging, characterized by comprising a selection control means to control the selection of said image selection means apparatus. An imaging means to output the shadow image signals Ta,
An image memory for storing frames included in the captured image signal, wherein the first image memory sequentially updates the frames stored in accordance with a first update interval ;
An image memory for storing frames that are temporally continuous with the frames stored in the first image memory, wherein the second image memory sequentially updates the frames stored in accordance with the first update interval;
Encoding means for compressing and encoding an image signal input at a second update interval shorter than the first update interval by intraframe encoding or interframe predictive encoding;
Image synthesizing means for synthesizing two consecutive frames of image signals stored in the first image memory and the second image memory and supplying the synthesized image signal to the encoding means in accordance with the second update interval; ,
And when the inter-frame predictive coding is performed by said encoding means, and characterized in that based on the first update interval comprises an image combining control means to control the mixing ratio of the image synthesizing unit An imaging device.

Images