JP2007067823A - Imaging and compressive recording and reproducing device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To reduce the power consumption of a camera recorder constituted by combining a pixel-shift applied imaging section and a compressive recording section while suppressing the deterioration of picture quality of a reproduced image to a necessary minimum. <P>SOLUTION: An imaging section which is applied with a pixel shift sends an image output from an imaging element to the compressive recording section without interpolation synthesis to compress the image, which is recorded on a recording medium. When the recorded compressed image data are reproduced, the compressed image data are decompressed by a reproduction section and then the pixel interpolation synthesis is carried out. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention provides low power consumption in a camera recorder having an imaging unit that captures an image, a compression recording unit that compresses and records the captured image, and a player that has a playback unit that expands the recorded compressed image. The present invention relates to a method and apparatus for recording / reproducing with high image quality.

Many camera-integrated recorders (camera recorders) using pixel-type solid-state imaging devices such as CCDs and CMOSs as imaging units have been commercialized. As the configuration of the imaging unit, as shown in FIG. 2A, a 1CCD system in which color filters different in RGB are two-dimensionally arranged for each pixel to obtain a color image with one imaging element, or FIG. As described above, various types such as a 3CCD system that obtains a color image with a higher resolution by independently arranging three image sensors for each RGB component have been proposed. In a camera recorder, an image sensor is a central important component both in terms of function and cost. In particular, in a method in which an image pickup unit is configured by a plurality of image pickup devices, such as the 3CCD method, the cost ratio of the image pickup device to the product cost is increased, and thus the cost reduction of the image pickup device is strongly demanded. The cost reduction of the CCD can be achieved by reducing the area of the CCD, which is a semiconductor chip. However, if the area is simply reduced with the same number of pixels, the light irradiation area per pixel also becomes smaller and the sensitivity is lowered. The signal-to-noise ratio (S / N ratio) decreases. Due to these properties, when the total area of the semiconductor chip is constant, the design of the image sensor has the following two ideas.
(1) The area per pixel is small and the SN ratio is reduced, but high resolution is targeted. (2) Although the resolution is low, the area per pixel is large and a high SN ratio is targeted. The resolution and the SN ratio are in a trade-off relationship, and it is necessary to determine a well-balanced resolution and chip area in order to achieve both desired performance and cost.

  Against this background, a technique called “pixel shifting” has been devised in order to synthesize a higher resolution image using a low resolution CCD with a good S / N ratio (see Patent Document 1). By shifting pixels, for example, by using three CCDs having only half the resolution in the horizontal direction and interpolating and synthesizing images from the CCD, it becomes possible to obtain a color image having twice the horizontal resolution. As described above, if the total area of the CCD is the same, the CCD with half the horizontal resolution can increase the area per pixel, so that a video signal with high sensitivity and high S / N ratio is obtained by shifting pixels. It becomes possible.

  Hereinafter, resolution conversion by horizontal pixel shift will be described in detail with reference to FIGS. FIG. 3 is a schematic diagram showing how the three CCDs are arranged in a three-CCD camera that independently images RGB channels with three CCDs. In this example, the vertical positions of the RGB individual pixels are made the same, that is, the light incident from the lens is split in the vertical direction by a prism or the like, and is incident on the corresponding pixel position of the CCD of each RGB channel. Assume that three CCDs are arranged. Each of FIGS. 3A and 3B shows that CCDs of G channel, R channel, and B channel are arranged vertically from above. In a camera that does not use pixel shifting, as shown in FIG. 3A, three CCDs with independent RGB, whose horizontal resolution (hereinafter referred to as horizontal resolution) is N pixels, each pixel has a horizontal position. Physically arranged to be the same. On the other hand, a camera that applies pixel shifting uses three CCDs with a horizontal resolution of only N / 2 pixels as shown in FIG. 3B, and uses the R channel and B channel with reference to the CCD pixels for the G channel. The CCD pixels of the channel are arranged so as to come to a position physically shifted horizontally by 1/2 pixel.

  FIG. 4 shows G channel, R channel, and B channel signals (hereinafter referred to as G signal, R signal, and B signal, respectively) sampled by an image pickup unit that has been subjected to horizontal pixel shift as shown in FIG. FIG. 6 is a schematic diagram showing a positional relationship in a horizontal direction and a signal level of a pixel. In FIG. 4, the horizontal axis represents the pixel position in the horizontal direction, and the vertical axis represents the signal level of the pixel. As described above, the R signal and the B signal sampled with a phase shifted by 1/2 pixel with respect to the G signal have a horizontal positional relationship as shown in FIG. Therefore, the G signal, the R signal, and the B signal are sample rows at positions shifted by ½ pixel in the horizontal direction, as shown in FIG. 4B.

Next, a signal having a horizontal resolution of N / 2 pixels is oversampled in the horizontal direction, and a process of interpolating and synthesizing the horizontal resolution to N pixels is performed. FIG. 5 is a schematic diagram showing a process of over-sampling and outputting pixel signals obtained by the CCD of each RGB channel. In FIG. 5, the horizontal axis represents the pixel position in the horizontal direction, and the vertical axis represents the signal level of the pixel. 5 times over so that the same sample data is arranged two by two as shown in FIG. 5A with respect to the signal of horizontal resolution N / 2 pixels including R signal and B signal shifted by 1/2 pixel in the horizontal direction. A G ′ signal, an R ′ signal, and a B ′ signal, which are signals subjected to sampling processing, are generated. Then, Y ′ = (G ′ + R ′) / 2 from these signals
The luminance signal Y ′ having a horizontal resolution of N pixels is synthesized by the approximate expression of Finally, using this Y ′ signal, the high frequency component of the RGB signal is approximated by the following formula to reinforce the high frequency component, and the obtained R ″ signal, G ″ signal and B ″ signal are output. . However, the subscript _L means the low frequency component of the signal, and _H means the high frequency component of the signal.
R ″ = LPF (R′−Y ′) + Y ′ = R _L ′ + Y _H ′
G '' = LPF (G'- Y ') + Y' = G L '+ Y H'
B ″ = LPF (B′−Y ′) + Y ′ = B _L ′ + Y _H ′
The function LPF () represents a low-pass filter that passes only the low-frequency component of the signal.

  Such pixel interpolation and pixel interpolation / combination by oversampling makes it possible to synthesize an image with a higher resolution using a signal captured by a low-cost CCD with a good S / N ratio.

On the other hand, in the camera recorder, an image taken at the same time as imaging is recorded on a medium. At this time, in order to increase the utilization efficiency of the recording medium, it is a general practice to record the image by compressing and encoding the image with a compression recording unit. There are the following two image compression methods performed by the compression recording unit.
(1) Lossless compression in which information is completely restored through a compression / decompression process (2) Lossless compression that reduces visually redundant information by using spatial and temporal correlation of images General natural images In order to obtain high compression efficiency, irreversible compression represented by JPEG (see Non-patent Document 1) and MPEG (see Non-Patent Document 2 and Non-Patent Document 3) is indispensable, but is irreversible. Therefore, it is known that the higher the compression rate, the more the distortion in the reproduced image.

Conventional imaging compression / recording / reproduction devices such as digital still cameras and digital camera recorders that are widely used today are products in which an imaging unit and a compression recording unit are integrated. Optimal design is being tried in the trade-off.
Japanese Patent Laid-Open No. 56-8978 ISO / IEC 10918-1 Information technology-Digital compression and coding of continuous-tone still images: Requirements and guidelines ISO / IEC 13818-2 Information technology-Generic coding of moving pictures and associated audio information: Video ITU-T Infrastructure of audiovisual services-Coding of moving videos: Advanced video coding for generic audio services

  However, in a conventional camera recorder that combines an image pickup unit that applies pixel shift and a compression recording unit, a signal whose data rate has been increased by oversampling the image pickup unit is converted to a desired data rate by the compression recording unit. The compression not only reduces the quality of the reproduced image more than necessary, but also increases the processing clock rate of the compression recording unit and the reproduction unit, which causes a problem of consuming more power. In other words, the imaging unit performs an operation of increasing the data rate by up-converting the image to a higher resolution image by shifting pixels based on the image obtained from the lower resolution imaging device. On the other hand, in the compression recording unit, there is a contradiction in that an up-converted high-resolution image to be input is irreversibly compressed so as to have a desired data rate, thereby causing distortion more than necessary.

  In view of such a problem, an object of the imaging compression / recording / reproducing apparatus of the present invention is to suppress degradation of the image quality of a reproduced image to a necessary minimum and to reduce power consumption.

  In order to solve the above problems, an imaging compression recording / reproducing apparatus of the present invention includes an imaging unit in which at least one of a plurality of imaging elements is arranged at a position shifted in phase from a position corresponding to another imaging element, and an imaging A compression recording unit for compressing an image from the recording unit, an image recording / reproducing unit for recording the compressed image data from the compression recording unit on the recording medium, and reading the compressed image data from the recording medium, and a compressed image data from the image recording / reproducing unit And an interpolating and synthesizing means for increasing the resolution by interpolating and synthesizing the image from the image expanding means.

  According to the imaging compression / recording / reproducing apparatus of the present invention, in an imaging unit to which pixel shift is applied, an image output from the imaging device is sent to the compression recording unit without interpolation and synthesis, and is recorded on the recording medium. When image data is reproduced, pixel interpolation / synthesis is performed in the reproduction unit, so that the input data rate to the compression recording unit is reduced, and the compression rate when the image data is compressed to a desired data rate can be reduced. As a result, there is an effect of suppressing the deterioration of the image quality of the reproduced image to the minimum necessary. In addition, since the input data rate to the compression recording unit decreases, the compression / decompression processing clock rate also decreases, and there is an effect of reducing power consumption. Further, H.C. By adopting a compression recording method that can multiplex and record and transmit the phase information of pixel samples like the H.264 / MPEG4-AVC compression method, it becomes possible to flexibly be compatible with various pixel shifting methods.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

(Embodiment 1)
The imaging compression recording / reproducing apparatus according to the first exemplary embodiment of the present invention will be described with reference to FIG. FIG. 1 is a block diagram showing a configuration of a camera recorder as an imaging compression recording / reproducing apparatus according to Embodiment 1 of the present invention. This camera recorder has an image recording unit 105 that applies pixel shift, a compression recording unit 111, an image recording / reproducing unit 112, a recording control unit 106, a recording memory control unit 108, A recorded image memory 107 is provided. The imaging unit 105 includes three imaging elements (CCD) 102, A / D conversion unit 103, and imaging unit control unit 104 arranged by shifting pixels. The three image sensors correspond to the R channel, the G channel, and the B channel, respectively, and the horizontal resolution is half that required for reproduction. The arrangement is the same as that of a conventional camera recorder using pixel shift. That is, two of the three CCDs (R channel and B channel in FIG. 3B) are shifted in phase by 1/2 pixel in the horizontal direction from the position corresponding to the other CCD (G channel). Placed in position. The compression recording unit 111 includes an image correction unit 109 and an image compression unit 110. Further, as a configuration for image reproduction, the image recording / reproducing unit 112, the reproducing unit 117, the reproduction control unit 113, the reproduction memory control unit 118, and the reproduction image memory 114 are provided. The reproduction unit 117 includes an image expansion unit 115 and a pixel shift interpolation synthesis unit 116. Reference numeral 119 denotes a recording medium for recording compressed image data.

  Hereinafter, operations during image recording and image reproduction will be described in order.

  At the time of recording, in the image pickup unit 105, the image pickup element 102 arranged by pixel shifting picks up an image, and the A / D conversion means 103 converts the picked up video signal into a digital signal and outputs it. The imaging unit control unit 104 appropriately controls the operation timing of each unit of the imaging unit 105. A digital image signal photographed by the imaging unit 105 is stored in the recorded image memory 107 by the recording memory control unit 108. The image correcting unit 109 sequentially reads digital images from the recorded image memory 107 and optimally corrects the level and color tone.

  Next, the image compression means 110 compresses the corrected digital image so as to obtain a desired data rate. The recording control unit 106 appropriately controls the operation timing of each unit of the compression recording unit 111. At the time of recording, the image recording / reproducing unit 112 receives the compressed image data compressed to a desired data rate by the compression recording unit 111 and writes the compressed image data to the recording medium 119.

  Thus, at the time of recording, the image output from the image sensor 102 is output as it is from the imaging unit 105 without being interpolated, and this image is compressed by the compression recording unit 111 and recorded as compressed image data on the recording medium 119. To do.

  At the time of reproduction, the reproduction unit 117 receives the compressed image data read from the recording medium 119 by the image recording / reproduction unit 112, decompresses the image by the image expansion unit 115, and inputs an input signal to the image compression unit 110 in the compression recording unit 111. That is, a decoded image substantially equivalent to the image before compression at the time of recording is reproduced. However, when the image compression means 110 is an irreversible compression method, the input signal of the image compression means 110 and the output signal of the image expansion means 115 do not completely match.

  The pixel shift interpolation synthesis unit 116 performs pixel interpolation synthesis processing on the decoded image. That is, in the same manner as pixel interpolation synthesis processing in a camera recorder using conventional pixel shift, after performing oversampling of the RGB channel signals, synthesis is performed, and an image having a resolution twice that of the image sensor is obtained. obtain. As described above, the imaging compression / recording / reproducing apparatus of the present invention does not perform the pixel interpolating / synthesizing process at the time of recording. Process.

  With the above configuration, in a camera recorder intended to capture and compress and record using a low-resolution imaging device by pixel shifting, up-conversion by pixel shifting to improve resolution is performed by the pixel shifting interpolation synthesizing means 116 during playback. Therefore, the image input to the image compression unit 110 has a low resolution data rate. For this reason, the compression rate when the image compression unit 110 compresses to a desired data rate is minimized. When the compression method of the image compression unit 110 is irreversible compression, the smaller the compression rate, the higher the image quality. Therefore, it becomes possible to record a video signal captured with high image quality on the recording medium 119. Further, since the processing clock of the compression recording unit may be a speed for processing a low-resolution image, it is possible to reduce power consumption. For the up-conversion processing by pixel shift in the pixel shift interpolation / combination means 116 at the time of reproduction, the same method as that performed at the time of recording in the conventional example can be applied.

  Further, since only the number of low-resolution pixels is recorded in the compressed image data, the processing clock of the image expansion means may be a speed for processing a low-resolution image, so that power consumption can be reduced.

  In the present embodiment, the pixel shifting direction has been described as the horizontal direction, but the vertical direction or both horizontal and vertical directions may be used. In that case, the arrangement of the image pickup elements may be shifted in such a direction, and the process may be performed in accordance with the arrangement of the image pickup elements in the interpolation / synthesis process during reproduction.

  For convenience of explanation, the compressed image data has been described in the form of reading what is recorded on the recording medium. However, the same processing can be performed when the compressed image data is transmitted over a network or the like. It may be a statue.

(Embodiment 2)
Next, an imaging compression recording / reproducing apparatus according to the second embodiment of the present invention will be described with reference to FIGS. FIG. 6 is a block diagram showing the configuration of the recording side of the camera recorder as the imaging compression recording / reproducing apparatus according to the second embodiment of the present invention. The difference between the camera recorder of the second embodiment and the first embodiment is that the compression recording unit 611 has sample phase information multiplexing means 613, and information relating to the phase shift of the image sensor in the imaging unit 105, in other words, an imaging pixel sample. Is multiplexed with the compressed image data. Reference numeral 102 denotes an image sensor arranged by shifting pixels, 103 denotes an A / D conversion means, 104 denotes an imaging section control means, 105 denotes an imaging section, 106 denotes a recording control means, 107 denotes a recording image memory, 108 denotes a recording memory control means, 109 is an image correcting unit, 110 is an image compressing unit, 611 is a compression recording unit, 112 is an image recording / reproducing unit, and 613 is a sample phase information multiplexing unit. It is characterized in that a sample phase information multiplexing unit 613 is added after the image compression unit 110. Various methods such as horizontal, vertical, or both horizontal and vertical are conceivable for pixel shifting of the imaging unit 105. It is possible to ensure implicit compatibility with the playback side by adopting a fixed method, but in the case of variable, the phase relationship between the pixels on the recording side and the playback side It is necessary to share this information. Therefore, in the present invention, the sample phase information multiplexing means 613 multiplexes and records and transmits the phase information of the pixel sample on the compressed image data.

  The phase information of the sample may be multiplexed in any format, but as an example, it can be considered that the phase type is encoded and transmitted. As a specific example of coding and multiplexing the phase information of the sample, FIG. The sample phase type is defined by the parameter syntax of VUI (Video Usability Information) described in the H.264 / MPEG4-AVC (hereinafter referred to as H.264) standard. H. In H.264, these types can be specified by chroma_sample_loc_type_top_field and chroma_sample_loc_type_bottom_field. For example, assuming that the position of the G sample is the position of the luminance sample and the sample positions of x and □ are used as a reference, the positions of the R sample and B sample shifted horizontally by 1/2 pixel are type 3 samples. When both horizontal and vertical are shifted by 1/2 pixel, it becomes a type 1 sample. In an interlaced signal, when a frame is composed of two fields, the top field that becomes the upper line is chroma_sample_loc_type_top_field, and the bottom field that is the lower line is shifted by specifying the phase number of the sample phase in chroma_sample_loc_type_bottom_field. Can be transmitted.

  As described above, H.P. H.264 specifies that the phase information of the sample can be multiplexed with emphasis on compatibility between the recording side and the reproduction side. However, the phase information of the pixel shift according to the present invention is recorded and transmitted using the VUI parameter syntax. Is possible. As in this example, the compression recording method of the camera recorder to which the pixel shift of the present invention is applied can be applied while maintaining compatibility with the international standard method.

  8 is almost the same as the reproduction side of FIG. 1, 811 is a sample phase information separation means, 112 is an image recording / reproduction unit, 113 is a reproduction control means, 114 is a reproduction image memory, 115 is an image expansion means, and 116 is a pixel. Shift interpolating and synthesizing means, 817 is a reproduction unit, 118 is reproduction memory control means, and 119 is a recording medium. A feature is that a sample phase information separating unit 811 is added after the image recording / reproducing unit 112.

  The sample phase information separation means 811 separates the phase information of the sample from the read compressed image data, and instructs the pixel shift interpolation synthesis means 116 what type of interpolation synthesis is performed. The compression encoding method is H.264. In the case of H.264, the processing of the sample phase information separating means detects VUI parameter, and extracts the fields of chroma_sample_loc_type_top_field and chroma_sample_loc_type_bottom_field according to the syntax of parameter.

  In the second embodiment, the H.264 standard is used. The example of H.264 has been mainly described, but it is possible to multiplex sample phase information into compressed image data as application specific data even in other compression coding systems.

  According to the imaging compression / recording / reproducing apparatus of the present invention, it is possible to suppress deterioration in the image quality of a reproduced image and to reduce power consumption. It can also be applied to a camera recorder or the like.

FIG. 2 is a block diagram showing a basic configuration of the first embodiment. 1CCD and 3CCD system explanatory diagram Conceptual diagram of pixel shift in 3CCD Imaging sample diagram of the imaging unit using pixel shifting Illustration of up-conversion using pixel shift Block diagram on the recording side of the second embodiment H. A diagram showing sample phase types defined by H.264 VUI parameters Block diagram on the playback side in the second embodiment

Explanation of symbols

DESCRIPTION OF SYMBOLS 101 Pixel shifting means 102 Image pick-up element 103 A / D conversion means 104 Imaging part control means 105 Imaging part 106 Recording control means 107 Recording image memory 108 Recording memory control means 109 Image correction means 110 Image compression means 111 Compression recording of Embodiment 1 Unit 112 image recording / playback unit 113 playback control unit 114 playback image memory 115 image expansion unit 116 pixel shift interpolation synthesis unit 117 playback unit of the first embodiment 118 playback memory control unit 119 recording medium 611 compression recording unit 613 of the second embodiment Sample phase information multiplexing means 811 Sample phase information separation means 817 Reproduction unit of embodiment 2

Claims (2)

An imaging unit in which at least one of the plurality of imaging elements is arranged at a position shifted in phase from a position corresponding to another imaging element, a compression recording unit that compresses an image from the imaging unit,
An image recording / reproducing unit for recording the compressed image data from the compression recording unit on a recording medium, and reading the compressed image data from the recording medium;
Image expansion means for expanding the compressed image data from the image recording / reproducing unit;
An imaging compression / recording / reproducing apparatus comprising: an interpolating and synthesizing unit that performs interpolation and synthesizing processing on the image from the image expansion unit to increase the resolution. Sample phase information multiplexing means for multiplexing sample phase information, which is information related to the phase shift of the image sensor, on the compressed image data;
Sample phase information separating means for separating the sample phase information multiplexed on the compressed image data;
The imaging compression recording / reproducing apparatus according to claim 1, wherein the interpolation synthesis unit selects a pixel shift interpolation synthesis method based on the sample phase information separated from the compressed image data.

Images