KR20100124855A - Dynamic image processing device, dynamic image processing method, and dynamic image processing program - Google Patents

Abstract

When performing demosaicing and compression stream generation from image data output from a color image sensor, a moving picture processing device and a moving picture processing that can reduce storage capacity and bandwidth for frame rearrangement, and can realize low power consumption and low cost. Provide a method. A frame sequence rearranged from the image pickup unit 110 and arranged in a first data format according to the processing order of the compressed stream; and a frame sequence alternately arranged in the frame rearrangement unit 130. The moving picture which encodes and compresses the color data generation part 140 which converts the 1st data format in a color image into a color image, and the frame sequence converted into the color image based on the difference between several frames before and after time. The compression unit 150 is provided.

Description

A moving image processing apparatus, a moving image processing method, and a moving image processing program {DYNAMIC IMAGE PROCESSING DEVICE, DYNAMIC IMAGE PROCESSING METHOD, AND DYNAMIC IMAGE PROCESSING PROGRAM}

The present invention relates to a moving image processing apparatus, a moving image processing method, and a moving image processing program for generating a compressed stream by encoding a frame sequence of a moving image which is divided into a plurality of frames in time series from a color image sensor.

Background Art [0002] Conventionally, in a video camera for shooting a moving image, an image of a subject is formed on an image pickup device through a lens, the image of the subject is photoelectrically converted by the image pickup device, and a plurality of frame data are generated in time series. BACKGROUND OF THE INVENTION Moving picture processing techniques (MPEGs) for predicting motion (so-called inter-frame prediction) are generated.

In a moving image processing technique, in general, a motion vector indicating a vector of how much a pixel of a current frame has moved compared to a pixel of a previous frame is estimated, and instead of transmitting the entire image, the difference between these motion vectors is calculated. By transmitting, the transmission information is compressed.

In detail, in the moving picture processing technique represented by MPEG, the type of the frame includes the difference between the I frame that encodes the image signal in the frame as it is without using inter-frame prediction, and the difference between the image signal of the preceding reference frame in time. A P frame to be encoded and a B frame to encode a difference between a temporally preceding reference frame and a subsequent reference frame are set, and the arrangement and repetition period of these frames are set.

For example, in the case of MPEG of M = 3, as shown in Fig. 2 (b), GOP (Group of Pictures) has I, B, B, P, B, B, P, B, B, P, B, B, I... It consists of frames that are continuous in time series, such as.

On the other hand, since the input frame order and the encoded and transmitted order are different, it is necessary to rearrange the order of the frames at the time of encoding. For rearrangement, a frame buffer memory for temporarily storing B frames is required. For example, in the case of M = 3 in which two B frames are inserted between I and P, two frames of memory are required to store them (see Patent Document 1, for example).

In addition, as a single plate type imaging element, a plurality of photoelectric conversion elements are formed in a matrix form, and on the entire surface thereof, each color filter of R (red) G (green) B (blue) is provided corresponding to the photoelectric conversion element. There is a technique of generating a color image by adding signal processing to a single color image signal output through the color filter.

In the image output through the single-plate type imaging element, each pixel is a color mosaic image having only color information of a single color, and in order to generate a color image, red (R), green (G), and blue (B) pixels are generated. It is necessary to provide a plurality of color information such as).

Here, in the image processing using a single-plate type imaging element, each pixel performs demosaicing processing (also referred to as color interpolation processing) based on a color mosaic image having color information of only one of R, G, and B components. A color image is generated from the mosaic image. Here, the demosaicing process interpolates other color information lacking in each pixel of a color mosaic image by using color information of other pixels around the pixel, so that each pixel has all colors of R, G, and B components, respectively. It is a process of generating a color image having information (so-called color interpolation processing).

[Prior Art Literature]

[Patent Documents]

Patent Document 1: Japanese Patent Application Laid-Open No. 10-056652

However, according to the conventional moving picture processing technique, when the frames are alternately arranged, in general, the image data stored in the frame buffer is a color image having a plurality of color information for each pixel, and an image requiring demosaicing processing. Regarding data, a moving picture processing technique that is effective for reducing memory capacity has not been disclosed.

Here, the present invention is a storage capacity for rearrangement of frames when demosaicing (so-called color image generation processing in the present invention) and compressed streams are generated from image data output from a color image sensor. In addition, an object of the present invention is to provide a moving image processing apparatus, a moving image processing method, and a moving image processing program that can reduce power consumption and reduce power consumption and cost.

The invention according to claim 1, which is made to achieve this object, is encoded by an inter-frame prediction method from a frame sequence of a moving image which is divided into a plurality of frames in time series from a color image sensor and output in a first data format. A moving picture processing apparatus for generating a compressed stream of a color image, comprising: a frame rearrangement unit for arranging a frame sequence having the first data format in correspondence with the processing order of the compressed stream; A color image generation unit for converting the first data format in the converted frame sequence into a color image and the frame sequence converted into the color image based on a difference between a plurality of frames before and after A moving picture compression unit for compressing is provided. .

According to the moving picture processing apparatus according to claim 1, the frame rearrangement unit for arranging the frame sequence having the first data format in correspondence with the processing order of the compressed stream and the frame sequence rearranged in the frame rearrangement unit A frame includes a color image generation unit for converting one data format into a color image, and a moving image compression unit for encoding and compressing the frame sequence converted into a color image based on a difference between a plurality of frames before and after time. It is possible to reduce the memory capacity and the bandwidth required for rearrangement (memory capacity and bandwidth of the frame buffer), and to reduce the amount of heat generated in the moving image processing circuit due to lower power consumption, lower cost, and lower power consumption.

That is, since the frame data used for the frame rearrangement is the first data format output from the image sensor before generating the color image, the memory capacity required for the rearrangement of the frame is lower than that of using the color image data during the frame rearrangement. Bandwidth can be saved.

In addition, the moving image processing apparatus according to claim 1 includes, as in the invention according to claim 2, a plurality of photoelectric conversion elements in which the color image sensor is arranged in a matrix, and a color of a plurality of colors corresponding to each of the photoelectric conversion elements. A single-plate color image sensor having a filter and outputting pixel information of monochromatic light of a plurality of color lights for each photoelectric conversion element, wherein said first data format is a color mosaic image having color information of monochromatic light for each pixel, and said color image Since the generation unit is configured to perform demosaicing processing for generating pixel information of a plurality of colors of light for each of the pixels, it is necessary to rearrange the frames when performing demosaicing processing and compression stream generation from image data output from a color image sensor. Memory capacity and bandwidth can be reduced.

In the moving image processing apparatus according to claim 1, as in the invention according to claim 3, the color image sensor is constituted by a plurality of color image sensors having different spectral sensitivity distributions, and each light receiving surface is shifted in the pixel arrangement direction. The color data generation unit is arranged such that the first data format is output from each of the plurality of color image sensors, and the color image generation unit is configured to synthesize image data of the plurality of color image sensors to increase resolution. Therefore, when the color image is generated and the compressed stream is generated from the image data output from the color image sensor, the storage capacity and the bandwidth required for frame rearrangement can be reduced.

Furthermore, the moving image processing apparatus according to any one of claims 1 to 3, as in the invention according to claim 4, includes a non-F frame in which the plurality of frames are encoded without referring to a frame that is temporally followed, and a trailing time. It can be applied when the frame rearrangement unit is configured to give a delay according to the type of the input NonF frame and the F frame and output the frame.

In addition, the moving picture processing apparatus according to claim 4 includes, as in the invention according to claim 5, the NonF frame further includes an I frame that encodes an image signal in a frame as it is without using the inter-frame prediction. The frame is made up of a P frame that encodes a difference from an image signal of a reference frame, and the F frame is a B frame that encodes a difference between a reference frame temporally preceding and a reference frame that follows. The present invention can be applied when it is configured to give a delay according to the type of I frame, P frame, and B frame to be output.

The moving picture processing apparatus according to any one of claims 1 to 3 is configured such that, as in the invention according to claim 6, the frame rearrangement unit outputs a delay with respect to some of the plurality of frame types. It is enough.

In the moving picture processing apparatus according to any one of claims 1 to 6, as in the invention according to claim 7, the frame rearrangement unit includes at least two frames in the image data of the first data format. It is preferable that a frame buffer for storing image data is provided for each. As a result, in M = 3 MPEG, which is a general moving picture processing technique, two B frames arranged between an I frame and a P frame can be stored.

In the moving image processing apparatus according to any one of claims 1 to 7, as in the invention according to claim 8, the image generation unit preferably includes an image deformation processing unit for performing image deformation of the color image. . As a result, the frame buffer of the frame rearrangement unit can be used as the frame buffer required for the color image deformation process, so that the storage capacity and the bandwidth required for the separate image deformation can be reduced.

The moving picture processing apparatus of claim 8 is preferably configured such that the frame rearrangement unit outputs the image data in a frame in a non-rasterized order when the image is deformed, as in the invention of claim 9. . As a result, image modification such as digital zoom, camera shake, and aberration correction can be performed using the frame buffer of the frame rearrangement unit without separately preparing a frame buffer for image distortion.

Furthermore, the moving image processing apparatus in any one of Claims 1-9 is equipped with the 2nd image generation part which produces | generates the color image output to a finder like the invention of Claim 10, and produces | generates the 2nd image When the color image is generated in the finder by generating a color image without performing frame rearrangement in the frame rearrangement unit, the image is displayed in the order of display in the finder, and there is no delay due to the frame rearrangement. Good trackability of the output from the image sensor can be maintained.

The moving image processing apparatus according to any one of claims 1 to 10, as in the invention according to claim 11, transmits a frame sequence output from the color image sensor to the color image generating unit without passing through the frame rearranging unit. And a rearrangement means for outputting the rearrangement unit, wherein the frame sequence output from the color image sensor and the frame sequence output from the frame rearrangement unit of the frame sequence input to the color image generation unit can be switched. If necessary, it is possible to select whether or not to rearrange the frame, thereby improving convenience.

Like the invention described in claim 12, the camcorder is operated by using the moving picture processing device and color image sensor according to any one of claims 1 to 11, and a recording device for storing moving picture information compressed by the moving picture compression unit. Good to configure. Accordingly, in the camcorder, when generating a compressed stream of a frame sequence of a moving picture, the memory capacity and bandwidth (frame capacity and bandwidth of the frame buffer) required for frame rearrangement can be reduced, resulting in low power consumption, low cost, and moving picture processing. Reduction of the amount of heat generated in the circuit can be realized.

In addition, as in the invention described in claim 13, the video processing apparatus and color image sensor according to any one of claims 1 to 11, and a transmission device for transmitting the moving picture information compressed by the video compression unit to an external device are remotely used. It is good to configure a monitor camera. As a result, in the remote monitor camera, when generating a compressed stream of a moving picture frame sequence, the memory capacity and bandwidth (frame capacity and bandwidth of the frame buffer) required for frame rearrangement can be reduced, resulting in low power consumption and low cost. In this way, it is possible to reduce the amount of heat generated in the moving image processing circuit.

Next, the invention described in claim 14 is a compression of a color moving picture coded by an inter-frame prediction method from a frame sequence of a moving picture divided into a plurality of frames in time series from a color image sensor and output in a first data format. A moving picture processing method for generating a stream, comprising: a frame rearrangement step of rearranging a frame sequence formed of the first data format in correspondence with a processing order of the compressed stream; and a frame rearranged in the frame rearrangement step A color image generating step of converting the first data format in the sequence into a color image, and encoding and compressing the frame sequence converted into the color image based on the difference between a plurality of frames before and after time. A moving picture compression step is used.

According to the moving picture processing method according to claim 14, in the frame rearrangement step of rearranging and arranging the frame sequence of the first data format in correspondence with the processing order of the compressed stream, the frame sequence rearranged in the frame rearrangement step By using a color image generating step of converting the first data format of the image into a color image and a moving image compression step of encoding and compressing a frame sequence converted into a color image based on a difference between a plurality of frames before and after time. As in the invention described in claim 1, the memory capacity and bandwidth (frame capacity and bandwidth of the frame buffer) required for frame rearrangement can be reduced, and the amount of heat generated in the moving picture processing circuit according to low power consumption, low cost, and low power consumption. Can be reduced.

Next, the invention described in claim 15 is a compression of a color moving picture coded by an inter-frame prediction method from a frame sequence of a moving picture which is divided into a plurality of frames in time series from a color image sensor and output in a first data format. A moving picture processing program for generating a stream, comprising: a frame rearrangement step of rearranging a frame sequence having the first data format in correspondence with a processing order of the compressed stream, and a frame sequence rearranged in the frame rearrangement step A color image generation step of converting the first data format in a color image into a color image, and a moving image encoding and compressing the frame sequence converted into the color image based on a difference between a plurality of frames before and after Run the compression step on the computer.

According to the moving picture processing program according to claim 15, in the frame rearrangement step of arranging the frame sequence having the first data format in correspondence with the processing order of the compressed stream, and in the frame sequence rearranged in the frame rearrangement step, The computer generates a color image generating step of converting the first data format of the image into a color image, and a moving image compression step of encoding and compressing the frame sequence converted into the color image based on a difference between a plurality of frames before and after in time. By executing the same, the memory capacity and the bandwidth (memory capacity and bandwidth of the frame buffer) required for frame rearrangement can be reduced as in the invention of claim 1, and the moving image processing circuit according to the low power consumption, low cost, and low power consumption can be reduced. Can reduce the amount of heat generated.

The moving picture processing apparatus, the moving picture processing method, and the moving picture processing program of the present invention arrange a frame sequence made up of a first data format in correspondence with the processing order of a compressed stream, and then, in the alternately arranged frame sequence, Since the data format of 1 is converted into a color image and the frame sequence converted into the color image is encoded and compressed based on a difference between a plurality of frames before and after time, the memory capacity and bandwidth required for frame rearrangement ( The memory capacity and bandwidth of the frame buffer) can be reduced, thereby reducing the amount of heat generated in the moving image processing circuit due to the low power consumption, low cost, and low power consumption.

Furthermore, a camcorder is constructed by using a moving picture processing device and a color image sensor according to the present invention and a recording device for storing moving picture information compressed by the moving picture compression unit of the moving picture processing device. When generating a compressed stream of a sequence, the memory capacity and bandwidth (frame memory capacity and bandwidth of the frame buffer) required for frame rearrangement can be reduced, thereby realizing low power consumption, low cost, and reduced heat generation in a moving image processing circuit. have.

In addition, a remote monitor camera is constructed by using a moving image processing apparatus and a color image sensor according to the present invention and a transmitting device for transmitting moving image information compressed by the moving image compression unit of the moving image processing apparatus to an external device. In the high-performance camera, when generating a compressed stream of a moving picture frame sequence, the memory capacity and bandwidth (frame capacity and bandwidth of the frame buffer) required for frame rearrangement can be reduced, thereby reducing power consumption, cost, and moving picture processing circuits. Reduction of the amount of heat generated can be realized.

Fig. 1 is a block diagram of a first embodiment of the present invention in which (a) shows a configuration of an imaging device 1A to which the moving image processing device of the present invention is applied. It is explanatory drawing of the imaging part in this.
2 is an explanatory diagram of an operation of the frame rearrangement unit in the imaging device 1A of the first embodiment.
Fig. 3 is a block diagram showing the configuration of the imaging device 1B to which the moving image processing apparatus of the present invention is applied, in which the second embodiment of the present invention is shown, (b) in the imaging device 1B. (C) is explanatory drawing of the image data synthesis process in the imaging device 1B.
4 is an explanatory diagram of an operation of the frame rearrangement unit in the imaging device 1B of the second embodiment.
Fig. 5 is a block diagram showing the configuration of an imaging device 1C to which the moving image processing apparatus of the present invention is applied in the third embodiment of the present invention.
Fig. 6 is a block diagram showing the configuration of the imaging device 1D to which the moving image processing apparatus of the present invention is applied in the fourth embodiment of the present invention.

(1st embodiment)

Next, 1st Embodiment of this invention is described using FIG. 1, FIG.

In FIG. 1, (a) is a block diagram which shows the structure of the imaging device 1A of the 1st Embodiment to which the moving image processing apparatus of this invention was applied, (b) is the imaging part in the imaging device 1A. It is explanatory drawing. 2 is explanatory drawing of operation | movement of the frame rearrangement part in the imaging device 1A of the said 1st Embodiment.

As shown in Fig. 1 (a), the imaging device 1A is, for example, a video camcorder from the imaging section 110 and the imaging section 110 that sequentially converts photographed image signals into analog electrical signals and outputs them. AFE 120 for converting the outputted analog electric signal into digital data and outputting the digital data output from the AFE 120 for each frame, and changing the order of the frames and outputting them (so-called rearrangement of frames). The frame rearrangement unit 130, the image generation unit 140 for converting the image data of each frame output from the frame rearrangement unit 130 into a color image, and the frame sequence of the color image output from the image generation unit 140, A moving picture compression unit 150 for compressing as a moving picture and outputting a compressed stream, and a recording unit for recording the compressed stream output from the moving picture compression unit 150, for example, in a flash memory or an optical / magnetic recording medium (1). 60) or the like.

In addition, the imaging device 1A includes a central processing unit (CPU) or a read only memory (ROM) (not shown), and the CPU performs each processing of the imaging device 1A in accordance with a control program stored in the ROM. To control.

The imaging unit 110 is a single-plate type color imaging element (it is a single-plate color image sensor in the present invention), in which a plurality of photoelectric conversion elements are arranged in a matrix, and the front side thereof corresponds to the photoelectric conversion element. As shown in Fig. 1 (b), a color filter composed of Bayer arrays of three primary colors of R (red), G (green), and B (blue) is provided, and the amount of light of a single color passing through the filter unit of each color is shown. Is converted to an electrical signal. In addition, in the Bayer arrangement, as shown in Fig. 1 (b), a column in which the G color filters are arranged in a check pattern, the G color filter and the R color filter are alternately arranged, and the G color filter and the B color filter are alternately arranged. The arranged rows are alternately arranged.

The AFE 120 performs a correlated double sampling on the analog image signal output from the image capturing unit 110 and removes the noise, and a correlated double sampling circuit (CDS) and an analog image input through the correlated double sampling circuit. A frame which is configured by an automatic gain control (AGC) for amplifying a signal, an A / D converter for converting an analog image signal input through the variable gain amplifier into a digital image signal, and the like and output from the imaging unit 110. The analog image signal is converted into a digital image signal corresponding to the Bayer array and output to the frame rearrangement unit 130.

Here, when the imaging unit 110 is composed of 2M pixels (pixels) and the accuracy of A / D conversion in the AFE 120 is 8 bits, the image signal per frame output from the AFE 120 is 16 Mbit. (2M * 8bit = 16Mbit).

The frame rearranging unit 130 converts the frame order of the digitized Bayer data input in the display order into a transmission order corresponding to the processing order of the moving picture compression unit 150.

For example, in the moving picture compression unit 150, when generating an M = 3 MPEG stream, as shown in Fig. 2 (a), k≡2 ( mod 3), an I frame or a P frame, and k≡0 or 1 (mod 3), a B frame. mod is an operator that divides a predetermined numerical value k by a mod number and finds the surplus.

Here, the frame rearrangement unit 130 outputs the delay to the image generation unit 140 by giving three frames more delay than the I frame or the P frame. In this embodiment, the frame rearrangement is realized by providing the frame buffers 130a to 130d for four frames, and allocating frame buffers 130a to 130d suitable for input and output. The frame buffers 130a to 130d each have a capacity of 16 Mbits and have a total of 64 Mbits of memory.

The image generating unit 140 sequentially converts the frame output from the frame rearranging unit 130 into a color image from a Bayer image. A color image signal is generated as YCrCb = 4: 2: 2. At this time, one value per pixel = 8bit / pix in a Bayer image, but two values per pixel = 16bit / pix in a color image, and the color image signal per frame is 32 Mbit (16bit / pix). * 2Mpix = 32Mbit). In the color image signal described above, Y is luminance, Cr is color difference of R with respect to Y, and Cb is color difference with respect to Y.

In this embodiment, the frame buffers 130a to 130d can be referred to the frame rearranging unit 130 for any frame. For this reason, the image generation unit 140 can perform non-rasterous access to the Bayer image.

Accordingly, the imaging device 1A according to the present embodiment uses digital frame zooms, camera shake, chromatic aberration, distortion aberration correction, and the like using the frame buffers 130a to 130d without separately preparing a buffer for image distortion. Can also be modified.

The moving image compression unit 150 compresses the frame sequence of the color image output from the image generation unit 140. In the present embodiment, MPEG compression is performed at intervals of three frames (M = 3) in a period M in which I frames or P frames appear. In the MPEG moving picture compression unit 150, two prediction memories 150a and 150b for inter-frame prediction are provided. The compressed data of each frame is output to the recording unit 160 as a series of compressed streams.

Next, details of an operation for generating a compressed stream will be described. First, in the imaging unit 110, exposure is performed every predetermined time (e.g., 1/60 second), and for each exposure, the AFE 120 is sequentially performed using the exposure amount in each photoelectric conversion element as an analog electric signal. ) At this time, the set of analog electric signals output from each photoelectric conversion element becomes a Bayer image signal corresponding to the color filter of the Bayer array. In addition, because of moving image shooting, exposure of the next frame is started at the same time as one exposure is completed, and analog image signals are sequentially outputted. Then, using each image as a frame, a moving image is formed by the frame sequence.

Next, the analog electric signal output from the imaging unit 110 is converted into a digital signal in the AFE 120. This digital signal is a Bayer image signal corresponding to the Bayer arrangement of the imaging unit, which is the first data format. The Bayer image signal output from the AFE 120 is sequentially output in the exposure order of the imaging unit 110 (in this specification, this output order is referred to as display order).

Next, as shown in Fig. 2A, k = 0, 1, 2,. Frame numbers are denoted by, and frames k≡2 (mod 3) are I frames or P frames, frames k≡0 (mod 3) or k≡1 (mod 3) are B frames, respectively. The frame is input to the frame rearranging unit 130. At this time, the Bayer image signals output in the display order are input to the frame rearranging unit 130 in raster order.

Next, in the frame rearrangement unit 130, the input frame k input from the AFE 120 is placed in any one of the frame buffers (the input buffers in Fig. 2A) 130a to 130d. It is stored and output to the image generating unit 140 in a different order from the stored order (this is the frame rearrangement step in the present invention).

In detail, as shown in Figs. 2A and 2B, a delay of one frame is performed for an I frame or a P frame, and a delay of four frames is performed for a B frame. At this time, in the present invention, regardless of the absolute amount of delay, M frames are used for M frames rather than I frames or P frames (M is a period in which I frames or P frames appear in the MPEG video processing technique). It is important to give a lot of delay. As a result, the frame order input to the image generating unit 140 becomes the transmission order.

Next, in the image generating unit 140, each frame input through the frame rearranging unit 130 is converted into a color image signal from the Bayer image signal (the color image generating step in the present invention). Here, in addition to color interpolation processing or demosaicing processing which are generally known, image processing such as color conversion, edge enhancement of an image, noise suppression, tone curve processing, and the like is performed, so as to obtain an image quality suitable for viewing.

In addition, the image generating unit 140 reads the Bayer image signals stored in the frame buffers 130a to 130d of the frame rearranging unit 130 in a non-rasterized order, and performs image modification such as enlargement, reduction, and rotation. Can be. The color image signal converted by the image generating unit 140 is output to the moving image compression unit 150.

Next, the moving picture compression unit 150 performs a compression process of the frame sequence of the color image input from the image generating unit 140 (the moving picture compression step in the present invention).

In this embodiment, MPEG compression processing of M = 3 is performed, and a different compression method is used for three types of I frame, P frame, and B frame. In addition, NonF frames in the present invention correspond to I frames and P frames, and F frames in the present invention correspond to B frames.

Specifically, the I-frame is encoded in the frame without using the predictive memory, outputted to the recording unit 160, and updated in the predictive memory 150a or 150b (i.e., updated first). To update).

The P frame is encoded using inter-frame prediction from the frame whose update history in the prediction memory 150a or 150b is stored on the new side, output to the recording unit 160, and the prediction memory 150a or 150b. The update history of) is input to the old one and updated. When a P frame is input to the image generating unit 140, an I frame or a P frame input immediately before three frames is stored in either new side of the prediction memory 150a or 150b.

The B frame is encoded using interframe prediction from two frames stored in the prediction memories 150a and 150b and output to the recording unit 160. When a B frame is input to the moving picture compression unit 150, an I frame or a P frame input immediately before the image generating unit 140 is stored in either of the prediction memories 150a or 150b. On the other side (i.e., the old one of the prediction memory 150a or 150b), an I frame or P frame input immediately before the three frames is further stored.

However, the order of the frames input to the image generating unit 140 is the transfer order rearranged by the frame rearranging unit 130. In order of display order, imaging is performed on either of the prediction memories 150a or 150b. In the unit 110, the I frame or P frame exposed immediately before (of the B frame to be processed) is stored, and on the other side, the imaging unit 110 (of the B frame to be processed) is stored. Immediately exposed I frames or P frames are stored (so-called temporally preceding reference frames and temporally following reference frames in the present invention are stored), whereby bidirectional prediction is realized in B frames. .

Next, in the recording unit 160, the compressed stream of the image sequence encoded by the moving image compression unit 150 is recorded.

In addition, in the imaging device 1A, a transmitting unit may be provided in place of the recording unit 160. At this time, the transmitter is connected to another video device via a wired or wireless communication means. The transmitting unit transmits the image sequence encoded by the moving image compression unit 150 to another video device as a compressed stream. Thereby, a camera for remote monitoring can be comprised.

As described above, the imaging device 1A according to the first embodiment generally performs the frame rearrangement process required for MPEG compression on a Bayer image instead of a color image, thereby determining the memory capacity and the bandwidth required for the rearrangement of the frame. Can be reduced. Further, in the image generating unit 140, image deformation is performed using the frame buffers 130a to 130d used for rearranging the frames, so that image deformation can be performed without requiring a memory for image deformation.

That is, when the frame rearrangement unit and the moving image compression unit are configured behind the image generation unit as in the related art, in order to switch and arrange the color images in the frame rearrangement unit, a frame buffer having a larger memory capacity is required as compared with the first embodiment. Thus, the required band also becomes large. In addition, according to the prior art, generally, a frame buffer for image deformation is required separately in the image generating unit. In addition, in the imaging unit 110 of the present embodiment, a pixel interleaved array (PIA) array of 1 million pixels (1M pixels) may be used, and the image generating unit 140 may perform high pixel processing in addition to demosaicing. . In this case, the application of the present invention can further reduce the memory capacity and bandwidth required for frame rearrangement. Here, the PIA array is a pixel array in which the square lattice is rotated 45 degrees. Compared with the Bayer array, the condensing area can be widened, and a resolution close to 2M pixels of the Bayer array can be obtained with 1M pixels.

(2nd embodiment)

Next, 2nd Embodiment of this invention is described using FIG. 3 and FIG. In FIG. 3, (a) is a block diagram which shows the structure of the imaging device 1B of 2nd Embodiment to which the moving image processing apparatus of this invention was applied, (b) is the imaging part in the imaging device 1B. Explanatory drawing, (c) is explanatory drawing of the image data synthesis process in the imaging device 1B. 4 is an explanatory view of the operation of the frame rearrangement unit in the imaging device 1B of the embodiment.

In addition, since the imaging device 1B in 2nd Embodiment is basically the same structure as the imaging device 1A shown in 1st Embodiment, the same code | symbol is attached | subjected about the structural part which is common, and detailed description is carried out. It abbreviate | omits and demonstrates the below-mentioned part.

As shown in Fig. 3A, the imaging device 1B is, for example, a video camcorder, from the imaging section 111 and the imaging section 111, which sequentially convert photographed image signals to analog electric signals and output them. AFE 120 for converting the outputted analog electric signal into digital data and outputting the digital data output from the AFE 120 for each frame, and changing the order of the frames and outputting them (so-called rearrangement of frames). The frame rearranging unit 131, the image generating unit 141 for converting each frame data output from the frame rearranging unit 131 into color data corresponding to the color image, and the frame of the color image output from the image generating unit 141. The moving picture compression unit 150 for compressing a sequence as a moving picture and outputting a compressed stream, and the compressed stream output from the moving picture compression unit 150 are stored in, for example, a flash memory or an optical or magnetic recording medium. It is comprised by the recording part 160 etc. which record.

The imaging part 111 is comprised by three imaging elements 111a, 111b, and 111c from which a spectral sensitivity distribution differs, and each light receiving surface is arrange | positioned so that it may shift | deviate to a pixel arrangement direction.

In detail, in the imaging section 111, a color separation prism for separating the color light of R, G, and B is disposed on the optical path of the imaging optical system, and the imaging elements 111a, 111b, and 111c are formed on the imaging surface of each color light. Is a 3rd edition color camera.

As shown in Fig. 3 (b), by positioning the positions of the imaging elements 111a, 111b, and 111c with the sub-pixel precision shifted, a resolution higher than the number of pixels of the imaging elements 111a, 111b, and 111c is obtained. It is configured to.

That is, the imaging part 111 is comprised by the color image sensor which consists of several imaging elements 111a, 111b, and 111c from which a spectral sensitivity distribution differs, and each light receiving surface is arrange | positioned so that it may shift | deviate to a pixel arrangement direction.

In the second embodiment, a resolution of 2M pixels of full HD (full high definition) is obtained, and each of the imaging elements 111a, 111b, and 111c is composed of 0.5M pixels (pixels), and R and G for G. It is assumed that B is arranged to be shifted by half pixels ((Py / 2) and (Px / 2)) vertically and horizontally.

In the same manner as in the first embodiment, the AFE 120 converts the analog image signal output from the imaging unit 111 into a digital image signal consisting of three planes of R, G, and B and rearranges the frame rearrangement unit 131. Output to.

If the accuracy of the A / D conversion in the AFE 120 is 8 bits, the image signal per frame output from the AFE 120 is 12 Mbits (0.5 M * 8 bits * 3 planes = 12 Mbits).

As in the first embodiment, the frame rearrangement unit 131 converts the frame order of the digitized pixel shift data input in the display order into the transmission order corresponding to the processing order of the moving picture compression unit 150. In the second embodiment, the frame rearrangement unit 131 is provided with frame buffers 131a and 131b for two frames, and the frame buffers 131a and 131b are appropriately adapted in accordance with the input from the imaging unit 111. By rearranging or outputting the input as it is, rearrangement of frames is realized. The frame buffers 131a and 131b each have a capacity of 12 Mbits and a total of 24 Mbits of memory.

The image generating unit 141 sequentially performs high resolution processing on the frames output from the frame rearranging unit 131 from the pixel shifted image, and the number of pixels per frame is obtained in each of the imaging elements 111a, 111b, and 111c. The image is converted into a color image of 2M pixels, which is four times the number of pixels. That is, as shown in Fig. 3 (c), 4 times per frame using image signals arranged by shifting half-pixels ((Px / 2) and (Py / 2)) by R and B vertically and horizontally with respect to G. Interpolation generation (high-density interpolation processing) of the number of pixels is performed, and a color image signal is generated for each pixel as YCrCb = 4: 2: 2.

At this time, in the pixel shift image, the image signal per frame is represented by 12 Mbit, whereas in the color image, the color image signal per frame is 32 Mbit (16 bits / pix * 2Mpix = 32 Mbit).

Next, similarly to the first embodiment, the moving picture compression unit 150 compresses the frame sequence of the color image output from the image generating unit 141, and the compressed data of each frame is a series of compressed streams. It is output to the recording unit 160.

Next, details of an operation for generating a compressed stream will be described. First, in the imaging section 111, exposure is performed every predetermined time (e.g., 1/60 second), and the AFE 120 is sequentially performed with the exposure amount in each photoelectric conversion element as the analog electric signal for each exposure. )

At this time, since the imaging part 111 consists of three imaging elements 111a, 111b, and 111c, the set of analog electric signals output from each photoelectric conversion element becomes an analog image signal of three planes. In addition, each plane is a low-resolution video of 0.5M pixels. At the same time as one exposure ends, exposure of the next frame is started, and analog image signals of three pixel shifted planes are sequentially output.

At this time, three plane image signals corresponding to one exposure are collected to form one frame, and a moving image is formed by the frame sequence.

Next, the analog electric signal (which is the first data format in the present invention) of the pixel shift three-plane output from the imaging unit 111 is converted into a digital signal in the AFE 120. The digital image signal output from the AFE 120 is sequentially output in the exposure order of the imaging unit 111 (in this specification, this output order is referred to as display order).

Next, as shown in Fig. 4A, for each frame, k = 0, 1, 2, ... in the display order. , Frame number is given, frame k≡2 (mod 3) is I frame or P frame, frame k≡0 (mod 3) or k≡1 (mod 3) is B frame, It is input to the rearrangement unit 131. At this time, the image shift signal of the pixel shift three planes output in the display order is input to the frame rearrangement unit 131.

Next, as shown in Figs. 4A and 4B, in the frame rearranging unit 131, if the input frame k is an I frame or a P frame, it is output to the image generating unit 141 as it is. . In the frame rearrangement unit 131, if the input frame k is the B frame, the data is written to the frame buffers 131a and 131b alternately, and the original frame buffers 131a, The data recorded in 131b is output to the image generating unit 141.

As a result, three frames of delay processing are performed on the B frames, and the frame order input to the image generating unit 141 becomes the transmission order.

Next, in the image generating unit 141, each frame input through the frame rearranging unit 131 is converted into a high resolution color image signal from a three plane image signal. Here, image processing such as color conversion, edge enhancement of an image, noise suppression, tone curve processing, and the like, in addition to a process called generally known as high resolution processing (high density interpolation processing), is performed to obtain an image quality suitable for viewing.

Next, the color image signal converted by the image generating unit 141 is output to the moving image compression unit 150. In the moving picture compression unit 150, a compression process of the frame sequence of the color image input from the image generating unit 140 is performed, and the compressed data is recorded in the recording unit 160.

(Third embodiment)

Next, 3rd Embodiment of this invention is described using FIG. 3 is a block diagram showing the configuration of the imaging device 1C of the third embodiment to which the moving image processing device of the present invention is applied.

In addition, since the imaging device 1C in 3rd Embodiment is basically the same structure as the imaging devices 1A and 1B shown by 1st, 2nd Embodiment, the code | symbol same about the component which is common is the same. The detailed description will be omitted, and the features will be described below.

As shown in FIG. 5, the imaging device 1C is, for example, a video camcorder and is configured to respond to a request for an image display function to the finder 190 and an image output function to the monitor output unit 220.

In detail, in order to generate the color image output to the finder 190, the imaging device 1C comprises a signal dividing means 170 between the AFE 120 and the frame rearranging unit 131. It is comprised so that the output from 120 may be output by both the signal rearrangement part 131 and the 2nd image generation part 180 by the signal division means 170. FIG.

The second image generating unit 180 generates a color image and outputs it to the finder 190 without rearranging the frame in the frame rearranging unit 131. At this time, since the second image generating unit 180 is not required to generate a high resolution image like the image generating unit 141, the second image generating unit 180 has a simpler configuration than the image generating unit 141. FIG.

Since the image is displayed on the finder 190 in the display order, and the delay processing of the frame is not performed by the frame rearranging unit 131, the imaging device 1C captures images with good followability for the user of the imaging device. I can do it.

The finder 190 is composed of a video display device such as a small CRT or a liquid crystal screen, for example, and displays an image signal generated by the second image generator 180.

In addition, the image pickup device 1C bypasses the frame rearrangement unit 131 and inputs the image signal of the display order output from the signal dividing unit 170 to the image generation unit 141. Output switching means 210 and output switching means 210 for switching the input destination of the frame data of the color image output from the image generating unit 141 to either the moving image compression unit 150 or the monitor output unit 220. And a monitor output unit 220 for outputting the image signal of the color image input via the < RTI ID = 0.0 > In addition, the function of the rearrangement part bypass means in the present invention is expressed by the frame rearrangement part bypass means 200.

The frame rearrangement bypass means 200 is disposed in parallel with the frame rearrangement unit 131, and outputs an image signal in display order to the image generation unit 141 in conjunction with the output switching unit 210. At this time, the frame rearrangement bypass means 200 may be configured to control the frame rearrangement unit 131 so that the output from the frame rearrangement unit 131 is in the display order.

The output switching means 210 is provided between the image generating unit 141 and the moving image compression unit 150, and inputs the output of the image generating unit 141 to the moving image compression unit 150 during video recording. When outputting to the monitor, the output of the image generating unit 141 is input to the monitor output unit 220.

The monitor output unit 220 is provided as a video output terminal for outputting an image signal from the imaging device 1C to the outside, and connects the video signal when the high resolution display 225 or the like is connected through the video output terminal. It is configured to output to the system.

When the imaging device 1C outputs an image to the external high resolution display 225, the output switching means 210 is set to input the output of the image generating unit 141 to the monitor output unit 220. The output of the frame rearrangement unit 131 is stopped, and the frame rearrangement bypass means 190 operates to input an image in display order to the image generation unit 141, and to output an image signal in high resolution display order to an external high resolution display. It is output to 225 and displayed.

On the other hand, the imaging device 1C is set so that the output switching means 210 inputs the output of the image generating unit 141 to the moving image compression unit 150 when recording an image, and the frame rearrangement bypass means ( The 200 does not operate, and the images in the transmission order via the frame rearranging unit 131 are input to the image generating unit 141 and the moving image compression unit 150.

As a result, the imaging device 1C has functions of both monitor output and compressed recording of the color moving image through the image generating unit 141.

Next, 4th Embodiment of this invention is described using FIG. Fig. 6A is a block diagram showing the configuration of the imaging device 1D of the fourth embodiment to which the moving image processing device of the present invention is applied, and Fig. 6B is a configuration diagram of the imaging unit in the embodiment. to be.

In addition, since the imaging device 1D in 4th Embodiment is basically the same structure as the imaging device 1A shown in 1st Embodiment, the same code | symbol is attached | subjected about the structural component which is common, and is explained in full detail. The parts which become a characteristic are abbreviate | omitted and it demonstrates below.

As shown in FIG. 6A, the imaging device 1D includes an image pickup unit 112, a frame rearrangement unit 132, an AFE 120, and an image generation unit corresponding to an analog signal output from the image pickup unit 112. 140, the moving picture compression unit 150, the recording unit 160, and the like.

In the imaging device 1D, a frame rearrangement unit 132 is provided between the imaging unit 112 and the AFE 120 so as to alternately arrange a frame sequence with respect to the analog signal output from the imaging unit 112. have.

The imaging part 112 is comprised by the single plate type color imaging element similarly to 1st Embodiment. In addition, the frame rearrangement unit 132 is configured using a charge coupled device (CCD) of an analog memory as a frame buffer.

Then, the image pickup unit 112 is configured using the CCD image sensor similarly to the frame rearrangement unit 132, so that the image pickup unit 112 and the frame rearrangement unit 132 are integrated as shown in FIG. It can comprise with the CCD 135.

As shown in Fig. 6 (b), the CCD 135 includes a light receiving area 135s as an image capturing unit, two buffer areas 135a and 135b, and horizontal transfer units (I frame and P frame for I / P frames). Horizontal transmission unit), and B frame horizontal transmission unit.

The horizontal transmission unit for the I / P frame is provided between the light receiving area 135s and the buffer areas 135a and 135b, and the horizontal transmission unit for the B frame is located on the opposite side of the light receiving area 135s via the buffer area 135a. It is provided in the edge part of the buffer area 135b located.

The functions of the frame rearrangement unit 132 are expressed by the two buffer areas 135a and 135b, the horizontal transfer unit for the I / P frame, and the horizontal transfer unit for the B frame.

In the light receiving area 135s, each element (the so-called element corresponding to each pixel) of the CCD 135 is provided with a photoelectric conversion element and a mosaic color filter (which is a filter for passing monochromatic light in RGB). have.

Next, the operation of image pickup and frame rearrangement in the CCD 135 will be described. First, exposure is performed for a predetermined time in the light receiving area 135s, and the exposure amount in each photoelectric conversion element is accumulated as a charge (the first data format in the present invention becomes this charge amount).

If the exposed image frame is an I frame or a P frame, vertical transfer is performed to the light receiving area 135s, and the horizontal transfer unit for the I / P frame is operated to charge each pixel in the light receiving area 135s. Are sequentially output to the AFE 120 as an analog signal.

On the other hand, if the exposed image frame is a B frame, vertical transfer is performed to the entire CCD 135, and the horizontal transfer unit for the B frame is operated to charge the charge stored in the buffer area 135b as an analog signal. Output to. At this time, the charge accumulated in the light receiving area 135s is accumulated in the buffer area 135a, and the charge accumulated in the buffer area 135a is transferred to the buffer area 135b.

Accordingly, if the I frame or the P frame is arranged every three frames, the B frame is output to the AFE 120 after three frames of exposure as in the second embodiment (Fig. 3 (b)). In addition, since the I frame or the P frame is output to the AFE 120 immediately after the exposure, similarly to the frame rearrangement unit 131 in the second embodiment, the input is input to the AFE 120 and the image generating unit 140. The order of frames is the order of transmission.

As described above, according to the imaging device 1D of the fourth embodiment, the frame rearranging unit 132 can be provided in the CCD 135 integrated with the image sensor, so that the rearrangement of the frame as a separate digital process is unnecessary. Can be.

As mentioned above, although one Example of this invention was described, this invention is not limited to the said Example, A various aspect can be taken.

[Industry availability]

The frame sequence of the moving image output from the color image sensor in the first data format can be arranged in correspondence with the processing order of the compressed stream and then converted into a color image.

1A, 1B, 1C, 1D: imaging device 110, 111, 112: imaging section
111a, 111b, 111c: Imaging element 120: AFE (Analog Front End)
130,131,132: frame rearrangement
130a to 130d, 131a, 131b: frame buffer
135: CCD (Charge Coupled Devices) 135s: Light receiving area
135a, 135b: buffer area 140, 141: image generating unit
150: moving picture compression unit 150a, 150b: prediction memory
160: recording unit 170: signal dividing means
180: second image generating unit 190: finder
200: frame rearrangement bypass means 210: output switching means
220: monitor output unit 225: external high resolution display

Claims (15)

A moving picture processing apparatus for generating a compressed stream of a color moving picture coded by an inter-frame prediction method from a frame sequence of a moving picture divided into a plurality of frames in time series from a color image sensor and output in a first data format.
A frame rearrangement unit for arranging a frame sequence having the first data format in correspondence with the processing order of the compressed stream;
A color image generation unit for converting the first data format in the frame sequence exchanged and arranged in the frame rearrangement unit into a color image;
And a moving picture compression unit that encodes and compresses the frame sequence converted into the color image based on a difference between a plurality of frames before and after time. The method according to claim 1,
The color image sensor includes a plurality of photoelectric conversion elements arranged in a matrix, and a color filter corresponding to each of the photoelectric conversion elements, and the pixel information of the single color light in the plurality of color lights for each of the photoelectric conversion elements. It is a single-color color image sensor to output,
The first data format is a color mosaic image having color information of monochromatic light for each pixel,
And the color image generating unit is configured to perform demosaicing processing for generating pixel information of a plurality of colors of light for each of the pixels. The method according to claim 1,
The color image sensor,
It is comprised by the several color image sensor from which a spectral sensitivity distribution differs, and each light receiving surface is arrange | positioned shift | deviating to the pixel arrangement direction,
The first data format is image data output from each of the plurality of color image sensors,
And the color image generating unit is configured to synthesize image data of the plurality of color image sensors to increase the resolution. The method according to any one of claims 1 to 3,
The plurality of frames are composed of a NonF frame that encodes without referring to a temporally trailing frame, and an F frame that encodes with reference to a temporally trailing frame.
And the frame rearranging unit is configured to give a delay and output the delayed frame according to the type of the NonF frame or F frame to be input. The method according to claim 4,
The NonF frame further includes an I frame that encodes the image signal in the frame as it is without using the inter-frame prediction, and a P frame that encodes the difference from the image signal of the temporally preceding reference frame.
The F frame is a B frame that encodes a difference between a temporally preceding reference frame and a following reference frame,
And the frame rearrangement unit is configured to output the delayed output according to the type of the input I frame, P frame, or B frame. The method according to any one of claims 1 to 3,
And the frame rearrangement unit is configured to give a delay to a part of the frame types of the plurality of frame types and output the delayed part. The method according to any one of claims 1 to 6,
The frame rearrangement unit includes a frame buffer which stores image data for each of at least two frames of the image data of the first data format. The method according to any one of claims 1 to 7,
And the image generation unit is provided with an image deformation processing unit for performing image deformation of the color image. The method according to claim 8,
And the frame rearrangement unit is configured to output image data in a frame in a non-rasterized order when the image is deformed. The method according to any one of claims 1 to 9,
And a second image generating unit for generating a color image output to the finder, wherein the second image generating unit generates a color image without rearranging the frames in the frame rearrangement unit. Processing unit. The method according to any one of claims 1 to 10,
A rearrangement unit bypass means for outputting a frame sequence output from the color image sensor to the color image generation unit without passing through the frame rearrangement unit,
And a frame order output from the color image sensor and a frame order output from the frame rearrangement unit of the frame sequence input to the color image generation unit. A camcorder comprising the moving picture processing device and color image sensor according to any one of claims 1 to 11, and a recording device for storing moving picture information compressed by the moving picture compression unit. A camera for a remote monitor comprising a moving picture processing device according to any one of claims 1 to 11, a color image sensor, and a transmitting device for transmitting moving picture information compressed by the moving picture compression unit to an external device. A moving picture processing method for generating a compressed stream of a color moving picture coded by an inter-frame prediction method from a frame sequence of a moving picture divided into a plurality of frames in time series from a color image sensor and output in a first data format.
A frame rearranging step of rearranging a frame sequence formed of the first data format in correspondence with the processing order of the compressed stream;
A color image generation step of converting the first data format in the frame sequence exchanged and arranged in the frame rearrangement step into a color image;
And a moving picture compression step of encoding and compressing the frame sequence converted into the color image based on a difference between a plurality of frames before and after time. A moving picture processing program for generating a compressed stream of a color moving picture coded by an inter-frame prediction method from a frame sequence of a moving picture divided into a plurality of frames in time series from a color image sensor and output in a first data format.
A frame rearranging step of rearranging a frame sequence formed of the first data format in correspondence with the processing order of the compressed stream;
A color image generation step of converting the first data format in the frame sequence exchanged and arranged in the frame rearrangement step into a color image;
A moving picture processing program for causing a computer to execute a moving picture compression step of encoding and compressing the frame sequence converted into the color image based on a difference between a plurality of frames before and after time.

Images