WO2009125673A1 - Dynamic image processing device, dynamic image processing method, and dynamic image processing program - Google Patents

Abstract

Provided are a dynamic image processing device and a dynamic image processing method which can realize a lower power consumption at a low cost by reducing a storage capacity and band for frame rearrangement when performing a demosaic process and a compressed stream generation from image data outputted from a color image sensor. The dynamic image processing device includes: a frame rearrangement unit (130) which rearranges a frame sequence formed in a first data format and outputted from an imaging unit (110) in accordance with a compressed stream processing order; a color image generation unit (140) which converts the first data format used in the frame sequence rearranged by the frame rearrangement unit (130) into a color image; and a dynamic image compression unit (150) which encodes and compresses the frame sequence converted into the color image, in accordance with a difference between temporally preceding and following frames.

Description

Movie processing apparatus, movie processing method, and movie processing program

The present invention relates to a moving image processing apparatus, a moving image processing method, and a moving image processing program for generating a compressed stream by encoding a frame sequence of a moving image output by being divided into a plurality of frames in a time series from a color image sensor.

2. Description of the Related Art Conventionally, in a video camera for shooting a moving image, a subject image is formed on an image sensor via a lens, and the subject image is photoelectrically converted by the image sensor to generate a plurality of frame data in time series. There is known a moving image processing technique (MPEG) that predicts a motion between frames (a so-called inter-frame prediction method) and generates a compressed stream.

In general, video processing technology estimates (predicts) a motion vector that represents how much the pixel of the current frame has moved compared to the pixel of the previous frame, and instead of transmitting the entire image, the difference between these motion vectors The transmission information is compressed by transmitting.

Specifically, in the moving image processing technology represented by MPEG, the type of frame is determined from an I frame that encodes an image signal in the frame as it is without using inter-frame prediction, and an image signal of a reference frame that precedes in time. And a B frame for encoding a difference between a temporally preceding reference frame and a subsequent reference frame, and an arrangement of these frames and a repetition cycle are set. .

For example, as shown in FIG. 2 (b), M = 3 MPEG has GOP (Group of Pictures) based on I, I, B, B, P, B, B, P, B, B, P, B, B, I... Are composed of frames that are continuous in time series.

On the other hand, since the order of frames input in this way is different from the order of transmission after being encoded, it is necessary to rearrange the order of frames at the time of encoding. For rearrangement, a frame buffer memory for temporarily storing B frames is required. For example, in the case of M = 3 in which two B frames are inserted between I and P, two frames of memory are required to store this (for example, see Patent Document 1). .

In addition, a plurality of photoelectric conversion elements are configured in a matrix form as a single-plate image pickup element, and R (red), G (green), and B (blue) color filters are associated with the photoelectric conversion elements on the front surface thereof. There is a technique for generating a color image by applying signal processing to a single color image signal output through the color filter.

In an image output via a single-plate image sensor, each pixel is a color mosaic image having only color information of a single color, and in order to generate a color image, each pixel has red (R), green ( It is necessary to provide a plurality of color information such as G) and blue (B).

Therefore, in image processing using a single-plate image sensor, demosaic processing (also referred to as color interpolation processing) is performed based on a color mosaic image in which each pixel has color information of only one of R, G, and B components. A color image is generated from the color mosaic image. Here, the demosaic process is performed by interpolating other color information deficient in each pixel of the color mosaic image using the color information of other pixels around the pixel, so that each pixel is R, G, This is a process for generating a color image having all color information of the B component (so-called color interpolation process).
JP-A-10-056552

However, according to the conventional moving image processing technique, generally, when rearranging the frames, the image data stored in the frame buffer is a color image having color information of a plurality of colors for each pixel, and demosaic processing is performed. In connection with the required image data, no moving image processing technique effective for saving memory capacity has been disclosed.

Therefore, the present invention stores memory for frame rearrangement when performing demosaic processing (so-called color image generation processing in the present invention) and compressed stream generation from image data output from a color image sensor. It is an object of the present invention to provide a moving image processing apparatus, a moving image processing method, and a moving image processing program capable of reducing power consumption and cost by reducing capacity and bandwidth.

In order to achieve the above object, the invention according to claim 1 is directed to a frame sequence of a moving image which is divided into a plurality of frames in a time series from a color image sensor and output in a first data format. In a moving image processing apparatus that generates a compressed stream of a color image encoded by a prediction method, a frame rearrangement unit that rearranges the frame sequence composed of the first data format in association with the processing order of the compressed stream; A color image generation unit for converting the first data format in the frame sequence rearranged by the frame rearrangement unit into a color image, and a plurality of the frame sequences converted into the color image before and after in time A video compression unit that encodes and compresses based on a difference between frames And features.

According to the moving image processing apparatus of claim 1, the frame rearrangement unit that rearranges the frame sequence having the first data format in association with the processing order of the compressed stream, and the frame rearranged by the frame rearrangement unit A color image generation unit that converts a first data format in a sequence into a color image, and the frame sequence converted into a color image is encoded and compressed based on differences between a plurality of frames before and after the time. And a video compression unit, the memory capacity and bandwidth (frame buffer memory capacity and bandwidth) required for frame rearrangement can be reduced, resulting in lower power consumption, lower cost, and lower power consumption. A reduction in the amount of heat generated in the moving image processing circuit can be realized.

In other words, since the frame data used for the frame rearrangement is the first data format output from the image sensor before the color image is generated, the frame rearrangement is performed rather than using the color image data for the frame rearrangement. Memory capacity and bandwidth required for the array can be reduced.

According to a first aspect of the present invention, in the moving image processing apparatus according to the second aspect, the color image sensor includes a plurality of photoelectric conversion elements arranged in a matrix and each of the photoelectric conversion elements. A single-plate color image sensor that outputs pixel information of single-color light of the plurality of color lights for each photoelectric conversion element, wherein the first data format is a pixel A color mosaic image having color information of a single color light every time, and the color image generation unit is configured to perform a demosaic process for generating pixel information of a plurality of color lights for each pixel, whereby a color image When performing demosaic processing and generation of a compressed stream from image data output from a sensor, it is possible to reduce the storage capacity and bandwidth necessary for frame rearrangement.

Further, in the moving image processing apparatus according to claim 1, as in the invention according to claim 3, the color image sensor is configured by a plurality of color image sensors having different spectral sensitivity distributions, and each light receiving surface has a light receiving surface. The first data format is image data output from each of the plurality of color image sensors, and the color image generation unit includes the plurality of color images. Necessary for frame rearrangement when generating color image and generating compressed stream from image data output from color image sensor because it is configured to increase resolution by synthesizing sensor image data Saving storage capacity and bandwidth.

Further, in the moving image processing device according to any one of claims 1 to 3, as in the invention according to claim 4, the plurality of frames are encoded without referring to a temporally subsequent frame. The frame rearrangement unit gives a delay according to the type of the input NonF frame and F frame. Applicable when configured to output.

According to a fourth aspect of the present invention, in the moving image processing apparatus according to the fifth aspect, the NonF frame further encodes an image signal in the frame as it is without using the inter-frame prediction. A frame and a P frame that encodes a difference from an image signal of a temporally preceding reference frame, and the F frame encodes a difference between a temporally preceding reference frame and a subsequent reference frame This is applicable when the frame rearrangement unit is configured to output with a delay according to the type of the input I frame, P frame, or B frame.

Further, in the moving image processing device according to any one of claims 1 to 3, as in the invention according to claim 6, the frame rearrangement unit includes a part of the plurality of frame types. It may be configured to output with a delay.

The moving image processing apparatus according to any one of claims 1 to 6, as in the invention according to claim 7, includes at least the image data of the first data format in the frame rearrangement unit. It is preferable that a frame buffer for storing image data is provided for every two frames. Thereby, in the MPEG of M = 3 which is a general moving image processing technique, two B frames arranged between the I frame and the P frame can be stored.

Further, in the moving image processing device according to any one of claims 1 to 7, as in the invention according to claim 8, the image generation unit includes an image deformation processing unit that performs image deformation of the color image. It is preferable that it is provided. As a result, the frame buffer of the frame rearrangement unit can be used as a frame buffer required when performing color image deformation processing, and the storage capacity and bandwidth required for image deformation can be reduced separately.

According to an eighth aspect of the present invention, in the moving image processing device according to the ninth aspect, the frame rearrangement unit outputs the image data in the frame in a non-raster sequential manner when the image is deformed. It is preferable that it is comprised. As a result, it is possible to perform image deformation such as digital zoom, camera shake, and aberration correction by using the frame buffer of the frame rearrangement unit without separately preparing a frame buffer for image deformation.

The moving image processing apparatus according to any one of claims 1 to 9 includes a second image generation unit that generates a color image to be output to a viewfinder, as in the invention according to claim 10, In the second image generation unit, when the color image is generated in the finder by generating a color image without performing the frame rearrangement in the frame rearrangement unit, the images are displayed on the finder in the order of display. There is no delay due to frame rearrangement, and the follow-up performance to the output from the color image sensor can be maintained well.

The moving image processing apparatus according to any one of claims 1 to 10 does not pass the frame sequence output from the color image sensor through the frame rearrangement unit as in the invention according to claim 11. A rearrangement unit bypassing means for outputting to the color image generation unit, and a frame sequence input to the color image generation unit and a frame order output from the color image sensor and output from the frame rearrangement unit By being configured to be able to switch the frame order, the presence / absence of frame rearrangement can be selected as necessary, and convenience can be improved.

Further, as in the invention described in claim 12, the moving image processing device and the color image sensor according to any one of claims 1 to 11, and a recording device that stores the moving image information compressed by the moving image compression unit, A camcorder may be configured using This enables the camcorder to reduce the memory capacity and bandwidth (frame buffer memory capacity and bandwidth) required for frame rearrangement when generating a compressed stream of a frame sequence of a moving image, reducing power consumption and cost. Thus, it is possible to reduce the amount of heat generated in the moving image processing circuit.

Further, as in the invention described in claim 13, the moving image processing apparatus and color image sensor according to any one of claims 1 to 11 and the moving image information compressed by the moving image compression unit are transmitted to an external device. A remote monitor camera may be configured using the transmitter. As a result, the remote monitor camera can reduce the memory capacity and bandwidth (frame buffer memory capacity and bandwidth) necessary for frame rearrangement when generating a compressed stream of a frame sequence of a moving image, reducing power consumption, Cost reduction and reduction in the amount of heat generated in the moving image processing circuit can be realized.

Next, the invention according to claim 14 is encoded by an inter-frame prediction method from a frame sequence of a moving image which is divided into a plurality of frames in time series from a color image sensor and output in a first data format. In the moving image processing method for generating a compressed stream of color moving images, a frame rearrangement step for rearranging the frame sequence composed of the first data format in association with the processing order of the compressed stream, and the frame rearrangement step. A color image generation step of converting the first data format in the rearranged frame sequence into a color image, and the frame sequence converted into the color image into temporal differences between a plurality of frames. And a video compression step for encoding and compressing based on To.

According to the moving image processing method of claim 14, a frame rearrangement step for rearranging the frame sequence having the first data format in association with the processing order of the compressed stream, and the frames rearranged in the frame rearrangement step A color image generation step for converting the first data format in the sequence into a color image, and a moving image in which the frame sequence converted into the color image is encoded and compressed based on differences between a plurality of frames before and after the time By using the compression step, the memory capacity and bandwidth required for frame rearrangement (memory capacity and bandwidth of the frame buffer) can be reduced, and the power consumption can be reduced and the cost can be reduced. The amount of heat generated in the moving image processing circuit can be reduced along with the reduction in power consumption.

Next, the invention according to claim 15 is encoded by an inter-frame prediction method from a frame sequence of a moving image which is divided into a plurality of frames in time series from a color image sensor and output in a first data format. A moving image processing program for generating a compressed stream of color moving images, wherein a frame rearrangement step of rearranging a frame sequence composed of the first data format in association with a processing order of the compressed stream; and the frame rearrangement A color image generation step of converting the first data format in the frame sequence rearranged in the step into a color image, and the frame sequence converted into the color image between a plurality of frames before and after in time A video compression step for encoding and compressing based on the difference; To be executed in.

According to the moving image processing program of claim 15, a frame rearrangement step for rearranging the frame sequence having the first data format in association with the processing order of the compressed stream, and the frame rearranged by the frame rearrangement step A color image generation step for converting the first data format in the sequence into a color image, and a moving image in which the frame sequence converted into the color image is encoded and compressed based on differences between a plurality of frames before and after the time By causing the computer to execute the compression step, the memory capacity and bandwidth (frame buffer memory capacity and bandwidth) required for frame rearrangement can be reduced and the power consumption can be reduced, as in the first aspect of the invention. , Reduce the amount of heat generated in the video processing circuit due to low cost and low power consumption It can be current.

The moving image processing apparatus, the moving image processing method, and the moving image processing program of the present invention rearrange the frame sequence composed of the first data format in association with the processing order of the compressed stream, and then the first in the rearranged frame sequence The data format is converted to a color image, and the frame sequence converted to a color image is encoded and compressed based on the difference between multiple frames before and after the frame. The memory capacity and bandwidth (frame buffer memory capacity and bandwidth) required for the video processing circuit can be reduced, and the heat generation amount in the moving image processing circuit can be reduced with lower power consumption, lower cost, and lower power consumption.

Further, by configuring the camcorder using the moving image processing device and color image sensor according to the present invention and the recording device that stores the moving image information compressed by the moving image compression unit of the moving image processing device, in the camcorder, When generating a compressed stream of a frame sequence, the memory capacity and bandwidth (frame buffer memory capacity and bandwidth) required for frame rearrangement can be reduced, reducing power consumption and cost, and reducing the amount of heat generated in the video processing circuit. Reduction can be realized.

Further, by configuring a remote monitor camera using the moving image processing device and color image sensor according to the present invention, and a transmission device that transmits the moving image information compressed by the moving image compression unit of the moving image processing device to an external device, When generating a compressed stream of a video frame sequence in a remote monitor camera, the memory capacity and bandwidth required for frame rearrangement (memory capacity and bandwidth of the frame buffer) can be reduced, reducing power consumption and cost. Thus, it is possible to reduce the amount of heat generated in the moving image processing circuit.

In the first embodiment of the present invention, (a) is a block diagram showing a configuration of an imaging apparatus 1A to which the moving image processing apparatus of the present invention is applied, and (b) is an explanatory diagram of an imaging unit in the imaging apparatus 1A. is there. It is explanatory drawing of operation | movement of the frame rearrangement part in 1 A of imaging devices of the 1st embodiment. In the 2nd Embodiment of this invention, (a) is a block diagram showing the structure of the imaging device 1B to which the moving image processing apparatus of this invention was applied, (b) is explanatory drawing of the imaging part in the imaging device 1B, (C) is explanatory drawing of the image data synthesis process in the imaging device 1B. It is explanatory drawing of operation | movement of the frame rearrangement part in the imaging device 1B of the 2nd embodiment. It is a block diagram showing the structure of 1 C of imaging devices to which the moving image processing apparatus of this invention was applied in the 3rd Embodiment of this invention. It is a block diagram showing the structure of imaging device 1D to which the moving image processing device of this invention was applied in the 4th Embodiment of this invention.

Explanation of symbols

1A, 1B, 1C, 1D ... imaging device, 110, 111, 112 ... imaging unit, 111a, 111b, 111c ... imaging device, 120 ... AFE (Analog Front End), 130, 131, 132 ... frame rearrangement unit, 130a 130d, 131a, 131b ... Frame buffer, 135 ... CCD (Charge Coupled Devices), 135s ... Light receiving area, 135a, 135b ... Buffer area, 140, 141 ... Image generation unit, 150 ... Video compression unit, 150a, 150b ... Prediction Memory 160, recording unit 170, signal dividing unit 180, second image generating unit 190, viewfinder, 200, frame rearrangement bypass unit, 210, output switching unit, 220, monitor output unit, 225, external high Resolution display.

(First embodiment)
Next, a first embodiment of the present invention will be described with reference to FIGS.

1A is a block diagram showing the configuration of the imaging apparatus 1A of the first embodiment to which the moving image processing apparatus of the present invention is applied, and FIG. 1B is an explanatory diagram of an imaging unit in the imaging apparatus 1A. is there. FIG. 2 is an explanatory diagram of the operation of the frame rearrangement unit in the imaging apparatus 1A according to the first embodiment.

As illustrated in FIG. 1A, the imaging device 1A is, for example, a video camcorder, and sequentially converts captured image signals into analog electrical signals and outputs the analog image signals output from the imaging unit 110. The AFE 120 that converts electrical signals into digital data and outputs them, the digital data output from the AFE 120 is divided into frames, and the frame order is changed and output (so-called frame rearrangement) 130 The image generation unit 140 that converts the image data of each frame output from the frame rearrangement unit 130 into a color image, the color image frame sequence output from the image generation unit 140 is compressed as a moving image, and a compressed stream is output The compressed video output from the moving image compression unit 150 and the moving image compression unit 150 Recording unit 160 for recording the Shumemori and optical / magnetic recording medium is constituted by a like.

Further, the imaging apparatus 1A includes a CPU (Central Processing Unit) and a ROM (Read Only Memory) (not shown), and the CPU controls each process of the imaging apparatus 1A according to a control program stored in the ROM. To do.

The imaging unit 110 is a single-plate color image sensor (single-plate color image sensor according to the present invention), in which a plurality of photoelectric conversion elements are arranged in a matrix, and a front surface thereof is associated with the photoelectric conversion element. As shown in FIG. 1B, a color filter having a Bayer array of three primary colors of R (red), G (green), and B (blue) is provided, and passes through the filter portion of each color. It is configured to convert a single color light amount into an electrical signal. Note that, as shown in FIG. 1B, the Bayer arrangement is such that the G color filters are arranged in a checkered pattern, the G color filters and the R color filters are alternately arranged, the G color filters, and the B colors. The columns in which the filters are alternately arranged are alternately arranged.

The AFE 120 is input via a correlated double sampling circuit (CDS: Correlated Double Sampling) and a correlated double sampling circuit that performs correlated double sampling on the analog image signal output from the imaging unit 110 and removes noise. The imaging unit 110 includes a variable gain amplifier (AGC) that amplifies the analog image signal, an A / D converter that converts the analog image signal input through the variable gain amplifier into a digital image signal, and the like. The output analog image signal of the frame is converted into a digital image signal associated with the Bayer array and output to the frame rearrangement unit 130.

Here, if the imaging unit 110 is configured with 2M pixels (pixels) and the A / D conversion accuracy in the AFE 120 is 8 bits, an image signal per frame output from the AFE 120 is 16 Mbits (2M * 8 bits = 16 Mbits). )

The frame rearrangement unit 130 converts the frame order of the digitized Bayer data input in the display order into a transmission order associated with the processing order of the moving picture compression unit 150.

For example, when the M = 3 MPEG stream is generated in the moving image compression unit 150, as shown in FIG. 2A, k≡2 ( If mod 3), it is an I frame or P frame, and if k≡0 or 1 (mod 3), it is a B frame. mod is an operator that divides a predetermined numerical value (k) by a mod number to obtain the remainder.

Therefore, the frame rearrangement unit 130 gives a delay of 3 frames more than the I frame or P frame to the B frame and outputs it to the image generation unit 140. In this embodiment, frame buffers 130a to 130d for four frames are provided, and frame rearrangement is realized by assigning frame buffers 130a to 130d as appropriate to inputs and outputs. Each of the frame buffers 130a to 130d has a capacity of 16 Mbits and has a total memory of 64 Mbits.

The image generation unit 140 sequentially converts the frames output from the frame rearrangement unit 130 from Bayer images to color images. A color image signal is generated with YCrCb = 4: 2: 2. At this time, in a Bayer image, one value per pixel = 8 bits / pix, whereas in a color image, two values per pixel = 16 bits / pix, and the color image signal per frame is increased. , 32 Mbit (16 bits / pix * 2 Mpix = 32 Mbit). In the color image signal described above, Y is the luminance, Cr is the R color difference with respect to Y, and Cb is the B color difference with respect to Y.

In this embodiment, the frame buffers 130a to 130d can be referred to the frame rearrangement unit 130 for any frame. Therefore, the image generation unit 140 can perform non-raster sequential access to the Bayer image.

Accordingly, the imaging apparatus 1A according to the present embodiment also performs image deformation such as digital zoom, camera shake, chromatic aberration, and distortion correction using the frame buffers 130a to 130d without separately preparing a buffer for image deformation. be able to.

The moving image compression unit 150 compresses the color image frame sequence output from the image generation unit 140. In the present embodiment, MPEG compression is performed with a period M in which an I frame or a P frame appears at an interval of 3 frames (M = 3). The MPEG moving image compression unit 150 includes two prediction memories 150a and 150b for performing inter-frame prediction. The compressed data of each frame is output to the recording unit 160 as a series of compressed streams.

Next, details of the operation for generating a compressed stream will be described. First, in the imaging unit 110, exposure is performed every predetermined time (for example, 1/60 seconds), and for each exposure, the exposure amount in each photoelectric conversion element is converted into an analog electric signal and sequentially output to the AFE 120. At this time, a set of analog electric signals output from each photoelectric conversion element becomes a Bayer image signal associated with the color filter of the Bayer array. In addition, since it is moving image shooting, the exposure of the next frame is started simultaneously with the end of one exposure, and analog image signals are sequentially output. Each image is used as a frame, and a moving image is formed by the frame sequence.

Next, the analog electrical signal output from the imaging unit 110 is converted into a digital signal by the AFE 120. This digital signal is a Bayer image signal corresponding to the Bayer array of the imaging unit, and has the first data format. In addition, the Bayer image signals output from the AFE 120 are sequentially output in the order of exposure of the imaging unit 110 (this output order is referred to as display order in this document).

Next, as shown in FIG. 2 (a), for each frame, frame numbers are given as k = 0, 1, 2,... In the order of display, and a frame of k≡2 (mod 3) is an I frame or Each frame is input to the frame rearrangement unit 130 as a P frame and a frame with k≡0 (mod 3) or k≡1 (mod 3) as a B frame. At this time, the Bayer image signals output in the display order are input to the frame rearrangement unit 130 in raster order.

Next, in the frame rearrangement unit 130, the input frame (k) input from the AFE 120 is stored and stored in one of the frame buffers (input buffers in FIG. 2A) 130a to 130d. Are output to the image generation unit 140 in an order different from the order (this is a frame rearrangement step in the present invention).

Specifically, as shown in FIGS. 2A and 2B, a delay of 1 frame is performed for the I frame or the P frame, and a delay of 4 frames is performed for the B frame. In this case, according to the present invention, M frames (B is the period in which I frames or P frames appear in the MPEG moving image processing technology) for B frames rather than I frames or P frames, regardless of the absolute amount of delay. It is important to give as much delay as possible. As a result, the frame order input to the image generation unit 140 becomes the transmission order.

Next, the image generation unit 140 converts each frame input via the frame rearrangement unit 130 from a Bayer image signal to a color image signal (this is a color image generation step in the present invention). Here, in addition to generally known color interpolation processing or demosaicing processing, image processing such as color conversion, image edge enhancement, noise suppression, and tone curve processing is performed to obtain an image quality suitable for viewing.

Further, the image generation unit 140 can read the Bayer image signals stored in the frame buffers 130a to 130d of the frame rearrangement unit 130 in a non-raster order and perform image deformation such as enlargement, reduction, and rotation. The color image signal converted by the image generation unit 140 is output to the moving image compression unit 150.

Next, the moving image compression unit 150 performs compression processing of the frame sequence of the color image input from the image generation unit 140 (this is a moving image compression step in the present invention).

In this embodiment, MPEG compression processing of M = 3 is performed, and different compression methods are used for the three types of I frame, P frame, and B frame. Note that the NonF frame in the present invention corresponds to an I frame and a P frame, and the F frame in the present invention corresponds to a B frame.

Specifically, for the I frame, intra-frame coding without using the prediction memory is performed and output to the recording unit 160, and the update history of the prediction memory 150a or 150b is older (that is, updated earlier). Update).

For the P frame, encoding is performed using inter-frame prediction from a frame in which the update history of the prediction memory 150a or 150b is stored in a newer one, and the result is output to the recording unit 160 and the prediction memory 150a. Alternatively, the update history of 150b is input and updated. When the P frame is input to the image generation unit 140, the I frame or the P frame input immediately before the third frame is stored in the newer one of the prediction memories 150a and 150b.

The B frame is encoded using inter-frame prediction from two frames stored in the prediction memories 150a and 150b, and is output to the recording unit 160. When the B frame is input to the moving picture compression unit 150, the newer one of the prediction memories 150a and 150b stores the I frame or the P frame input immediately before to the image generation unit 140, and the other (That is, which one of the prediction memories 150a and 150b is the older one) and the I frame or P frame input immediately before the three frames are stored.

However, the order of frames input to the image generation unit 140 is the transmission order rearranged by the frame rearrangement unit 130, and in terms of display order, the prediction memory 150a or 150b, whichever is older, The I frame or P frame exposed immediately before (the B frame to be processed) in the unit 110 is stored, and the I frame exposed immediately after (the B frame to be processed) in the imaging unit 110 is stored. Alternatively, P frames are stored (so-called reference frames that precede in time and reference frames that follow in time in the present invention are stored), and bi-directional prediction is realized in B frames. Is done.

Next, the recording unit 160 records a compressed stream of the image sequence encoded by the moving image compression unit 150.

Note that the imaging apparatus 1A may include a transmission unit instead of the recording unit 160. At this time, the transmission unit is connected to another video device via a wired or wireless communication means. Further, the transmission unit transmits the image sequence encoded by the moving image compression unit 150 to another video device as a compressed stream. Thus, a remote monitor camera can be configured.

As described above, the imaging apparatus 1A described in the first embodiment is necessary for frame rearrangement by performing frame rearrangement processing generally required for MPEG compression on a Bayer image instead of a color image. Memory capacity and bandwidth can be reduced. Further, in the image generation unit 140, image deformation can be performed without requiring a separate image deformation memory by performing image deformation using the frame buffers 130a to 130d used for frame rearrangement. it can.

In other words, when the frame rearrangement unit and the moving image compression unit are configured after the image generation unit as in the past, the frame rearrangement unit has a larger memory than the first embodiment in order to rearrange the color images. A capacity frame buffer is required, and the required bandwidth also increases. In addition, according to the conventional example, generally, a frame buffer for image deformation is separately required in the image generation unit. Note that the imaging unit 110 of the present embodiment may have a PIA (Pixel Interleaved Array) array of 1 million pixels (1M pixels), and the image generation unit 140 may perform high pixel processing in addition to demosaic processing. In this case, the memory capacity and bandwidth required for frame rearrangement can be further halved by applying the present invention. Here, the PIA array is a pixel array obtained by rotating a square lattice by 45 degrees. Compared with the Bayer array, the PIA array can expand the light collection area, and the resolution close to 2M pixels of the Bayer array can be obtained with 1M pixels. be able to.

(Second Embodiment)
Next, a second embodiment of the present invention will be described with reference to FIGS. 3, (a) is a block diagram showing the configuration of the imaging apparatus 1B of the second embodiment to which the moving image processing apparatus of the present invention is applied, and (b) is an explanatory diagram of an imaging unit in the imaging apparatus 1B. (C) is explanatory drawing of the image data synthesis process in the imaging device 1B. FIG. 4 is an explanatory diagram of the operation of the frame rearrangement unit in the imaging apparatus 1B of the embodiment.

Note that the imaging apparatus 1B in the second embodiment is basically the same in configuration as the imaging apparatus 1A shown in the first embodiment, and therefore, the same components are assigned the same reference numerals for detailed description. Omitted and characteristic parts will be described below.

As illustrated in FIG. 3A, the imaging device 1B is, for example, a video camcorder, and sequentially converts captured image signals into analog electrical signals and outputs them, and the analog output from the imaging unit 111. The AFE 120 that converts electrical signals into digital data and outputs them, the digital data output from the AFE 120 is divided into frames, and the frame order is changed and output (so-called frame rearrangement) 131. The image generation unit 141 that converts each frame data output from the frame rearrangement unit 131 into color data associated with the color image, the color image frame sequence output from the image generation unit 141 is compressed as a moving image, Video compression unit 150 that outputs a compressed stream, and a compression stream output from video compression unit 150 The beam, for example, the recording unit 160 for recording in the flash memory or optical or magnetic recording medium are constituted by the like.

The imaging unit 111 includes three imaging elements 111a, 111b, and 111c having different spectral sensitivity distributions, and the respective light receiving surfaces are arranged so as to be shifted in the pixel arrangement direction.

Specifically, in the imaging unit 111, color separation prisms for separating R, G, and B color lights are arranged on the optical path of the imaging optical system, and the imaging elements 111a, 111b, and 111c are provided on the imaging surfaces of the respective color lights. This is a three-plate color camera arranged.

Then, as shown in FIG. 3B, by shifting the positions of the image sensors 111a, 111b, and 111c with subpixel accuracy, a resolution higher than the number of pixels for each of the image sensors 111a, 111b, and 111c is obtained. Configured to get.

That is, the imaging unit 111 is configured by a color image sensor including a plurality of imaging elements 111a, 111b, and 111c having different spectral sensitivity distributions, and the respective light receiving surfaces are arranged so as to be shifted in the pixel arrangement direction.

In the second embodiment, a resolution of 2M pixels of full HD (full high definition) is obtained, and each imaging element 111a, 111b, 111c is configured with 0.5M pixels (pixels). And B are arranged so as to be shifted by half pixels ((Py / 2) and (Px / 2)) vertically and horizontally.

As in the first embodiment, the AFE 120 converts the analog image signal output from the imaging unit 111 into a digital image signal including three planes R, G, and B, and outputs the digital image signal to the frame rearrangement unit 131. .

Here, if the accuracy of A / D conversion in the AFE 120 is 8 bits, the image signal per frame output from the AFE 120 is 12 Mbits (0.5 M * 8 bits * 3 planes = 12 Mbits).

The frame rearrangement unit 131 converts the frame order of the digitized pixel shift data input in the display order into the transmission order associated with the processing order of the moving picture compression unit 150, as in the first embodiment. In the second embodiment, the frame rearrangement unit 131 includes frame buffers 131a and 131b for two frames, and the frame buffers 131a and 131b are assigned or input as appropriate according to the input from the imaging unit 111. The frame rearrangement is realized by outputting as is. The frame buffers 131a and 131b each have a capacity of 12 Mbit and have a total of 24 Mbit of memory.

The image generation unit 141 sequentially performs high resolution processing on the frame output from the frame rearrangement unit 131 from the pixel-shifted image, and the number of pixels per frame is equal to the number of pixels in each of the image sensors 111a, 111b, and 111c. The image is converted to a color image of 2M pixels that is quadrupled. In other words, as shown in FIG. 3C, by using image signals in which R and B are shifted by half pixels ((Px / 2) and (Py / 2)) vertically and horizontally with respect to G, Four times the number of pixels per frame is generated by interpolation (high-density interpolation processing), and a color image signal is generated as YCrCb = 4: 2: 2 for each pixel.

At this time, in the pixel shifted image, the image signal per frame is represented by 12 Mbit, whereas in the color image, the color image signal per frame is 32 Mbit (16 bits / pix * 2 Mpix = 32 Mbit). .

Next, as in the first embodiment, the moving image compression unit 150 compresses the frame sequence of the color image output from the image generation unit 141, and the compressed data of each frame is recorded as a series of compressed streams in the recording unit 160. Is output.

Next, details of the operation for generating a compressed stream will be described. First, the imaging unit 111 performs exposure every predetermined time (for example, 1/60 second), and for each exposure, the exposure amount in each photoelectric conversion element is converted into an analog electric signal and sequentially output to the AFE 120.

At this time, since the imaging unit 111 includes the three imaging elements 111a, 111b, and 111c, a set of analog electric signals output from the photoelectric conversion elements becomes a three-plane analog image signal. Each plane is a low-resolution moving image shooting of 0.5 M pixels. As soon as one exposure is completed, the exposure of the next frame is started, and an analog image signal of 3 planes is output sequentially with pixel shifting.

Also, at this time, image signals of 3 planes corresponding to one exposure are gathered to form one frame, and a moving image is formed by the frame sequence.

Next, the pixel-shifted three-plane analog electrical signal output from the imaging unit 111 (which is the first data format in the present invention) is converted into a digital signal by the AFE 120. In addition, digital image signals output from the AFE 120 are output one after another in the exposure order of the imaging unit 111 (this output order is referred to as display order in this document).

Next, as shown in FIG. 4A, for each frame, frame numbers k = 0, 1, 2,... Are given in the order of display, and a frame of k≡2 (mod 3) is an I frame or The frame is input to the frame rearrangement unit 131 as a P frame and a frame with k≡0 (mod 3) or k≡1 (mod 3) as a B frame. At this time, the pixel-shifted three-plane image signal output in the display order is input to the frame rearrangement unit 131.

Next, as shown in FIGS. 4A and 4B, in the frame rearrangement unit 131, if the input frame (k) input is an I frame or a P frame, it is output to the image generation unit 141 as it is. To do. Further, in the frame rearrangement unit 131, if the input frame (k) input is a B frame, the data is once recorded alternately in the frame buffers 131a and 131b, and at the same time, originally in the frame buffers 131a and 131b. Is output to the image generation unit 141.

Thereby, a delay process for three frames is performed on the B frame, and the frame order input to the image generation unit 141 becomes the transmission order.

Next, the image generation unit 141 converts each frame input via the frame rearrangement unit 131 from a 3-plane image signal to a high-resolution color image signal. Here, in addition to the processing known as high resolution processing (high-density interpolation processing), image processing such as color conversion, image edge enhancement, noise suppression, tone curve processing, etc. Image quality suitable for

Next, the color image signal converted by the image generation unit 141 is output to the moving image compression unit 150. The moving image compression unit 150 compresses the frame sequence of the color image input from the image generation unit 140, and the recording unit 160 records the compressed data.

(Third embodiment)
Next, a third embodiment of the present invention will be described with reference to FIG. FIG. 3 is a block diagram showing a configuration of an imaging apparatus 1C according to the third embodiment to which the moving image processing apparatus of the present invention is applied.

Note that the imaging apparatus 1C in the third embodiment is basically the same in configuration as the imaging apparatuses 1A and 1B shown in the first and second embodiments, and therefore, the same reference numerals are given to common components. The detailed description will be omitted, and the characteristic parts will be described below.

As shown in FIG. 5, the imaging apparatus 1 </ b> C is, for example, a video camcorder, and is configured to be able to meet the demand for an image display function to the finder 190 and an image output function to the monitor output unit 220.

Specifically, in order to generate a color image to be output to the finder 190, the imaging apparatus 1C includes a signal dividing unit 170 between the AFE 120 and the frame rearrangement unit 131, and outputs the signal from the AFE 120 to the signal dividing unit 170. Thus, it is configured to output to both the frame rearrangement unit 131 and the second image generation unit 180.

The second image generation unit 180 generates a color image and outputs it to the finder 190 without performing frame rearrangement in the frame rearrangement unit 131. At this time, since the second image generation unit 180 is not required to generate a high-resolution image unlike the image generation unit 141, the second image generation unit 180 has a simpler configuration than the image generation unit 141.

Since the image is displayed in the display order on the finder 190 and the frame rearrangement unit 131 does not perform frame delay processing, the imaging device 1C can perform shooting with good followability for the user of the imaging device. .

The finder 190 is composed of a video display device such as a small cathode ray tube or a liquid crystal screen, and displays the image signal generated by the second image generation unit 180.

Furthermore, the imaging apparatus 1C receives the image signal in the display order output from the signal dividing unit 170 from the frame rearrangement bypass unit 200 and the image generation unit 141 that bypass the frame rearrangement unit 131 and input the image signals to the image generation unit 141. An output switching unit 210 that switches the input destination of the frame data of the output color image to either the moving image compression unit 150 or the monitor output unit 220, and an image signal of the color image input via the output switching unit 210 A monitor output unit 220 for outputting to the resolution display 225 is provided. The function of the rearrangement unit bypassing means in the present invention is expressed by the frame rearrangement unit bypassing means 200.

The frame rearrangement bypass unit 200 is arranged in parallel with the frame rearrangement unit 131 and outputs an image signal in the display order to the image generation unit 141 in conjunction with the output switching unit 210. At this time, the frame rearrangement bypass unit 200 may be configured to control the frame rearrangement unit 131 so that the output from the frame rearrangement unit 131 is in display order.

The output switching unit 210 is installed between the image generation unit 141 and the moving image compression unit 150. The output of the image generation unit 141 is input to the moving image compression unit 150 at the time of video recording, and the output of the image generation unit 141 is output to the monitor. Is input to the monitor output unit 220.

The monitor output unit 220 is installed as a video output terminal for outputting an image signal from the imaging apparatus 1C to the outside. When the high-resolution display 225 or the like is connected via the video output terminal, the monitor output unit 220 is connected to the video signal. It is configured to output.

When the image capturing apparatus 1C outputs video to the external high-resolution display 225, the output switching unit 210 is set to input the output of the image generation unit 141 to the monitor output unit 220, and frame rearrangement is performed. The output of the unit 131 is stopped, the frame rearrangement detouring unit 190 operates, the display order image is input to the image generation unit 141, and the high resolution display order video signal is output to the external high resolution display 225 for display. Is done.

On the other hand, in the imaging apparatus 1C, when recording a video, the output switching unit 210 is set to input the output of the image generation unit 141 to the moving image compression unit 150, and the frame rearrangement bypass unit 200 operates. First, an image in the transmission order via the frame rearrangement unit 131 is input to the image generation unit 141 and the moving image compression unit 150.

Thereby, the image pickup apparatus 1C has both functions of color video monitor output and compression recording via the image generation unit 141.

Next, a fourth embodiment of the present invention will be described with reference to FIG. FIG. 6A is a block diagram showing a configuration of an imaging apparatus 1D of the fourth embodiment to which the moving image processing apparatus of the present invention is applied, and FIG. 6B is a configuration diagram of an imaging unit in the same embodiment. It is.

Note that the imaging apparatus 1D in the fourth embodiment is basically the same in configuration as the imaging apparatus 1A shown in the first embodiment. Therefore, the same components are assigned the same reference numerals and detailed description is given. Omitted and characteristic parts will be described below.

As illustrated in FIG. 6A, the imaging device 1D includes an imaging unit 112, a frame rearrangement unit 132 corresponding to the analog signal output from the imaging unit 112, an AFE 120, an image generation unit 140, a moving image compression unit 150, The recording unit 160 is configured.

In the imaging apparatus 1D, a frame rearrangement unit 132 is installed between the imaging unit 112 and the AFE 120 so that the frame sequence is rearranged with respect to the analog signal output from the imaging unit 112.

The image pickup unit 112 is configured by a single-plate color image pickup device as in the first embodiment. The frame rearrangement unit 132 is configured using an analog memory CCD (Charge Coupled Devices) as a frame buffer.

The imaging unit 112 is configured using a CCD image sensor in the same manner as the frame rearrangement unit 132, so that the imaging unit 112 and the frame rearrangement unit 132 are integrated as shown in FIG. A CCD 135 can be used.

As shown in FIG. 6B, the CCD 135 includes a light receiving area 135s as an imaging unit, two buffer areas 135a and 135b, and an I / P frame horizontal transfer unit (I frame and P frame horizontal transfer unit). ), A B frame horizontal transfer unit.

The I / P frame horizontal transfer unit is provided between the light receiving area 135s and the buffer areas 135a and 135b, and the B frame horizontal transfer unit is located on the opposite side of the light receiving area 135s via the buffer area 135a. It is provided at the end of the area 135b.

The function of the frame rearrangement unit 132 is expressed by the two buffer areas 135a and 135b, the horizontal transfer unit for I / P frame, and the horizontal transfer unit for B frame.

In the light receiving area 135s, each element of the CCD 135 (so-called element corresponding to each pixel) is provided with a photoelectric conversion element and a mosaic color filter (a filter that passes RGB single color light). It has been.

Next, operations of imaging and frame rearrangement in the CCD 135 will be described. First, exposure for a predetermined time is performed in the light receiving area 135s, and the exposure amount in each photoelectric conversion element is accumulated as a charge (the first data format in the present invention is this charge amount).

If the exposed image frame is an I frame or a P frame, vertical transfer is performed to the light receiving area 135s, and the horizontal transfer unit for the I / P frame is operated, and the charge of each pixel in the light receiving area 135s is converted into an analog signal. Sequentially output to the AFE 120.

On the other hand, if the exposed image frame is a B frame, vertical transfer is applied to the entire CCD 135 and the horizontal transfer unit for the B frame is operated to output the charge accumulated in the buffer area 135b to the AFE 120 as an analog signal. At this time, the charges accumulated in the light receiving area 135s are accumulated in the buffer area 135a, and the charges accumulated in the buffer area 135a are transferred to the buffer area 135b.

As a result, if the I frame or the P frame is arranged every three frames, the B frame is output to the AFE 120 after the third frame of exposure as in the second embodiment (FIG. 3B). become. Further, since the I frame or the P frame is output to the AFE 120 immediately after the exposure, the frame order input to the AFE 120 and the image generation unit 140 is the transmission order, as in the frame rearrangement unit 131 in the second embodiment.

As described above, according to the imaging device 1D of the fourth embodiment, the frame rearrangement unit 132 can be installed in the CCD 135 integrated with the image sensor, and frame rearrangement as a separate digital process can be made unnecessary.

As mentioned above, although one Example of this invention was described, this invention is not limited to the said Example, It can take a various aspect.

The frame sequence of the moving image output in the first data format from the color image sensor is rearranged in association with the processing order of the compressed stream, and can be used for conversion to a color image.

Claims (15)

1. Video processing that generates a compressed video of color video encoded by the inter-frame prediction method from a video frame sequence that is divided into multiple frames in time series from the color image sensor and output in the first data format In the device
   A frame rearrangement unit that rearranges the frame sequence composed of the first data format in association with the processing order of the compressed stream;
   A color image generation unit that converts the first data format in the frame sequence rearranged by the frame rearrangement unit into a color image;
   A video compression unit that encodes and compresses the frame sequence converted into the color image based on a difference between a plurality of temporally preceding and following frames;
   A moving image processing apparatus comprising:
2. The color image sensor includes a plurality of photoelectric conversion elements arranged in a matrix, and a color filter of a plurality of color lights corresponding to each of the photoelectric conversion elements, and the color image sensor includes a plurality of color lights for each photoelectric conversion element. A single-plate color image sensor that outputs pixel information of single color light,
   The first data format is a color mosaic image having color information of single color light for each pixel,
   The color image generation unit is configured to perform demosaic processing for generating pixel information of a plurality of color lights for each pixel.
   The moving image processing apparatus according to claim 1.
3. The color image sensor is
   Consists of a plurality of color image sensors having different spectral sensitivity distributions, and each light receiving surface is arranged shifted in the pixel arrangement direction,
   The first data format is image data output from each of the plurality of color image sensors,
   The color image generation unit is configured to combine the image data of the plurality of color image sensors to increase resolution.
   The moving image processing apparatus according to claim 1.
4. The plurality of frames are configured by a NonF frame that is encoded without referring to a temporally subsequent frame, and an F frame that is encoded with reference to a temporally subsequent frame,
   The frame rearrangement unit is configured to output with a delay according to the type of the input NonF frame and F frame,
   The moving image processing apparatus according to claim 1, wherein the moving image processing apparatus is a moving image processing apparatus.
5. The NonF frame further includes an I frame that encodes the image signal in the frame as it is without using the inter-frame prediction, and a P frame that encodes a difference from the image signal of the temporally preceding reference frame. Become
   The F frame is a B frame that encodes a difference between a temporally preceding reference frame and a subsequent reference frame;
   The frame rearrangement unit is configured to output with the delay according to the type of the input I frame, P frame, or B frame.
   The moving image processing apparatus according to claim 4.
6. The frame rearrangement unit is configured to output with a delay with respect to some of the plurality of frame types.
   The moving image processing apparatus according to claim 1, wherein the moving image processing apparatus is a moving image processing apparatus.
7. The frame rearrangement unit includes a frame buffer that stores image data for each of at least two frames in the image data of the first data format.
   The moving image processing apparatus according to claim 1, wherein the moving image processing apparatus is a moving image processing apparatus.
8. The image generation unit includes an image deformation processing unit that performs image deformation of the color image.
   The moving image processing apparatus according to claim 1, wherein the moving image processing apparatus is a moving image processing apparatus.
9. The frame rearrangement unit is configured to output non-raster sequential image data in a frame when the image is deformed.
   The moving image processing apparatus according to claim 8.
10. A second image generation unit that generates a color image to be output to the viewfinder, and the second image generation unit generates a color image without performing frame rearrangement in the frame rearrangement unit;
    10. The moving image processing apparatus according to claim 1, wherein the moving image processing apparatus is a moving image processing apparatus.
11. A rearrangement unit bypassing means for outputting the frame sequence output from the color image sensor to the color image generation unit without passing through the frame rearrangement unit;
    The frame sequence input to the color image generation unit is configured to be switchable between a frame order output from the color image sensor and a frame order output from the frame rearrangement unit.
    The moving image processing apparatus according to claim 1, wherein the moving image processing apparatus is a moving image processing apparatus.
12. 12. A camcorder comprising the moving image processing device and color image sensor according to claim 1 and a recording device for storing moving image information compressed by the moving image compression unit.
13. 12. A remote monitor camera comprising: the moving image processing device and the color image sensor according to claim 1; and a transmission device that transmits the moving image information compressed by the moving image compression unit to an external device.
14. Video processing that generates a compressed video of color video encoded by the inter-frame prediction method from a video frame sequence that is divided into multiple frames in time series from the color image sensor and output in the first data format In the method
    A frame rearrangement step of rearranging the frame sequence composed of the first data format in association with the processing order of the compressed stream;
    A color image generation step of converting the first data format in the frame sequence rearranged in the frame rearrangement step into a color image;
    A moving image compression step of encoding and compressing the frame sequence converted into the color image based on a difference between a plurality of temporally preceding and following frames;
    A moving image processing method characterized by using the above.
15. Video processing that generates a compressed video of color video encoded by the inter-frame prediction method from a video frame sequence that is divided into multiple frames in time series from the color image sensor and output in the first data format A program,
    A frame rearrangement step of rearranging the frame sequence composed of the first data format in association with the processing order of the compressed stream;
    A color image generation step of converting the first data format in the frame sequence rearranged in the frame rearrangement step into a color image;
    A moving image compression step of encoding and compressing the frame sequence converted into the color image based on a difference between a plurality of temporally preceding and following frames;
    A video processing program that causes a computer to execute.

Images