WO2012026122A1 - Imaging device - Google Patents

Abstract

An imaging device capable of consistently reducing the volume of data per unit of time by at least a certain amount, regardless of the imaging conditions, is provided. This imaging device is provided with: a photoelectric conversion unit (11), in which a plurality of pixels (Pxy), which convert incoming light to electric signals, are arranged in a two-dimensional shape, and which outputs a plurality of converted electric signals as pixel data; a feature detection unit (14) that detects the features of each area of an image on the basis of the pixel data output by the photoelectric conversion unit; a compression ratio setting unit (13) that sets the compression ratio for each area on the basis of the features detected by the feature detection unit (14); and a compression unit (12) that performs lossy compression on the pixel data output by the photoelectric conversion unit (11) according to the compression ratios set by the compression ratio setting unit (13).

Description

Imaging device

The present invention relates to an imaging apparatus capable of compressing a video signal.

In recent years, technologies that support high-speed imaging functions have made remarkable progress in imaging devices such as digital still cameras. For example, technologies supporting a high-speed imaging function include increasing the number of pixels by miniaturizing the pixel cells of a single-plate color image sensor and increasing the functionality and speed of operation due to technological advances of MOS type image sensors. An image sensor corresponding to the technology that supports the high-speed imaging function is widely used as an image sensor constituting the imaging apparatus.

In the image sensor that constitutes the imaging device, pixel signals output from the image sensor itself have a high data rate in order to cope with high pixel operation and high-speed operation such as high-speed reading. However, there are problems described below in order to cope with this higher data rate. That is, in order to cope with the higher data rate, 1) faster signal receiving side interface circuit, 2) faster processing capacity of digital circuit for processing video signal, and 3) memory for temporarily storing video signal. It is necessary to cope with high data bandwidth and high data capacity. And in order to respond | correspond to these, the circuit scale as the whole camera system will increase, and there exists a subject that causes cost increase and increase in current consumption.

To deal with this problem, for example, Patent Document 1 discloses a technique for reducing the volume of video signal data and reducing the amount of data handled per unit time by performing lossy compression processing by quantization processing of digitally converted image data. ing.

For example, Patent Document 2 discloses a technique for preventing image quality degradation due to compression distortion caused by irreversible compression to a sensor pixel signal, and ON / OFF compression processing for the sensor pixel signal is performed according to shooting conditions and circumstances. A technique for turning off is disclosed.

JP 2007-228515 A JP 2008-1113070 A

However, the above conventional techniques have the following problems.

That is, in the technique for irreversibly compressing the pixel data of the image sensor disclosed in Patent Document 1, the data amount per unit time of the pixel data can be reduced. However, since the compression ratio of irreversible compression is fixed, when the suppression effect is increased in the data amount per unit time, the image observer feels uncomfortable due to, for example, a subject with a high spatial frequency, and is unacceptable. Image quality deterioration due to compression distortion occurs.

Further, as a technique for avoiding this, the above-mentioned Patent Document 2 can be cited. However, in the technique disclosed in the above-mentioned Patent Document 2, ON / OFF of compression processing in units of frames is simply performed according to subject conditions and photographing conditions. Stays switched. Therefore, the pixel signal compression process is not performed at all depending on the photographing conditions, and the function itself for suppressing the data amount per unit time is not implemented.

The present invention has been made in view of the above-described circumstances, and an object thereof is to provide an imaging apparatus that can always reduce the data amount per unit time or more regardless of shooting conditions.

In order to achieve the above object, an imaging apparatus according to an aspect of the present invention includes a plurality of pixels arranged in a two-dimensional manner for converting incident light into an electrical signal, and the plurality of pixels converted by the plurality of pixels. A photoelectric conversion unit that outputs the electrical signal as pixel data, a detection unit that detects a feature for each region of the image based on the pixel data output from the photoelectric conversion unit, and the detection unit that detects the feature A compression rate setting unit that sets a compression rate for each region based on characteristics; and a compression unit that performs irreversible compression on pixel data output from the photoelectric conversion unit according to the compression rate set by the compression rate setting unit. Is provided.

According to the present invention, it is possible to realize an imaging apparatus capable of reducing the data amount per unit time by a certain amount regardless of shooting conditions.

FIG. 1 is a block diagram illustrating a configuration of the solid-state imaging device according to the first embodiment. FIG. 2 is a diagram illustrating a detailed configuration of the photoelectric conversion unit according to the first embodiment. FIG. 3 is a diagram showing an equivalent circuit of the pixel cell in the first embodiment. FIG. 4 is a diagram illustrating a detailed configuration of the compression unit and the expansion unit in the first embodiment. FIG. 5 is a flowchart for explaining the image coding method according to the first embodiment. FIG. 6 is a diagram for explaining an arrangement of pixels used for calculation of a predicted value in the first embodiment. FIG. 7 is a flowchart for explaining the image decoding method according to the first embodiment. FIG. 8A is a diagram showing an example when a plurality of areas are set in the first exemplary embodiment. FIG. 8B is a diagram illustrating an example when a plurality of regions are set in the first exemplary embodiment. FIG. 9 is an example of a flowchart for explaining the compression rate selection method in the rectangular area according to the first embodiment. FIG. 10A is a diagram illustrating an example of a data format of the solid-state imaging element in the first embodiment. FIG. 10B is a diagram illustrating an example of a data format of the solid-state imaging element in the first embodiment. FIG. 11 is a flowchart for explaining a procedure for storing and transmitting independent compressed imaging data for each region in the first embodiment. FIG. 12A is a diagram illustrating an example of a data format of the solid-state imaging element in the first embodiment. FIG. 12B is a diagram illustrating an example of a data format of the solid-state imaging element in the first embodiment. FIG. 13A is a diagram showing an LVDS output of the solid-state imaging device in the first exemplary embodiment. FIG. 13B is a diagram illustrating an LVDS output of the solid-state imaging device according to Embodiment 1. FIG. 14 is a flowchart for explaining a frame data reception procedure in the first embodiment. FIG. 15 is a block diagram illustrating a configuration of a camera that performs motion detection of the entire subject according to the first exemplary embodiment. FIG. 16 is a diagram schematically illustrating an example of an image captured by the solid-state imaging device according to the first embodiment. FIG. 17A is a block diagram illustrating a configuration of a camera according to the second embodiment. FIG. 17B is an example of a motion vector output to the feature detection unit according to the second embodiment. FIG. 18 is a block diagram illustrating a detailed configuration of the image encoding unit according to the second embodiment. FIG. 19A is a schematic diagram of a face area when applied to the face area in the third embodiment. FIG. 19B is a schematic diagram of a face area when applied to the face area in the third embodiment. FIG. 20 is a diagram for explaining a procedure for converting the frame rate by the compression unit according to the fourth embodiment. FIG. 21 is a block diagram illustrating a configuration of the solid-state imaging device according to the fifth embodiment. FIG. 22 is a diagram illustrating an example of imaging data used for AF detection information in the fifth embodiment. FIG. 23 is a flowchart for explaining the feature detection processing when the AF detection information according to the fifth embodiment is used. FIG. 24 is a diagram illustrating an example of contrast in the fifth embodiment. FIG. 25 is a block diagram illustrating a configuration of a digital camera according to the sixth embodiment. FIG. 26 is a block diagram illustrating a configuration of the solid-state imaging device according to the second embodiment. FIG. 27 is a diagram illustrating a relationship between the pixel data level of each pixel of one horizontal line selected in the feature detection unit according to the second embodiment and the selected compression rate.

(Embodiment 1)
Embodiments of the present invention will be described below with reference to the drawings. The embodiment described below is merely an example, and various modifications can be made.

FIG. 1 is a block diagram showing the configuration of the solid-state imaging device according to Embodiment 1 of the present invention.

The solid-state imaging device 1 shown in FIG. 1 includes a solid-state imaging device 10, a feature detection unit 14, an expansion unit 15, and an image processing unit 16.

The feature detection unit 14 detects a feature for each region of the image from the pixel data output from the photoelectric conversion unit 11. Then, the feature detection unit 14 inputs the feature amount to the compression rate setting unit 13. Specifically, the feature detection unit 14 extracts a predetermined feature amount from the input image data, and feeds back the extracted feature amount to the compression rate setting unit 13 of the solid-state imaging device 10.

The decompression unit 15 decompresses the pixel data compressed by the compression unit 12. Specifically, the decompression unit 15 receives the compressed pixel data from the solid-state imaging device 10, decompresses the compressed pixel data, and inputs the decompressed pixel data to the image processing unit 16.

The image processing unit 16 processes the pixel data expanded by the expansion unit 15. Specifically, the pixel data (RAW data) input from the decompression unit 15 is processed, converted into a luminance (Y) signal and a color difference (C) signal, and output to the feature detection unit 14.

The solid-state imaging device 10 includes a photoelectric conversion unit 11, a compression unit 12, and a compression rate setting unit 13. Imaging data (pixel data) of the solid-state imaging device 10 is output from the compression unit 12 to the outside. In the solid-state imaging device 10, the feature amount is input to the compression rate setting unit 13 from the external feature detection unit 14.

The photoelectric conversion unit 11 has a plurality of pixels that convert incident light into electrical signals in a two-dimensional array, and outputs the converted plurality of electrical signals as pixel data. Specifically, the photoelectric conversion unit 11 converts an electrical signal (digital imaging signal) having a magnitude proportional to the incident light, and transfers the converted digital imaging signal to the compression unit 12 as pixel data.

The compression unit 12 performs irreversible compression on the pixel data output from the photoelectric conversion unit 11. Here, the compression unit 12 compresses the pixel data according to the compression rate set by the compression rate setting unit 13. Specifically, the pixel data output from the photoelectric conversion unit 11 is compressed using the compression rate set based on the compression rate information to which the pixel belongs given by the compression rate setting unit 13. The compression unit 12 outputs the compressed pixel data to the decompression unit 15. The compression unit 12 outputs the compressed pixel data to the outside based on a predetermined output method.

The compression rate setting unit 13 sets the compression rate for each area of the image based on the features detected by the feature detection unit 14. Specifically, the compression rate setting unit 13 sets the compression rate based on the feature amount output from the feature detection unit 14 that detects the feature for each region of the image from the pixel data output from the photoelectric conversion unit 11. To do.

As described above, the solid-state imaging device 1 is configured.

Note that the compression rate is set as follows, for example. That is, the feature detection unit 14 detects, from the pixel data expanded by the expansion unit 15, information on the magnitude relationship with respect to a predetermined threshold in the pixel value for each area of the image as a feature, and the compression rate setting unit 13 performs the feature detection. The compression rate is variably set based on the magnitude relationship information detected by the unit 14. Further, the feature detection unit 14 detects information on the magnitude relationship with respect to a predetermined value in the pixel value for each region of the image from the pixel data processed by the image processing unit 16, and the compression rate setting unit 13 14, the compression rate is variably set based on the magnitude relationship information detected by 14. Of course, as will be described later, it goes without saying that the method of determining the compression rate is not limited to this method.

Next, details of each component constituting the solid-state imaging device 1 will be described.

(Details of photoelectric conversion unit 11)
First, the detail of the photoelectric conversion part 11 which comprises the solid-state image sensor 10 is demonstrated.

FIG. 2 is a diagram showing a detailed configuration of the photoelectric conversion unit 11. Here, the photoelectric conversion unit 11 illustrated in FIG. 2 will be described as an example including 8 horizontal pixels and 8 vertical pixels.

As shown in FIG. 2, the photoelectric conversion unit 11 includes, for example, a MOS solid-state imaging device, and includes a pixel cell array 111, a timing generator 112, an AD conversion unit 113, and a horizontal scanning selector 114.

The pixel cell array 111 has a plurality of pixel cells Pxy (x = horizontal coordinates, y = vertical coordinates), and vertical common readout signal lines Lx (x = 1, 2,..., 8). Are connected to the horizontal selection signal lines S1y, S2y, S3y (y = 1, 2,..., 8).

The common readout signal line Lx (x = 1, 2,..., 8) is arranged in the vertical direction, is commonly connected to the pixel cells Pxy in the column direction, and receives signals of the pixel cells Pxy connected in common. Wiring for reading.

The selection signal lines S1y, S2y, S3y (y = 1, 2,..., 8) are arranged in the horizontal direction, and are signals for reading out the accumulation signals accumulated in the selected pixel cells Pxy in the row direction ( A wiring for transmitting a conduction signal or a non-conduction signal) to the pixel cells Pxy. Further, the selection signal lines S1y, S2y, S3y (y = 1, 2,..., 8) are connected to the timing generator 112 and correspond when a high pulse that is a conduction signal is applied.

The plurality of pixel cells Pxy are arranged in a pixel cell array 111 for each pixel in a two-dimensional horizontal and vertical grid, and input incident light is converted into an electrical signal (analog imaging signal) corresponding to the incident light. ). The pixel cell Pxy is connected to the common readout signal line Lx and the selection signal line S1y.

The AD conversion unit 113 is connected to each of the common readout signal line L1 to the common readout signal line L8, and converts the input data (analog imaging signal) into digital data (pixel data).

The horizontal scanning selector 114 includes switches S41 to S48 connected to the AD conversion unit 113. When the switches S41 to S48 are on, the data (pixel data) output from the AD conversion unit 113 is externally output. Output.

The timing generator 112 operates based on the vertical synchronization signal VD, the horizontal synchronization signal HD, and the mode selection signal that are generated outside the photoelectric conversion unit 11 (outside the solid-state imaging device 10). That is, the timing generator 112 selects any one of the selection signal readout line S11 to the selection signal readout line S18 according to a predetermined vertical position from the frame head.

Next, the operation of the photoelectric conversion unit 11 configured as described above will be described.

First, the timing generator 112 selects the selection signal readout line S11 when, for example, a predetermined vertical position from the top of the frame is the first line with reference to the vertical synchronization signal VD.

Next, each of the pixel cells P11 to P18 outputs the accumulated signal accumulated therein to the corresponding common signal readout line L1 to common signal readout line L8. The output accumulated signal is input to the corresponding AD conversion unit 113.

Next, the AD conversion unit 113 converts the data read from the pixel cells P11 to P18 into a digital signal (digital image pickup signal) binarized into N bits (N is a natural number).

Subsequently, in the horizontal scanning selector 114, the selection signal of any one of the switches S41 to S48 becomes valid (ON) at a predetermined horizontal timing with the horizontal synchronization signal HD as a reference. Then, the data output from the AD conversion unit 113 connected to the switches S41 to S48 is output to the outside of the photoelectric conversion unit 11.

The photoelectric conversion unit 11 operates as described above.

Here, an example of a specific circuit (equivalent circuit) constituting the pixel cell Pxy described above will be described. FIG. 3 is a diagram showing an equivalent circuit of the pixel cell.

As shown in FIG. 3, the pixel cell Pxy includes a photodiode 1101, a read transistor 1102, a floating diffusion 1103, a reset transistor 1104, and an amplifier 1105.

The photodiode 1101 photoelectrically converts incident light (incident light) to generate electric charge according to the incident light.

The read transistor 1102 has one of a source and a drain connected to the gate (cathode) of the photodiode 1101 and the other of the source and the drain connected to the floating diffusion 1103. Further, the read transistor 1102 is connected to the read signal line at the gate. Therefore, the read transistor 1102 reads the charge generated in the photodiode 1101 to the floating diffusion 1103 when a read signal is applied (turned on) to the read signal line.

The floating diffusion 1103 has a capacitance, and converts the signal charge read through the read transistor 1102 into a voltage according to the capacitance.

The source or drain of the reset transistor 1104 is connected to the floating diffusion 1103, and when the reset signal is input to the gate, the charge of the floating diffusion 1103 is reset. In other words, the floating diffusion 1103 is reset via the reset transistor 1104 to which the reset signal is input to the gate before the charge from the photodiode 1101 is read.

The amplifier 1105 has a floating diffusion 1103 connected to its input terminal and a common signal readout line Lx (x = 1, 2,..., 8) connected to its output terminal. Therefore, the amplifier 1105 amplifies the voltage of the floating diffusion 1103 and outputs it to the common signal readout line Lx (x = 1, 2,..., 8). For example, in the pixel cell P11 shown in FIG. 2, the amplifier 1105 outputs the amplified voltage to the common signal readout line L11 via a switching element (not shown).

Next, the movement of the pixel cell Pxy configured as described above will be described. Note that in FIG. 2, a signal line (read signal line) for transmitting a read signal input to the read transistor 1102 and a signal line (transmit signal signal) (for transmitting a reset signal input to the reset transistor 1104). The reset signal line is not shown. Both the readout signal line and the reset signal line are applied in common to the pixel cells Pxy in each row from the timing generator 112.

First, a reset signal is applied to the reset transistor 1104 to reset the floating diffusion 1103.

Next, the readout signal is applied (turned on) to the readout transistor 1102, whereby the charge of the photodiode 1101 is read out to the floating diffusion 1103. Such driving is performed in units of rows of the pixel cells Pxy.

Next, a high pulse indicating a conduction signal is applied to the selection signal readout line S11 to the selection signal readout line S18 commonly connected to the pixel cells Pxy in each row.

As a result, an electric signal (analog pixel signal, imaging signal) proportional to the light incident on the pixel cell Pxy is output from the amplifier 1105.

Next, details of the compression unit 12 and the expansion unit 15 constituting the solid-state imaging device 10 will be described.

Here, regarding the “compression rate” described below, the higher the compression rate, the smaller the ratio of the code amount after compression to the code amount before compression, and conversely, the lower the compression rate, the more compression with respect to the code amount before compression. This means that the subsequent code amount ratio becomes large and approaches the code amount before compression. Therefore, high compression means that compression is performed at a high compression rate, and low compression means that compression is performed at a low compression rate. “Non-compressed” means that compression is not performed and the code amount does not change.

(Details of compression section)
FIG. 4 is a diagram illustrating detailed configurations of the compression unit 12 and the decompression unit 15.

First, the compression unit 12 will be described. The compression unit 12 illustrated in FIG. 4 includes a processing target pixel value input unit 121, a prediction pixel generation unit 122, a code conversion unit 123, a change extraction unit 124, a quantization width determination unit 125, and a quantization processing unit 126. And a packing part 127.

The processing target pixel value input unit 121 receives a value (pixel data) of a pixel (hereinafter referred to as a target pixel) to be encoded by the compression unit 12 by the photoelectric conversion unit 11. The processing target pixel value input unit 121 receives pixel data from the photoelectric conversion unit 11 and outputs the input pixel data to the prediction pixel generation unit 122 and the code conversion unit 123 at an appropriate timing. When the input pixel data is input as the pixel data of the first target pixel, the processing target pixel value input unit 121 omits the quantization process and directly outputs the input pixel data to the packing unit 127. To do. On the other hand, when the input pixel data is not the pixel data of the first target pixel, the processing target pixel value input unit 121 outputs the pixel data to the prediction pixel generation unit 122 and the code conversion unit 123 at an appropriate timing.

The predicted pixel generation unit 122 generates a predicted value of the current target pixel of interest using the input pixel data. The prediction pixel generation unit 122 outputs the generated prediction value to the code conversion unit 123.

The code conversion unit 123 performs code conversion on each of the encoding target pixel (pixel data) input from the processing target pixel value input unit 121 and the prediction value input from the prediction pixel generation unit 122, and the change extraction unit 124. Output to.

The change extraction unit 124 performs an exclusive OR operation on the input encoding target pixel code and the prediction value code to calculate bit change information. The change extraction unit 124 outputs the calculated bit change information to the quantization width determination unit 125 and the quantization processing unit 126.

The quantization width determination unit 125 determines the quantization width based on the bit change information input from the change extraction unit 124 and outputs the quantization width to the quantization processing unit 126 and the packing unit 127.

The quantization processing unit 126 performs a quantization process for quantizing the bit change information input from the change extraction unit 124 using the quantization width calculated by the quantization width determination unit 125.

The packing unit 127 packs data into a combination of at least one or more leading target pixels, a plurality of quantized values, and at least one quantized width information. Then, the packing unit 127 outputs the packed packing data to a memory such as an SDRAM or the unpacking unit 151.

Next, a compression method (image encoding) of the compression unit 12 configured as described above will be described.

<Image coding processing>
FIG. 5 is a flowchart for explaining the image coding method according to the present embodiment.

Note that all or part of the encoding process of the compression unit 12 described below is not limited to the configuration in the solid-state imaging device 10. The compression unit 12 may be configured by hardware such as LSI (Large Scale Integration), a program executed by a CPU (Central Processing Unit), or the like. The same applies to the following.

In this embodiment, each pixel data is N-bit digital data, and pixel data after quantization (hereinafter referred to as a quantized value) corresponding to each pixel data is M-bit length. Further, in the compression unit 12, a leading target pixel of at least one pixel, a quantization value corresponding to a plurality of pixel data, and a code representing a quantization width of the quantization value (hereinafter referred to as quantization width information) Are packed into the S bit length by the packing unit 127. Then, the packed packing data is output. Here, the natural numbers N, M, and S are determined in advance.

First, pixel (target pixel) data (pixel data) to be encoded is input to the processing target pixel value input unit 121 by the photoelectric conversion unit 11.

Next, the pixel data input to the processing target pixel value input unit 121 is output from the processing target pixel value input unit 121 to the prediction pixel generation unit 122 and the code conversion unit 123 at an appropriate timing.

However, when the encoding target pixel of interest is input as the first target pixel (YES in step S101), the quantization processing is omitted, and the processing target pixel value input unit 121 is directly input. Pixel data is input to the packing unit 127. On the other hand, when the encoding target pixel of interest is not the first target pixel (NO in step S101), the process proceeds to a prediction pixel generation process.

Here, the pixel data input to the predicted pixel generation unit 122 is any of the first data to the third data. That is, the first data is the first target pixel that is input to the processing target pixel value input unit 121 before the target encoding target pixel. The second data is a previous encoding target pixel that is input to the processing target pixel value input unit 121 first. The third data is first encoded by the compression unit 12, the encoded encoded data is sent to the decompression unit 15, and the transmitted encoded data is decoded by the expansion unit 15 Pixel data.

Next, the predicted pixel generation unit 122 generates a predicted value of the current target pixel of interest using the input pixel data (step S102).

Note that there is a predictive encoding method as an encoding method for encoding pixel data. Predictive coding is a method of generating a prediction value for an encoding target pixel and encoding a difference value between the encoding target pixel and the prediction value. In the case of pixel data, this predicted value is likely to be the same as or close to the value of the pixel close to the target pixel. Therefore, based on this, the value of the target pixel to be encoded is predicted from neighboring pixel data. In this way, the difference value is made as small as possible to suppress the quantization width.

Here, calculation of predicted values will be described.

FIG. 6 is a diagram for explaining the arrangement of pixels used for calculation of a predicted value.

6 indicates the pixel value of the target pixel (target pixel). Further, a, b, and c shown in FIG. 6 indicate pixel values of three adjacent pixels with respect to the target pixel, which are used to obtain the predicted value y of the target pixel.

Here, prediction formulas generally used are shown in (Formula 1) to (Formula 7).

y = a (Formula 1)
y = b (Formula 2)
y = c (Formula 3)
y = a + bc (formula 4)
y = a + (bc) / 2 (Formula 5)
y = b + (ac) / 2 (Formula 6)
y = (a + b) / 2 (Expression 7)

Thus, the predicted pixel generation unit 122 obtains the predicted value y of the target pixel using the pixel values a, b, and c of the neighboring pixels of the target pixel. The prediction pixel generation unit 122 obtains a prediction error Δ (= y−x) between the prediction value y and the encoding target pixel x, and encodes the prediction error Δ. Then, the prediction pixel generation unit 122 calculates a prediction value using any one of the (Expression 1) to (Expression 7) used in the prediction encoding described above, and converts the calculated prediction value into code conversion. Output to the unit 123. In addition, when an internal memory buffer that can be used in compression processing can be secured, not only the prediction formula described above, but also peripheral pixels other than the pixel adjacent to the target pixel are held in the memory buffer, It may be used for prediction. In that case, it is also possible to use a prediction formula that improves the prediction accuracy. In the present embodiment, as an example, the case where the prediction formula (Formula 1) is used in Step S102 has been described.

Next, the code conversion unit 123 performs code conversion on each of the encoding target pixel received from the processing target pixel value input unit 121 and the prediction value received from the prediction pixel generation unit 122 into a code expressed by N bits. To do. The code converted code corresponding to the encoding target pixel (hereinafter referred to as the target pixel code) and the prediction value corresponding code (hereinafter referred to as the prediction value code) are sent to the change extraction unit 124. Sent by the converter 123 (step S103).

Next, the change extraction unit 124 performs an exclusive OR operation between the code of the pixel to be encoded and the code of the prediction value expressed in N bits, and calculates bit change information E having an N-bit length. The bit change information is a code for calculating the code of the target pixel from the bit change information and the code of the predicted value. Then, the change extraction unit 124 outputs the calculated bit change information E to the quantization width determination unit 125 and the quantization processing unit 126.

Next, the quantization width determination unit 125 determines the quantization width J based on the bit change information E sent from the change extraction unit 124 to the quantization width determination unit 125, and the determined quantization width J The data is output to the conversion processing unit 126 and the packing unit 127 (step S104). The quantization width J indicates a value obtained by subtracting the bit length M of the quantization value from the number of effective bit digits of the bit change information E. Here, J is a positive integer, and when the number of significant bit digits of the bit change information E takes a value smaller than M, J is set to 0.

Next, the quantization processing unit 126 performs quantization processing for quantizing the bit change information E received from the change extraction unit 124 using the quantization width J calculated by the quantization width determination unit 125 (step S106). . The quantization process using the quantization width J means that the bit change information E between the encoding target pixel and the predicted value corresponding to the encoding target pixel is shifted downward by the number of the quantization width J ( Bit shift). Further, the quantization processing unit 126 may understand that the quantization is not performed when the quantization width J is zero. The quantization result (quantized value) output from the quantization processing unit 126 is sent to the packing unit 127 by the quantization processing unit 126.

Next, the packing unit 127 combines at least one or more leading target pixels, a plurality of quantization values, and at least one or more quantization width information of Q bit length (Q is a natural number). , Packing into S-bit data (step S107). Then, the packing unit 127 outputs the packed packing data to a memory such as an SDRAM or the unpacking unit 151. The fixed bit length S to be set may be the same number of bits as the data transfer bus width of the integrated circuit to be used. In addition, when unused bits remain in the rear portion of the packing data bits, dummy data is recorded so as to reach the S bits.

When the encoding target pixel packing process is completed, the process proceeds to step S108.

Next, in step S108, the compression unit 12 determines whether or not the image encoding process has been completed for the number of pixels Pix packed into S bits. Here, it is assumed that Pix is calculated in advance by the following (Equation 8).

Pix = S / (Q + M) (Equation 8)

In step S108, if the result of the determination is NO (NO in step S108), the process proceeds to step S101, and then for the pixel data received by the processing target pixel value input unit 121, from step S101 to step S107. The compression unit 12 executes at least one of the following processes. On the other hand, if the determination result is YES in step S108 (YES in step S108), the compression unit 12 outputs the encoded data in the buffer memory in units of S bits, and proceeds to step S109.

Finally, in step S109, the compression unit 12 determines whether the encoding process for one image has been completed by the output of the encoded pixel data output in the previous step S108. If the determination result is YES (YES in step S109), the encoding process ends. Conversely, if NO (NO in step S109), the process proceeds to step S101, and at least one process from step S101 to step S108 is executed.

As described above, the compression unit 12 performs compression processing (image encoding processing).

As described above, this image encoding method inputs image data of a pixel to be compressed (processing object pixel value input unit 121), and compresses the input pixel data (to a fixed-length code). Is the method. Then, a prediction pixel generation step (prediction pixel generation unit 122) that generates a prediction value of the pixel data from at least one peripheral pixel located around the pixel to be compressed, and code conversion of the pixel data A code conversion step (code conversion unit 123) that generates a code in which the pixel data is code-converted; and the code of the pixel data that has been code-converted in the code conversion step and the prediction pixel generation step. Further, the bit change information (change extraction unit 124) between the prediction value code and the code value is converted into a quantized value having a bit number smaller than (and having a fixed length) the bit number of the bit change information. An image code including a quantization step (quantization processing unit 126) that compresses the pixel data into the quantized values by quantization. It is a method.

(Details of extension)
Next, the decompression unit 15 will be described.

4 includes an unpacking unit 151, a prediction pixel generation unit 152, a quantization width determination unit 153, an inverse quantization processing unit 154, a code generation unit 155, and an inverse code conversion unit. 156 and an output unit 157.

The unpacking unit 151 analyzes the encoded data input (sent) from the packing unit 127 or a memory such as an SDRAM and separates it into a plurality of data. The unpacking unit 151 outputs the analyzed encoded data to the quantization width determination unit 153, the inverse quantization processing unit 154, and the output unit 157 at an appropriate timing.

The quantization width determination unit 153 determines the quantization width in the inverse quantization process corresponding to each decoding target pixel from the encoded data (quantization width information) input from the unpacking unit 151, and performs inverse quantization. Is output to the processing unit 154.

The inverse quantization processing unit 154 performs inverse quantization using the quantization width in the inverse quantization process input from the quantization width determining unit 153.

The code generation unit 155 performs code conversion similar to the code conversion in the code conversion unit 123 on the prediction value input from the prediction pixel generation unit 152, and generates a code of the prediction value.

The reverse code conversion unit 156 performs reverse conversion of the code conversion performed by the code conversion unit 123 on the target pixel code input by the code generation unit 155 to restore the pixel data. Then, the reverse code conversion unit 156 outputs the pixel data subjected to the reverse code conversion to the output unit 157.

The predicted pixel generation unit 152 generates a predicted value using the input pixel data. Here, the input data is data that is input before the target decoding target pixel and is output from the output unit 157.

Next, a decompression method (image decoding) of the decompression unit 15 configured as described above will be described.

<Extension processing>
FIG. 7 is a flowchart for explaining the image decoding method according to the present embodiment. FIG. 7 shows image decoding processing (decompression processing) performed by the decompression unit 15.

First, encoded data is input to the unpacking unit 151. Here, the encoded data input to the unpacking unit 151 is encoded data necessary for restoring each pixel data.

Next, the unpacking unit 151 analyzes the S-bit fixed-length encoded data sent from the packing unit 127 or sent from a memory such as SDRAM, and converts the fixed-length encoded data into a plurality of pieces. Separate into data. In other words, the unpacking unit 151 converts the sent fixed-length encoded data into a N-bit long target pixel, a Q-bit quantization width information, and a M-bit decoding target pixel (hereinafter, referred to as a pixel to be decoded) It is separated into pixels to be decoded (quantized values) (step S201).

Next, the encoded data analyzed by the unpacking unit 151 is transmitted to the quantization width determining unit 153, the inverse quantization processing unit 154, and the output unit 157 at an appropriate timing.

If the encoded data of interest is input as the first target pixel (YES in step S202), the encoded data is received as N-bit pixel data holding the dynamic range before encoding. For this reason, the unpacking unit 151 omits the inverse quantization process and transmits the encoded data of interest to the output unit 157 directly.

Next, when the encoded data of interest is quantization width information (YES in step S203), the unpacking unit 151 transmits the encoded data to the quantization width determination unit 125, and performs inverse quantization. The process proceeds to quantization width determination processing in the conversion (step S204). The quantization width determination unit 153 determines the quantization width J ′ in the inverse quantization process corresponding to each decoding target pixel from the encoded data (quantization width information) received from the unpacking unit 151 and determines The quantized width J ′ thus output is output to the inverse quantization processing unit 154.

On the other hand, when the encoded data of interest is not quantization width information (NO in step S203), the unpacking unit 151 transmits the encoded data to the inverse quantization processing unit 154, and performs inverse quantization. Shift to the conversion process.

Next, in the inverse quantization processing, the inverse quantization processing unit 154 performs inverse quantization using the quantization width J ′ in the inverse quantization processing received from the quantization width determination unit 153. The inverse quantization process using the quantization width J ′ is a process of bit-shifting the encoded data (quantized value) received from the unpacking unit 151 upward by the number of J ′. The inverse quantization processing unit 154 calculates bit change information E ′ represented by N bits by inverse quantization processing. Note that the inverse quantization processing unit 154 may be understood not to perform the inverse quantization when the quantization width J ′ is “0” (step S205).

Here, the data input to the prediction pixel generation unit 152 is data that is input before the target pixel to be decoded and output from the output unit 157. This data is either the first target pixel or pixel data decoded first and output from the output unit 157 (hereinafter referred to as decoded pixel data).

Next, the prediction pixel generation unit 152 generates a prediction value represented by N bits using the pixel data input to the prediction pixel generation unit 152 as described above. Here, the prediction value generation method uses any one of the above-described prediction expressions (Expression 1) to (Expression 7), but uses a prediction expression similar to the expression used in the prediction pixel generation unit 122. The predicted pixel generation unit 152 calculates a predicted value. The calculated prediction value is output to the code generation unit 155 by the prediction pixel generation unit 152 (step S206).

Next, the code generation unit 155 performs code conversion similar to the code conversion in the code conversion unit 123 on the prediction value received from the prediction pixel generation unit 152, and the code of the prediction value after the conversion is performed. Generate. That is, the received predicted value is a value before code conversion such as Gray code conversion. The code generation unit 155 performs the same code conversion as the code conversion performed by the code conversion unit 123 on the received predicted value, and calculates the converted code. Thereby, the code generation unit 155 calculates a code corresponding to the received predicted value. Furthermore, the code generation unit 155 performs an exclusive OR operation between the N-bit length bit change information E ′ received from the inverse quantization processing unit 154 and the code of the predicted value after the code conversion. The N-bit target pixel code is generated, and the generated target pixel code is output to the inverse code conversion unit 156 (step S207).

Next, the reverse code conversion unit 156 performs reverse conversion of the code conversion performed by the code conversion unit 123 on the target pixel code received from the code generation unit 155 to restore the pixel data. The pixel data after the reverse code conversion is transmitted to the output unit 157 by the reverse code conversion unit 156 (step S208).

Next, in step S209, for example, the unpacking unit 151 determines whether or not the image decoding process has been completed for the number of pixels Pix packed into S bits by the packing unit 127. Here, it is assumed that Pix is calculated in advance using (Equation 8), as in the image encoding process.

In step S209, if the determination result for Pix is NO (NO in step S209), the process proceeds to step S203, and the decompression unit 15 receives the next code received by the unpacking unit 151. At least one of steps S203 to S208 is performed on the digitized data. The decompressing unit 15 repeatedly performs the processing from step S203 to step S208 on the pixels P3 to P6, and sequentially outputs the target pixels obtained by the processing.

On the other hand, if the determination result of the above determination regarding Pix is YES in step S209 (YES in step S209), the process proceeds to step S210.

Next, in step S210, the unpacking unit 151 determines whether or not the decoding process for one image has been completed with the decoded pixel data output from the output unit 157. If the determination result is YES (YES in step S210), the decoding process is terminated. Conversely, if NO (NO in step S210), the process proceeds to step S201, and at least one process from step S201 to step S209 is executed.

As described above, the decompression unit 15 performs decompression processing (image decoding processing).

(Compression rate setting part)
Next, details of the compression rate setting method of the compression rate setting unit 13 constituting the solid-state imaging device 10 will be described using an example.

The compression rate setting unit 13 sets the compression rate based on the feature amount output from the feature detection unit 14 as described above. Here, various information about the feature amount input from the feature detection unit 14 can be mentioned. Hereinafter, as an example, the case where the feature detection unit 14 individually sets a compression rate for a specific target area in the imaging screen will be described. The number of target areas, the reference coordinates of each area, the size of the area, The case where is input will be described.

The compression rate setting unit 13 receives the feature amount from the feature detection unit 14 through a connected control signal, and holds the input feature amount in, for example, an internal register. Here, the compression rate setting unit 13 has, for example, an internal register and holds information in the internal register.

When the input feature value is held in the internal register, N registers may be prepared that can hold all pieces of area information for N (N is a natural number). In addition, when the scanning of the region m (m is a natural number equal to or less than N) is clearly observed after observing the time when the output scanning is performed from the beginning of the frame, the internal register storing the information of the region n Thus, the information may be newly updated to the information of the region n.

The compression rate setting unit 13 calculates coordinates from the reference position in the horizontal direction and the vertical direction with the upper left corner of the screen to be imaged as a reference. Then, the compression rate setting unit 13 compares and determines whether or not the calculated horizontal coordinate and vertical coordinate are included in up to N regions that are outputs of the feature detection unit 14. Here, as the area information, one area may be set for the entire image, and the information is not limited to one, but the first area, the second area,. A plurality of areas may be set. Hereinafter, an example in which a plurality of areas are set will be described. 8A and 8B are diagrams illustrating an example in which a plurality of areas are set. FIG. 8A is an example showing a rectangular area in the case of N = 2. On the other hand, FIG. 8B is an example showing a non-rectangular region.

In FIG. 8A, as the area information, the area number N, the reference coordinates P (N) of each area N = (horizontal left edge, vertical top edge), the horizontal pixel number W (N), and the vertical pixel number H (N) of the area. Have been entered.

Here, the compression rate Q (N) for each region N and the standard compression rate Q for a region that does not belong to any of the N rectangular regions are selected. That is, the feature detection unit 14 selects a region of interest in the image as a rectangular region. Then, the compression rate setting unit 13 reduces the compression rate of the region of interest to leave details, and if the compression rate is set so that high compression is performed with the non-target region as the compression rate Q, the compression unit 12 is efficient. Can be compressed.

Next, the compression rate area determination method will be described.

FIG. 9 is an example of a flowchart for explaining a compression rate selection method in a rectangular area.

First, after frame imaging is started, the feature detection information of the region is input and initialization is performed (step S301), and the process proceeds to step S302.

Next, in step S302, it is determined by (Equation 9) below whether the coordinates (x, y) with respect to the upper left of the screen are included in any of the N areas.

Inside rectangular area 1: (X1 <= x <X1 + W (1), Y1 <= y <Y1 + W (1))
Inside the rectangular area 2: (X2 <= x <X2 + W (1), Y2 <= y <Y2 + W (2))
:
Inside rectangular area N: (XN <= x <XN + W (N), YN <= y <YN + W (N)) (Equation 9)

Next, when the coordinate (x, y) in the image is included in the rectangular area n (n = 1, 2,..., N) out of N and YES (YES in step S302). Then, the compression ratio = Q (n) is set (step S303), and the process proceeds to step S305.

On the other hand, in the case of NO (NO in step S302), the standard compression rate outside the region is separate from the compression rate set in the region up to N as the region not belonging to the N target rectangular regions (non-selected region). As Q (step S304), the process moves to step S305.

Next, in step S305, it is determined whether or not the determination of all N areas has been completed, and when completed (YES in step S305), the process moves to step S306, and when not completed (NO), the area number n is set to 1. After increasing, the process returns to step S302, and the next area n + 1 is discriminated.

In step S306, it is determined whether or not the horizontal pixel scanning is completed. If YES, the process moves to step S307. If NO, the horizontal coordinate x is incremented by 1, and the process returns to step S302 to continue.

In step S307, it is determined whether or not the vertical pixel scanning is completed in step S306. If YES, the compression ratio selection in one frame is completed. If NO, the vertical coordinate y is increased by 1. Then, the process returns to step S302 and continues.

Note that the compression rate Q (n) of any one of the regions n may be applied to pixels included in two or more regions out of the N rectangular regions. In this example, since the compression rate region is determined for the rectangular region, in step S302 during scanning, the compression rate Q (n set in the region at the stage where the upper left reference coordinate of the rectangular region n is matched. ) Sets the same compression rate. Therefore, it is not necessary to scan all the pixels in the horizontal W pixel and vertical H pixel regions one by one, and it is possible to skip, and the processing amount can be reduced by skipping.

In addition, here, the case where the region shape is a rectangular region has been described as an example, but the shape is not limited to a rectangular region, and any shape may be used. The case where the region shape is not rectangular may be, for example, the case of FIG. 8B, which can be dealt with by giving region information as shown in FIG. 8B.

The compression rate selection method in the area information in FIG. 8B is the same as the method described in FIG. 9, but unlike the rectangular area, the upper left coordinate cannot be used as the reference coordinate. Therefore, for example, the feature detection unit 14 may give information to the compression rate setting unit 13 as follows.

That is, when data is output from the solid-state imaging device 10 in units of lines, for each line l, the region number n and the leftmost coordinate including the region n are set as reference coordinates P (l, n), and P (l, Information such as n) = (Xn, Yl) and the number of horizontal pixels W (n) of the region n may be given for each line.

(data form)
Next, the data format when the solid-state imaging device 10 outputs the compressed data compressed by the compression unit 12 will be described.

10A and 10B are diagrams showing examples of the data format of the solid-state imaging device. Hereinafter, when outputting from the solid-state imaging device 10, scanning is performed in the horizontal direction and continuous data is output in units of lines.

FIG. 10A is an example schematically showing data stored in units of one line in the horizontal direction. Here, data for one line is arranged in the horizontal direction and data for each line is arranged in the vertical direction, and the data is stored with the synchronization code (header) added to the head as the storage form of the data of each line. In addition, the synchronization code does not straddle between lines, and is added and stored independently for each line.

As shown in FIG. 10A, the first line is data of the frame start line (first line), and a frame start synchronization code (SOF) is added to the head of this data. Similarly, in the second and subsequent lines, a line start synchronization code (SOL) is added to the head of the data of one line.

Also, immediately after the SOL, an identification synchronization code (SOR (n)) indicating each area n is added. The synchronization code SOR (n) may be added with information indicating, for example, a code indicating the compression rate for 1 byte, the number of pixels to be compressed for 2 bytes, and the total number of bits after compression for 4 bytes. The number of bytes may be set in each solid-state imaging device and is not a unique value. By adding this SOR (n), the decompression unit 15 can detect and expand only a specific area by detecting SOR (n).

Also, as the number of areas N per line increases as the size of the synchronization code SOR (n) indicating the area increases, the ratio of the header in one line increases and the data compression efficiency decreases, so the header size can be reduced. Needless to say, this is more appropriate.

Furthermore, a line end code EOL is added to the end of the last data of one line, and a frame end code EOF is added to the end of the frame last line.

Here, the identification headers such as SOF, SOL, and SOR are used to correctly identify and decode the imaging data, but 0 or 1 is generated in advance for the number of bits exceeding the bit length of the imaging data in advance. To achieve.

FIG. 10B is an example showing a sequence of SOR synchronization codes.

As shown in FIG. 10B, for the compression rate described in the header, for example, when 1 byte is prepared, in the case of non-compression, that is, the code amount ratio after compression to 100% after compression, the maximum value is 8 bits or 0. Not limited to that.

It should be noted that, at the time of compression, for example, a conversion table between a value of 1 to 254 and a compression rate may be prepared and converted and defined. Alternatively, the lower 4 bits may be defined as a denominator and the upper 4 bits as a numerator, and each may be defined in terms of a 4-bit value. In the case of this definition, the value at the time of 75% compression of the code amount ratio after compression with respect to before compression may be shown as 0x34, for example. Here, there are combinations of 0x68 and 0x9C other than 0x34, but they do not matter.

Also, as described above, it is not always necessary to create a numerator denominator relationship between the upper bits and the lower bits, and it is sufficient if the compression rate is uniquely known without overlapping in one byte. That is, for example, when the code amount ratio after compression with respect to before compression is 75%, the code amount ratio after compression with respect to before direct compression may be described as 0x4B (= 75). Or you may describe the quantization step used at the time of compression.

Next, a procedure for transmitting independent compressed imaging data for each area will be described.

FIG. 11 is a flowchart for explaining a procedure for transmitting compressed image data that is independent for each region.

First, transfer of frame data is started.

Next, feature information such as region information and compression rate information input from the feature detection unit 14 in step S401 is acquired and set, and the process proceeds to step S402.

Next, in step S402, the line number counter j = 1 is reset.

Next, in line-by-line processing, in step S403, an area number counter i = 1 for counting the number of areas for each line number is initialized.

Next, in step S404, if the first line (j = 1) from the value of the line number counter j (YES in step S404), a frame start code (SOF) is added (step S405). On the other hand, if j is not 1 (NO in step S404), a line start header (SOL) is added (step S406).

Next, in step S407, since it is the left end of the line, an area start code SOR0 indicating the brother 1 area is added, and information on the compression rate, the number of pixels in the area i, and the number of bits after compression is included in the area start code. Append.

Next, in step S408, the compressed imaging RAW data is stored, and in step S409, it is determined whether or not the area number counter i = the set area number N is reached. If not (NO in step S409), i = I + 1 (step S412), the process returns to step S407, and the storage of data in the second area and thereafter is continued. On the other hand, if the number of set areas has been reached in step S409 (YES in step S409), the process moves to step S410.

Next, if it is not the last line in step S410 (NO in step S410), a line end header (EOL) is added in step S413, the process returns to step S403, and the next line processing is continued. On the other hand, if j = L and the final line is selected (YES in step S410), the process ends by adding a frame end code EOF.

As described above, based on the compression rate information set by the compression rate setting unit 13, the compression unit 12 compresses pixel data by adding at least identification information including compression rate information and data amount information. To do. Then, the solid-state imaging device 10 outputs image data to which identification information is added.

The format of the compressed data has been described based on FIGS. 10A and 10B, but is not limited thereto. For example, as a modification, the format shown in FIGS. 12A and 12B may be used. That is, as shown in FIG. 12A, the SOL may be added after the SOF is added in the frame start line, and the EOL may be added before the EOF is added in the final frame end line. Further, as shown in FIG. 12B, in the last line of the frame, the processing may be ended only by EOL instead of adding EOF.

In addition, the data format described above is particularly effective when the solid-state imaging device 10 outputs with LVDS (Low Voltage Differential Signaling). Here, the LVDS is an interface that converts two differential signals into one set (pair) and operates at a low voltage and a high frequency to convert them into serial signals and perform high-speed transfer.

Next, a case where the solid-state imaging device 10 outputs with LVDS will be described.

First, the solid-state imaging device 10 includes one or more LVDS output pairs. When two or more output pairs are involved, the pairs are divided and assigned based on a predetermined rule. Hereinafter, a method for allocating and dividing pairs will be described with reference to FIGS. 13A and 13B.

FIG. 13A and FIG. 13B are diagrams showing LVDS output of the solid-state imaging device in the first embodiment. FIG. 13A shows an example in which the number of bits of one pixel data per pair is set to 10 bits, and four pairs A to D are output.

After compression in the compression unit 12 of one line, serial conversion is performed by the serial conversion 171, output control is performed by the output control unit 172, and output is performed to each of the connected A to D LVDS transceivers.

Fig. 13B describes the output format of each pair.

Since each LVDS pair performs transmission and reception independently, at the left end of the line, the output control unit 172 adds it to each of the A to D LVDS pairs that distribute the SOL header.

Thereafter, first, the code SOR (0) of the first area is stored for the pairs A to B, C, and D. Next, the compressed data is distributed and outputted to each pair of A to D by 10 bits. When the total size of the compressed data in the area is a fraction of 40 bits, dummy data in which all the surplus bits are filled with 1 is output, moved to the next area, and stored. Thereafter, the data is stored up to the end of EOL.

As described above, the pairs A to D each output independently.

It should be noted that the number of pairs per LVDS port and the number of bits per pixel are not limited to this example, and the number of pairs within the range up to the number of pairs connected by this solid-state imaging device and the number of bits of the AD conversion unit of the solid-state imaging device Settings can be changed.

<Frame data processing>
Here, a procedure in which the decompression unit 15 receives the LVDS output compressed data of the solid-state image sensor 10, decompresses, and restores the imaging RAW data usable in the image processing unit 16 will be described with reference to FIG.

FIG. 14 is a flowchart for explaining a frame data reception procedure in the first embodiment. The decompression unit 15 repeats the operation of converting each LVDS pair into parallel data in units of a predetermined number of bits after decompression is started.

First, in step S501, each pair waits to receive a frame start synchronization code (SOF).

After receiving the SOF (YES in step S501), the process moves to step S503. On the other hand, when the SOF is not received (NO in step S501), the reception determination of the line start code SOL is performed in step S502. If received (YES in step S502), the process moves to step S503. If not received (NO in step S502), the process moves to step S501 and continues until the SOF or SOL start code is received.

Next, in step S503, after receiving the synchronization code SOR (n) of the region n (YES in step S503), information such as the number of pixels, the compression rate, and the data size of the region n is acquired in step S504. Next, the process proceeds to step S505, where decompression processing of the compressed data in the region n is performed. After the decompression is completed, the process proceeds to step S506, where reception determination of the frame end code EOF is performed. If the frame end code EOF is received (YES in step S506), the process ends. On the other hand, if the frame end code EOF is not received (NO in step S506), the process proceeds to step S507 to determine whether or not the line end code EOL is received. If the line end code EOL is received (YES in step S507), the process returns to step S501 to continue the processing for the next line and thereafter. On the other hand, if not received (NO in step S507), the process returns to step S503 to continue the process for the next area.

Note that the decoding processing in step S505 is shown by taking the processing of the decompression unit 15 described in FIG. 7 as an example.

Example 1
In this example, a case where the solid-state imaging device 1 according to Embodiment 1 is applied to, for example, an in-vehicle camera or the like will be described. Specifically, an example in which features are detected based on motion information of the entire image will be described.

FIG. 15 is a block diagram illustrating a configuration of a camera that performs motion detection of the entire subject in the first embodiment. Elements similar to those in FIG. 1 are denoted by the same reference numerals, and detailed description thereof is omitted. The camera 2 shown in FIG. 15 differs from the solid-state imaging device 1 according to Embodiment 1 in the configuration of the feature detection unit 24 and the speed detection unit 28. On the other hand, the configuration of the solid-state imaging element 10 is the same as that in the first embodiment.

The feature detection unit 14 receives speed information from the speed detection unit 28 as a parameter in addition to the image data from the image processing unit 16.

For example, when the camera 2 is mounted on an automobile, the speed detection unit 28 is a speedometer or the like, and outputs a control signal indicating speed information to the feature detection unit 24.

FIG. 16 is a diagram schematically illustrating an example of an image captured by the solid-state imaging device 10 according to the first embodiment. In FIG. 16, for example, when the solid-state imaging element 10 is installed in front of the traveling direction of the automobile and is photographed, a photographed image is schematically illustrated.

As shown in FIG. 16, when the solid-state imaging device 10 is traveling straight forward, in the captured image, a point where the scenery hardly moves near the center of the screen, which is a point of infinity in the traveling direction based on perspective (disappearing point). 1601). And it seems that as the distance from the vanishing point approaches the edge of the screen, the movement increases, and as the distance increases, the speed increases.

For example, the distance L from the vanishing point coordinate P (Xp, Yp) is calculated from the vanishing point of the target coordinate (X, Y) concentrically using (Equation 10), for example, Let distance T from P be a threshold value. In this case, as shown in (Equation 11), the compression rate is reduced to a region of distance L <T from the vanishing point P (referred to as a region of interest 1602), and in a region separated from the radius T (peripheral region 1603). Control such as increasing the compression rate.

Distance L = √ (X−Xp) ^ 2 + (Y−Yp) ^ 2 (Equation 10)
Compression rate = low compression (L ≦ T), high compression (L> T) (Equation 11)

In addition, although the example which calculated the distance from the vanishing point P based on the perspective method was taken in the above, it is not restricted to it. For example, the higher the speed, the narrower the human field of view. Therefore, attention may be paid to the vicinity of the center, and it may be used that the edge is removed from the object of interest by illusioning that the whole is moving particularly quickly. In that case, the area and the compression ratio of the area may be changed using the speed value as a threshold value according to the speed information from the speed detection unit 28.

In addition, since the area close to the edge of the screen is easily deviated from attention (target object) as described above, an external input unit may be provided in the camera 2 and the area may be designated in advance by manual input or the like. In that case, the feature detection unit 24 determines whether or not it is a selection region by the external input unit. For example, taking FIG. 16 as an example, the feature detection unit 24 sets an area that is not the external input designation area (that is, the peripheral area 1603) as a detection target (that is, the attention area 1602), and the speed only in the attention area 1602 that is the detection target. Feature detection may be performed by detecting the speed by the detection unit 28.

As described above, the feature detection unit 24 uses the difference information between the image data composed of the plurality of pixel data processed by the image processing unit 16 and the image data processed by the image processing unit 16 in the past to calculate the image The overall motion is detected as a feature, and the compression rate setting unit 13 variably sets the compression rate for each region based on the detected motion information of the entire image. Specifically, the feature detection unit 24 adds speed information as a parameter to be given to the compression ratio setting unit 13 by using the speed parameter input by the speed detection unit 28, and from the vanishing point as the speed increases. A parameter indicating that the threshold value T of the radius is changed in a smaller direction can be given to the compression rate setting unit 13. As a result, the compression efficiency is improved because the peripheral area 1603 for increasing the compression ratio is widened, and the data of the non-target area is reduced by changing the compression ratio of the non-target area in a larger direction, and the compression efficiency is improved. The effect of becoming.

In addition, in FIG. 16, although the case where the solid-state image sensor 10 of the camera 2 was imaging with respect to the forward traveling direction was taken as an example, it is not limited thereto. When the image is taken in a direction transverse to the traveling direction, that is, in a direction in which no vanishing point exists, the entire screen moves in the same direction. Therefore, the feature detection unit 24 controls the entire screen uniformly. May be. Further, the camera 2 may be further detected by providing a configuration such as tracking a specific object.

(Example 2)
In the second embodiment, as a modification of the first embodiment, a case will be described in which the entire motion is detected using a motion vector based on a difference between frames instead of the speed detection unit 28.

FIG. 17A is a block diagram illustrating a configuration of the camera 3 according to the second embodiment. Elements similar to those in FIGS. 1 and 15 are denoted by the same reference numerals, and detailed description thereof is omitted. The camera 3 shown in FIG. 17A differs from the solid-state imaging device 1 according to Embodiment 1 in the configuration of a feature detection unit 34 and a moving image encoding unit 39. The camera 3 illustrated in FIG. 17A is different from the camera 2 according to the first embodiment in that the configuration of the feature detection unit 34 is different from the configuration of the feature detection unit 34 in that a moving image encoding unit 39 is not provided.

The moving image encoding unit 39 receives the luminance Y and color difference C information (also referred to as YC data) output from the image processing unit 16. The moving image encoding unit 39 performs encoding based on the input YC data, and outputs motion vector information 1908 generated during encoding to the feature detection unit 34.

FIG. 17B is an example of a motion vector output to the feature detection unit 34. The arrow in FIG. 17B schematically shows a motion vector in a state where the front in the traveling direction of the first embodiment described above is shown, and is represented by two-dimensional coordinates in the horizontal direction and the vertical direction. Here, the partition on the grid in the screen indicates a macro block which is a coding unit of the moving picture coding unit 39. In addition, the arrow increases outward from the vanishing point 1909, indicating that the absolute value of the motion vector of the macroblock is increased.

The moving image encoding unit 39 can hold at least one motion vector for each macroblock.

Next, a detailed configuration example of the moving image encoding unit 39 will be described. FIG. 18 is a block diagram illustrating a detailed configuration of the moving image encoding unit 39 according to the second embodiment. In FIG. 18, for example, the encoding format is H.264. A configuration example in the case of using H.264 is shown.

18, YC data 1901 is input by the image processing unit 16. The moving image encoding unit 39 includes a subtractor 3902, a DCT quantization unit 3903, an inverse quantization / inverse DCT unit 3904, an intra prediction unit 3905, a deblocking filter 3906, an external memory 3907, a motion detection unit 3908, and a motion compensation unit. 3909, an inter prediction unit 3910, a prediction determination unit 3911, a variable length coding unit 3912, and an adder 3913.

Here, the processing of the moving image encoding unit 39 will be described. First, the moving image encoding unit 39 performs processing by dividing one frame into macroblocks each having 16 horizontal pixels × 16 vertical pixels.

The YC data 3901 is YC data input from the image processing unit 16 and is input to the motion detection unit 3908 and the subtracter 3902 in units of macroblocks.

The input YC data 3901 is subtracted by the subtractor 3902 and the previous determination result is subtracted and input to the DCT quantization unit 3903. Then, DCT quantization section 3903 performs DCT and quantization, and inputs the result to inverse quantization / inverse DCT section 3904, intra prediction section 3905, and variable length coding section 3912.

Then, the intra prediction unit 3905 performs processing only on the image in the frame to calculate prediction data, and sends the calculated prediction data to the prediction determination unit 3911. The prediction determination unit 3911 determines the prediction values of the intra prediction unit 3905 and the inter prediction unit 3910 based on a predetermined determination criterion, and outputs the prediction values selected according to the determination result to the subtracter 3902 and the adder 3913.

In addition, the inverse quantization / inverse DCT unit 3904 restores the data after the DCT quantization by the DCT quantization unit 3903, and then the adder 3913 performs addition with the determination value output from the prediction determination unit 3911. And output to the deblocking filter 3906. Then, after filtering to reduce block noise generated between macroblocks by the deblocking filter 3906, the reference image is stored in the external memory 3907, and the reference image stored in the external memory 3907 is detected as motion between frames. The motion vector information 1908 is calculated from the difference information from the input data (YC data 3901) and is output to the motion compensation unit 3909 and the feature detection unit 34.

The motion compensation unit 3909 performs motion compensation using the output motion vector and the like, and outputs the motion compensation to the inter prediction unit 3910. The inter prediction unit 3910 uses the reference image data of a plurality of frames between frames to predict the pixel prediction value. Is calculated and used by the prediction determination unit 3911.

In the variable length encoding unit 3912, after the code amount control is performed by the DCT quantization unit 3903, variable length encoding such as CAVLC or CABAC is performed, and code data is output. Here, there is no reference image in the intra frame. On the other hand, in the inter frame, in the variable length encoding unit 3912, the motion detection unit 3908, the motion compensation unit 3909, and the inter prediction unit 3910 perform processing after the data of at least one frame before is prepared.

As described above, the camera 3 further includes the moving image encoding unit 39 that encodes the image data processed by the image processing unit 16 as compared with the solid-state imaging device 1 of the first embodiment, and includes the feature detection unit. 34 detects, as a feature, motion vector information for each region of the image from the pixel data processed by the image processing unit 16 based on the motion vector information output from the moving image encoding unit 39, and a compression rate setting unit 13 variably sets the compression rate for each region based on the detected motion information.

In this embodiment, H. Although an example in the case of H.264 has been described, the present invention is not limited thereto. Needless to say, the present invention can be applied to, for example, an encoding method using a motion vector between frames such as MPEG.

Whether or not the entire screen is moving is determined as follows based on the motion vector information 1908 output by the moving image encoding unit 39. For example, regarding the horizontal component vertical component of each motion vector MV (x, y) = (X, Y) (x, y are horizontal and vertical macroblock coordinates, XY is a horizontal and vertical motion vector value) output in units of macroblocks, After calculating the average value, the slope is calculated. Here, for example, when divided into W macro blocks and H macro blocks, W × H average values (Xave, Yave) are calculated for horizontal and vertical components. Then, it is determined whether or not the gradient of the motion vector MV (x, y) of each macroblock is in the same direction with respect to the average gradient (Xave, Yave), and the macroblock indicating the same direction as the average value Is uniformly distributed over the entire image W × H, it can be determined that the entire screen is moving.

(Example 3)
In this example, another application example of the solid-state imaging device 1 according to Embodiment 1 will be described. Specifically, as an example of a method for the feature detection unit 14 to perform feature detection, a method for detecting a human face and selecting a region including the detected human face will be described.

Specifically, the feature detection unit 14 gives (outputs) to the compression rate setting unit 13 a parameter that changes the compression rate depending on whether or not a face region is included.

Here, since the face of a person is information that is easily noticed in the video, it is desirable that image processing or recording be performed while maintaining high image quality. For example, feature detection based on skin color information, information such as the approximate shape of each face component such as eyes, nose, mouth, ear, and hair, information on the positional relationship between components, and face detection condition information such as contour information The unit 14 is caused to detect a human face. That is, the feature detection unit 14 is caused to detect face detection condition information and determine whether or not a face or a region including a face exists.

FIG. 19A and FIG. 19B are schematic views of face areas when applied to face detection in the third embodiment. FIG. 19A shows a case where one person's face is shown in the image, and FIG. 19B shows a case where two people's faces are shown.

When the feature detection unit 14 determines that a face exists, the feature detection unit 14 increases the number of regions n for each additional person.

For example, when the detected face is one person as shown in FIG. 19A, the feature detection unit 14 detects one area, and at the same time, uses the representative point C (not shown) of the face as a central starting point. Scan to the end of the area to calculate the face area. Here, the area shape of the detected area may be a rectangular area drawn with (minimum value in the horizontal direction, minimum value in the vertical direction) and (maximum value in the horizontal direction, maximum value in the vertical direction) The rectangular area is not limited, and as shown in FIG. 8B of the first embodiment, each area information may be given to each line by giving a horizontal left end and a width to an arbitrary shape. As described in the first embodiment, the feature detection unit 14 outputs region information including the face region detected as described above to the compression rate setting unit 13. Then, as shown in FIG. 19A, the compression rate setting unit 13 improves the image quality by setting the compression rate to A and lowering the compression rate for the face region, thereby improving the image quality, and for the region other than the face. As the compression rate B, the compression rate is increased to achieve high compression. In this way, the amount of data can be efficiently reduced while maintaining the image quality.

Further, for example, when two or more faces are shown as shown in FIG. 19B, the feature detection unit 14 detects two areas of the first face area and the second face area. The feature detection unit 14 may set individual compression ratios in the two areas after calculating the two areas. In addition, when the two regions overlap or when the distance between the region boundaries is smaller than the predetermined value Th, the feature detection unit 14 combines the plurality of regions as one region as shown in FIG. 19B. It may be reset as one area (first face area after combination). The feature detection unit 14 may output region information including the face region detected in this way to the compression rate setting unit 13.

As described above, an image composed of a plurality of pixels has a face area that is an area showing a human face. The feature detection unit 14 detects region information indicating a face region in the image as a feature from the pixel data output from the photoelectric conversion unit 11. Based on the region information detected by the feature detection unit 14, the compression rate setting unit 13 variably sets the compression rate individually for at least a region including a face region and a region not including a face region.

Example 4
In the present embodiment, a procedure in which the compression unit 12 converts the frame rate based on the compression rate set for each region by the compression rate setting unit 13 (for example, from the amount of imaging data to be reduced) will be described.

FIG. 20 is a diagram for explaining a procedure in which the compression unit 12 according to the fourth embodiment converts the frame rate.

In the first embodiment, the case of a MOS sensor (MOS type solid-state image pickup device) has been described using FIG. 2 and FIG. 3 as the solid-state image pickup device 10. The solid-state imaging device 10 is exposed after reading is completed. In order to keep the exposure time of each line the same, the period of the synchronization signal is reduced while keeping the interval of the synchronization signal of each line equal. is required. And that is done as follows.

In the following, it is assumed that the feature detection unit 14 divides the image into a plurality of areas. In other words, in each frame l in a frame composed of one frame L line, the number of divided areas is N, the number of pixels in each area n is P (n), and the compression rate is C (n). In this case, the total data amount SIZE (l) per line l can be expressed as (Equation 12).

SOL byte count + Σ (SOR byte count + P (n) * C (n)) (n = 1, 2,..., N) + EOL byte count (Formula 12)

Further, in order to eliminate the variation in the data amount between the lines, the value of SIZE (l) is calculated for the lines l = 1, 2,..., L and all the lines, and the maximum value is calculated. .

MAX (SIZE (1), SIZE (2), ..., SIZE (L)) ... (Formula 13)

In the above (Equation 13), it is determined whether or not the maximum value MAX (1 to L) in the L line is less than the total amount of data per line at the time of non-compression. The frame rate can be improved according to the amount.

For example, in the one line period shown in FIG. 20A based on the cycle of one line unit, for example, the data size after compression per line in FIG. When the size is 80%, the interval of one line of the synchronization signal can be reduced as shown in FIG. 20 (a) to FIG. 20 (b). Thus, the period of one line can be shortened to 80%, and the speed can be improved by 25% in calculation. Here, since the line maximum value in the frame is calculated, if the period of the synchronization signal is set to a value larger than the above value, the data output of one line does not straddle the next line, and all lines are output. Data can be output and an equally spaced period can be maintained.

As described above, the solid-state imaging device 1 outputs the pixel data compressed by the compression unit 12, but the compression unit 12 has a data amount after compression and a data amount of pixel data before compression by the compression unit 12. The frame rate can be changed by changing the cycle of one line based on the relationship with the data amount before compression. Here, the data amount after compression is the data amount of compressed pixel data, and the compression rate and the number of pixels for each region of the image per line of the compressed pixel data output from the solid-state imaging device 1. Is calculated based on the sum of

(Example 5)
In this example, another application example of the solid-state imaging device 1 according to Embodiment 1 will be described. Specifically, a method in which the feature detection unit 14 controls the compression rate of pixel data using detection information for AF control will be described as another example of the feature detection method.

Here, the AF control includes an active method using an external sensor, a passive method using an image sensor, a passive method, a phase difference detection method, a contrast detection method, and the like. Hereinafter, the contrast detection method will be described as an example.

FIG. 21 is a block diagram illustrating a configuration of the solid-state imaging device 4 according to the fifth embodiment.

The image processing unit 16 includes a preprocessing circuit 461 and a YC processing circuit 462 inside. Further, the feature detection unit 14 includes an HPF circuit 443, an area-by-area contrast integration circuit 444, and a central processing circuit (hereinafter referred to as CPU) 445.

The pixel data output from the decompression unit 15 is detected by the pre-process circuit 461 from the imaging data in the light-shielded area, and subtracts and corrects OB (Optical Black) clamp processing, which corrects flaws in the pixel data due to deterioration, etc. Preprocessing such as processing is performed, and the YC processing circuit 462 performs synchronization processing, edge enhancement processing, and gamma conversion processing. At the same time, the output of the preprocess circuit 461 is input to an HPF circuit 443 formed of a digital high-pass filter, and a spatial high-frequency component signal of the subject is extracted and input to a contrast integration circuit 444 for each region. The area-by-area contrast integration circuit 444 integrates the output signals from the HPF circuit 443 for each divided area (hereinafter referred to as AF area) of the imaging screen designated in advance in one frame for each imaging frame, and subjects for each AF area. A contrast value is generated and output to the CPU 445.

Here, the AF area is, for example, an 81 area obtained by dividing the imaging screen (in the image) into 9 parts vertically and horizontally as shown in FIG. In FIG. 22, a to i are horizontal area addresses, and 1 to 9 are vertical area addresses. For example, the upper left AF area is the address a1, and the lower right AF area is the address i9. Call it. The contrast value of each AF area is called AFC (afp) (afp is a1 to i9).

On the other hand, a subject image is formed by the optical lens 410 on the pixel cell array 111 in the photoelectric conversion unit 11 of the solid-state imaging device 10, and the optical lens 410 includes a lens driving unit including an actuator such as a stepping motor. The lens 411 is driven and controlled in the direction of the lens optical axis so that the subject-side focus position formed on the pixel cell array 111 can be adjusted. The lens driving unit 411 is driven and controlled by the CPU 445 for the focus position of the optical lens 410.

Further, the CPU 445 compresses the compression rate setting unit 13 for the compression rate setting unit 13 corresponding to each AF area for each frame, that is, the above-described region P (N) described with reference to FIGS. 8A and 8B. Set the rate Q (N). Here, N is 1 to 81. For example, the CPU 445 determines that the AF area a1 corresponds to P (1), b1 corresponds to P (2),..., I9 corresponds to P (81), and the horizontal pixel count W (N) and vertical pixel count H ( N) are set for the compression rate setting unit 13 together.

As described above, the solid-state imaging device 1 further includes the lens driving unit 411 that drives the optical lens 410, and the feature detection unit 14 uses the pixel data output from the photoelectric conversion unit 11 to contrast the predetermined region of the image. Information indicating that the predetermined area is an in-focus area by detecting whether or not the predetermined area is in focus based on the contrast value and a contrast integration circuit for each area 444 for detecting the value And a CPU 445 for controlling the lens driving unit 411 so that the predetermined area is in focus. Based on this information, the compression rate setting unit 13 variably sets the compression rate individually for a region including at least the in-focus region and the out-of-focus region.

Next, the operation of the CPU 445 will be described using the flowchart of FIG.

First, an AF operation is started by inputting a trigger for instructing the start of autofocus (hereinafter referred to as AF) from a user (not shown).

Next, in step S601, the CPU 445 controls the lens driving unit 411 so that the lens position is infinitely focused.

Next, in step S602, the CPU 445 resets the accumulated contrast value of each area to the contrast accumulation circuit 444 for each area.

Next, in step S603, the lens driving unit 411 is controlled so as to move the lens position of the optical lens 410 to the closest focus position side by a predetermined amount.

Next, in step S604, one frame output from the solid-state imaging device 10 after driving the lens in step S603 and acquired by the contrast integration circuit for each region via the extension unit 15, the preprocess circuit 461, and the HPF circuit 443. The contrast values of all the AF areas are acquired.

Next, in step S605, it is determined whether or not the lens position of the optical lens 410 is the close end. If it is not the close end (NO in step S605), the process proceeds to step S603. On the other hand, if it is the close end (YES in step S605), the process proceeds to step S606.

In this way, a profile of the contrast value with respect to the lens position of the contrast value AF (afp) of the entire AF region between the lens position from the infinite end to the closest end is acquired.

Next, in step S606, based on the contrast value profile obtained by the above operation, a profile and an AF region having a contrast value peak closest to the closest end are selected and become the peak of the profile. The lens position is calculated as the focusing lens position.

For example, FIG. 24 shows an example of a contrast value profile. However, when there is a person in the foreground and a distant view behind the person as in the subject pattern in FIG. 22, the person indicated by the dotted line in FIG. A profile 2601 is acquired in the hit AF area, and a profile 2602 is acquired in the other AF areas. In this case, in step S606, the lens position fp1 that is the peak of the profile 2601 in FIG. 24 is determined as the focus lens position.

Next, in step S607, the lens driving unit 411 is controlled to drive the optical lens 410 to the in-focus lens position determined in step S606. As a result, the AF operation is completed.

Next, in step S608, the compression ratio Q is set to low compression for the AF area (AF area indicated by the dotted line in FIG. 22) selected by the determination of the focusing lens position, that is, the focused AF area. A compression rate Q is set in the compression rate setting unit 13 so as to achieve high compression for an out-of-focus AF area that is not selected in determining the focus lens position.

As described above, since the compression ratio is set so that the pixel data is compressed with low compression for the AF area including the focused subject, the deterioration of the focused subject portion due to compression distortion is avoided. While achieving high image quality, pixel data is compressed in the solid-state image sensor so as to be highly compressed in areas that are not in focus, so the amount of compressed pixel data output from the solid-state image sensor can be reduced. This can reduce the frame rate and power consumption.

Further, by further applying this embodiment, when there are a plurality of focused AF areas, a process of setting the compression rate of the pixel data to a lower compression is performed in an AF area having a higher contrast value. It is also possible to easily improve the image quality by adaptively changing the compression ratio according to the subject contrast and the spatial frequency among the existing subjects.

(Example 6)
In this example, a case where the solid-state imaging device 1 according to Embodiment 1 is applied to, for example, a digital camera will be described.

FIG. 25 is a block diagram illustrating a configuration of the digital camera 5 according to the sixth embodiment. Elements similar to those in FIGS. 1 and 17A are denoted by the same reference numerals, and detailed description thereof is omitted.

A digital camera 5 shown in FIG. 25 includes a solid-state imaging device 10, an extension unit 15, an image processing unit 16, a feature detection unit 14, a moving image encoding unit 39, a recording unit 508, a recording memory 509, and a display. A unit 510, a display device 511, a CPU 512, a program memory 513, and an external memory 514 are provided. An external switch 515 is input to the digital camera 5. Note that the configurations of the solid-state imaging device 10, the expansion unit 15, the image processing unit 16, and the feature detection unit 14 are the same as those described in FIG. The differences will be described below.

The CPU 512 reads the program from the program memory 513 of the power source, determines a mode designated in advance by the input of the external switch 515, for example, a mode such as a recording mode or a reproduction mode, and starts the system in a predetermined mode.

Note that the operation described here is controlled by the CPU 512.

For example, in the recording mode, the decompression unit 15 decompresses the compressed image data input by the compression unit 12 from the solid-state image sensor 10, and the image processing unit 16 generates YC data. The moving image encoding unit 39 encodes a still image / moving image with respect to the YC image data obtained by the image processing unit 16 to generate image data for recording. Here, when recording a still image, encoding is performed in, for example, the JPEG format. Encoding is performed in a format such as H.264 or MPEG.

Further, the recording unit 508 receives the code data of the moving image encoding unit 39, performs processing such as header generation for recording in the recording memory 509 and area management of the recording medium, and records the data in the recording memory 509. . The display unit 510 receives processing data of the image processing unit 16 and performs processing for displaying on the display device 511. That is, the image data is converted into an image size and an image format corresponding to the display device 511, and data such as OSD (On Screen Display) is added and transferred to the display device 511.

The display device 511 includes a liquid crystal (LCD) display, for example, and outputs an input signal to display an image. Further, feature information is input from the feature detection unit 14 to the CPU 512, and the feature amount can be set in the compression rate setting unit 13 using various feature amounts obtained as described in the first embodiment. It is. Note that the feature detection performed by the feature detection unit 14 may be performed by the CPU 512 instead of the feature detection unit 14.

On the other hand, in the playback mode, for example, the code data recorded from the recording memory 509 is read out and input to the moving image encoding unit 39, and the input code data is decoded into YC data. The decrypted YC data is temporarily stored in the external memory 514, and then read out from the external memory 514 and processed by the display unit 510, so that it is sent to the display device 511 to perform a display operation.

In the digital camera 5, the image data output from the solid-state image sensor 10 may be stored and used in the external memory 514. In this case, the CPU 512 has a work memory for the external memory 514. The data after processing of the image processing unit 16 is stored, the display unit 510 reads out and uses it for display, the output code data of the moving image encoding unit 39 is temporarily stored, and the recording unit 508 reads out the data. Or use it. Since there are many accesses in this way, if the compressed data output from the solid-state imaging device 10 is first stored in the external memory 514 and then read out and input to the decompression unit 15 to perform processing, the memory access amount for the imaging data can be reduced. It becomes possible.

As described above, the digital camera 5 includes the solid-state imaging device 1 according to Embodiment 1, the moving image encoding unit 39 that encodes the pixel data processed by the image processing unit 16, and the moving image encoding unit 39. A recording unit 508 that records the pixel data encoded by the image processing unit 509 in the recording memory 509, a display unit 510 that displays the image data processed by the image processing unit 16, a program memory 513 that stores a program, and a program memory The CPU 512 performs system control of the digital camera 5 based on a program read from the program 513. The CPU 512 performs system control of a recording operation or a reproduction operation based on a setting given from the outside.

As described above, it is possible to set the compression rate for each area extracted by the feature detection unit 14 even in the recording / reproduction of the digital camera 5, and recording can be performed by distributing the optimum compression rate.

(Embodiment 2)
Hereinafter, in the second embodiment, a case where the configuration is different from that of the solid-state imaging device 1 in the first embodiment will be described.

FIG. 26 is a block diagram showing a configuration of the solid-state imaging device according to Embodiment 2 of the present invention. Elements similar to those in FIG. 1 are denoted by the same reference numerals, and detailed description thereof is omitted. The solid-state imaging device 6 shown in FIG. 25 differs from the solid-state imaging device 1 according to Embodiment 1 in that the feature detection unit 54 is included in the solid-state imaging device 60.

Specifically, the pixel data output from the photoelectric conversion unit 11 is input to the compression unit 12 and also input to the feature detection unit 64 built in the solid-state imaging device 60, and is horizontally output by the feature detection unit 54. Feature detection is performed for each line, and the result is output to the compression ratio setting unit 13. Note that the operation of the compression rate setting unit 13, the operation of the compression unit 12, and the mutual relationship are the same as those in the first embodiment, and thus the description thereof is omitted.

The feature detection unit 64 selects a compression rate according to the signal level of the pixel data of each pixel of one horizontal line input from the photoelectric conversion unit 11 in synchronization with a vertical and horizontal synchronization signal (not shown), and performs low compression. The pixel area within the horizontal line is output to the compression ratio setting unit 13 with the pixel as the attention area pixel.

Here, FIG. 27 is a diagram showing the relationship between the pixel data level of each pixel of one horizontal line selected in the feature detection unit 64 in the second embodiment and the selected compression rate.

The feature detection unit 64 has two threshold values set in advance with respect to the pixel data level, and the pixel P1, P2,... Pn of one horizontal line input from the photoelectric conversion unit 11 A level comparison having a threshold value and hysteresis is performed to select a compression rate. Specifically, the pixels after the level change from a pixel level lower than threshold 1 to a higher pixel level are identified as pixels whose compression rate is high compression, and more than threshold 2 set to a level lower than threshold 1 The area information and the compression rate are set so that the pixel after the pixel whose level has changed from the high pixel level to the low pixel level is low-compressed, that is, the low-compression compression rate that is recognized as a target pixel and stored in advance is set. Output to the setting unit 13. As a result, a high compression ratio is set at a high pixel level, that is, a high luminance portion of the subject, and a low compression ratio is set at medium luminance and low luminance, and pixel data is compressed.

In this way, pixel data that is insensitive to fine luminance changes in the human eye and that is positively compressed in a high-luminance part of the subject that is level-compressed by a non-linear gamma conversion process performed by the subsequent image processing unit 16. Perform the compression process. As a result, while avoiding deterioration due to compression distortion for medium- and low-brightness subjects where image quality degradation is easily recognized, high image quality is achieved while aggressively increasing for high-brightness subjects where degradation is difficult to recognize. Since the pixel data is compressed in the solid-state imaging device so as to be compressed, the data amount of the compressed pixel data which is the output of the solid-state imaging device can be reduced, and a high frame rate or low power consumption can be achieved. Further, by adopting the configuration of the present embodiment, the pixel data compression process is completed within the solid-state imaging device, so that the size of the apparatus can be reduced by reducing the number of input signal lines and terminals from outside the device, and in real time. In addition, since the compression rate is determined, the delay in reflecting the setting of the compression rate in the compression process can be minimized, so that in principle there is no delay in units of frames.

In the present embodiment, a black and white solid-state imaging device that does not implicitly have a color filter for each pixel is described, but the present invention is not limited to this. For example, a single-plate color solid-state imaging device having a Bayer array color-on-chip filter may be used, and in that case, the feature detection unit 64 is configured to have hysteresis due to a level change between adjacent pixels having the same color phase. It is inevitable.

As described above, according to the present invention, it is possible to realize an imaging apparatus capable of reducing the data amount per unit time by a certain amount regardless of shooting conditions.

Specifically, the imaging device of the present invention includes a solid-state imaging device 10, a feature detection unit 14, an expansion unit 15, and an image processing unit 16 outside the solid-state imaging device 10, and the solid-state imaging device 10 includes a photoelectric conversion unit. 11, a compression unit 12 and a compression rate setting unit 13. The pixel data of the photoelectric conversion unit 11 is subjected to irreversible compression processing by the compression unit 12 and output to the outside. Pixel data output from the solid-state imaging device 10 and decoded into pixel data by the decompression unit 15 is subjected to image processing by the image processing unit 16 and input to the feature detection unit 14 as a frame image. The feature detection unit 14 extracts the feature of the image in the frame, generates optimum compression rate information in the individual area, and gives information about the feature to the compression rate setting unit 13. The compression rate setting unit 13 sets the compression rate information from the information regarding the feature. Based on the compression rate information, the compression unit 12 performs the lossy compression operation by adaptively changing the compression rate within the frame.

With such a configuration, the pixel data can be subjected to compression processing in accordance with subject conditions and shooting conditions, and thus output from the solid-state imaging device 10 while the quantization error of the compressed and quantized pixel data is in an optimum state. The amount of data per unit time can always be reduced above a certain level.

Therefore, it is possible to output a high-quality and high-speed video signal, and an imaging apparatus can be realized without impairing image quality performance. Therefore, for example, it is possible to suppress an increase in the circuit scale of the camera system including the imaging device such as an image sensor and to reduce power consumption.

As mentioned above, although the imaging device and the digital camera of the present invention have been described based on the embodiment, the present invention is not limited to this embodiment. Unless it deviates from the meaning of this invention, the form which carried out the various deformation | transformation which those skilled in the art can think to this embodiment, and the structure constructed | assembled combining the component in different embodiment is also contained in the scope of the present invention. .

INDUSTRIAL APPLICABILITY The present invention can be used for an imaging apparatus, and more particularly to an imaging apparatus that is a device that electronically generates and displays or records a moving image or a still image such as a digital still camera, a digital video camera, a surveillance camera, or a drive recorder camera. Can be used.

DESCRIPTION OF SYMBOLS 1, 4, 6 Solid- state imaging device 2, 3 Camera 5 Digital camera 10, 60 Solid-state image sensor 11 Photoelectric conversion part 12 Compression part 13 Compression rate setting part 14, 24, 34, 54, 64 Feature detection part 15 Expansion part 16 Image Processing unit 28 Speed detection unit 39 Video encoding unit 111 Pixel cell array 112 Timing generator 113 AD conversion unit 114 Horizontal scan selector 121 Target pixel value input unit 122 Predicted pixel generation unit 123 Code conversion unit 124 Change extraction unit 125, 153 Quantum Quantization width determination unit 126 Quantization processing unit 127 Packing unit 151 Unpacking unit 152 Predicted pixel generation unit 154 Inverse quantization processing unit 155 Code generation unit 156 Inverse code conversion unit 157 Output unit 171 Serial conversion 172 Output control unit 410 Optical lens 411 Lens drive unit 443 HPF circuit 444 Contrast integration circuit for each region 445, 512 CPU
461 Preprocess circuit 462 YC processing circuit 508 Recording unit 509 Recording memory 510 Display unit 511 Display device 513 Program memory 514, 3907 External memory 515 External switch 1101 Photodiode 1102 Read transistor 1103 Floating diffusion 1104 Reset transistor 1105 Amplifier 1601 Vanishing point 1602 Area 1603 Peripheral area 1901, 3901 YC data 1908 Motion vector information 1909 Vanishing point 2601, 2602 Profile 3902 Subtractor 3903 DCT quantization section 3904 Inverse quantization / inverse DCT section 3905 Intra prediction section 3906 Deblocking filter 3908 Motion detection section 3909 Motion Compensator 3910 Inter Predictor 3911 Prediction Part 3912 variable-length encoder 3913 adders L1 ~ L8 common signal read lines S11 ~ S18 selection signal read lines S41 ~ S48 switches P11 ~ P88 pixel cells

Claims (12)

1. A photoelectric conversion unit having a plurality of pixels arranged in a two-dimensional shape for converting incident light into an electrical signal, and outputting the plurality of electrical signals converted by the plurality of pixels as pixel data;
   Based on the pixel data output from the photoelectric conversion unit, a detection unit that detects features for each region of the image,
   A compression rate setting unit that sets a compression rate for each region based on the features detected by the detection unit;
   An image pickup apparatus comprising: a compression unit that performs irreversible compression on pixel data output from the photoelectric conversion unit according to a compression rate set by the compression rate setting unit.
2. The imaging device further includes:
   A decompression unit for decompressing the pixel data compressed by the compression unit;
   An image processing unit that processes the pixel data expanded by the expansion unit,
   The imaging device according to claim 1, wherein the detection unit detects a feature for each region of the image from the pixel data processed by the image processing unit.
3. The imaging device according to claim 1, wherein the photoelectric conversion unit constitutes a MOS solid-state imaging device.
4. The detector is
   From the difference information between the image data composed of a plurality of pixel data processed by the image processing unit and the image data processed by the image processing unit in the past, the movement of the entire image is detected as the feature,
   The imaging apparatus according to any one of claims 1 to 3, wherein the compression rate setting unit variably sets a compression rate for each of the regions based on the detected motion information of the entire image.
5. The imaging device further includes:
   An encoding unit for encoding the image data processed by the image processing unit;
   The detector is
   Based on the motion vector information output by the encoding unit, motion vector information for each region of the image is detected as the feature from the pixel data processed by the image processing unit,
   The imaging apparatus according to any one of claims 1 to 3, wherein the compression rate setting unit variably sets a compression rate for each region based on detected motion information.
6. The image has a face area that is an area showing a human face,
   The detection unit detects region information indicating a face region in the image as the feature based on pixel data output from the photoelectric conversion unit,
   The compression rate setting unit variably sets the compression rate individually for at least a region including a face region and a region not including a face region based on the region information detected by the detection unit. The imaging device according to any one of the above.
7. The compression unit is
   The pixel data is further compressed based on compression rate information set by the compression rate setting unit by adding identification information including at least compression rate information and data amount information. The imaging apparatus of Claim 1.
8. The imaging apparatus further includes a lens driving unit that drives the lens,
   The detector is
   A contrast detection unit that detects a contrast value for a predetermined region of the image based on pixel data output from the photoelectric conversion unit;
   By determining whether or not the predetermined area is in focus based on the contrast value, information indicating that the predetermined area is the in-focus area is detected, and the predetermined area is in focus A CPU (Central Processing Unit) for controlling the lens driving unit,
   The imaging apparatus according to any one of claims 1 to 4, wherein the compression rate setting unit variably sets a compression rate individually for a region including at least a focused region and a non-focused region based on the information.
9. The detection unit detects, from the pixel data expanded by the expansion unit, information on a magnitude relationship with respect to a predetermined threshold in a pixel value for each area of the image as the feature,
   The imaging apparatus according to any one of claims 1 to 4, wherein the compression rate setting unit variably sets a compression rate based on the magnitude relationship information detected by the detection unit.
10. The detection unit detects information on a magnitude relationship with respect to a predetermined value in the pixel value for each region of the image from the pixel data processed by the image processing unit,
    The imaging apparatus according to any one of claims 1 to 4, wherein the compression rate setting unit variably sets a compression rate based on the magnitude relationship information detected by the detection unit.
11. The imaging device outputs pixel data compressed by the compression unit,
    The data amount of the compressed pixel data is calculated based on the compression rate and the total number of pixels for each area of the image around one line of the compressed pixel data output from the imaging device,
    The compression unit further changes the cycle of one line based on the relationship between the data amount of the compressed pixel data and the data amount of the pixel data before the compression unit compresses the frame. The imaging apparatus according to any one of claims 1 to 4, wherein the rate is changed.
12. A digital camera,
    An imaging device according to any one of claims 2 to 9,
    An encoding unit for encoding pixel data processed by the image processing unit;
    A recording unit for recording the pixel data encoded by the encoding unit in a recording memory;
    A display unit for displaying the image data processed by the image processing unit;
    A program memory storing the program;
    A CPU for performing system control of the digital camera based on a program read from the program memory,
    The CPU is a digital camera that performs system control of a recording operation or a reproducing operation based on a setting given from the outside.

Images