JP2005341561A - Image processing circuit - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processing circuit which can enhance the reliability of image processing by securely performing real-time image processing while performing pixel interpolation. <P>SOLUTION: The pixel data of a first field stored in an original image data buffer 26b is DMA (Direct Memory Access) transferred to an RPU (Real-time Processing Unit) 14. In parallel therewith, synchronized with the read out of the pixel data of the first field, the pixel data of a second field is read out from a CCD sensor 12 in conformity to the pixel transfer clock, and is input to the RPU 14. The pixel data of the first and the second fields are processed for pixel interpolation, and the two pixel data of the two lines are output toward postprocessing units 14c to 14f during one cycle (1/f1) of the pixel transfer clock. The postprocessing units 14c to 14f execute image processing such as gamma correction processing for each pixel during one cycle (1/f2) of the pixel processing clock. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to an image processing circuit for image data captured by a digital photographing apparatus such as a digital still camera.

  FIG. 26 is a schematic diagram showing a configuration of a general digital still camera 100. As shown in the figure, an original image signal captured by an image sensor 105 such as a mounted CCD sensor or CMOS sensor is A / D converted into a digital image signal and then taken into an image processing unit 106 to be subjected to pixel interpolation processing, color space. Various image processing such as conversion processing and contour enhancement processing are performed. The image data subjected to the image processing is displayed on a finder such as a liquid crystal monitor 109. In general, the image sensor 105 reads an image signal from an odd field consisting of only odd lines and an even field consisting of only even lines constituting all pixels at different timings, and the image signal is read from each line. It is driven by any one of progressive methods for reading sequentially. In FIG. 26, reference numeral 101 denotes an optical lens, 102 denotes a color correction filter, 103 denotes an optical LPF (low-pass filter), 104 denotes a color filter array, 107 denotes a drive unit that drives and controls the image sensor 105, and 111 denotes an external device. Shows the external interface to connect.

  The operation of the digital still camera 100 is as follows. First, the liquid crystal monitor 109 receives an image signal obtained by thinning out a predetermined number of lines of all pixels of the image sensor 105 and slightly reducing the resolution after the image processing unit 106 performs image processing. Appears on the screen (viewfinder operation).

  When the photographer presses the shooting button while confirming the subject image on the liquid crystal monitor 109, pixel data is read from all lines of the image sensor 105 by the interlace method or the progressive method (all pixel reading operation). The pixel data is A / D converted and subjected to image processing by the image processing unit 106, JPEG compression, and the like, and then stored in a storage medium such as the built-in memory 108 or the memory card 110. Such an all-pixel reading operation will be described in detail with reference to FIG. 27. First field pixel data (CCD data) read out first from the image sensor 105 driven by the interlace method and second field read out next. The pixel data is A / D converted and then temporarily stored in the original image data buffer 108a provided in the built-in memory 108 (step 100). Next, a real-time processing unit (hereinafter referred to as RPU) 120 configured by hardware sequentially reads out pixel data of the first field and the second field stored in the original image data buffer 108a, and performs pixel interpolation processing, Image processing such as color space conversion processing and contour enhancement processing is executed in real-time processing, and the processed data is output to the processing data buffer 108b and stored (step 101). Next, the CPU (central control unit) 121 reads out the processing data from the processing data buffer 108b, performs JPEG (Joint Photographic Experts Group) compression or the like by software processing using the temporary storage data buffer 108c, and then the memory card or the like. Is stored in the storage medium 122 (step 102).

  When the image sensor 105 is driven in a progressive manner, the processing of steps 101 to 102 can be executed by directly outputting the CCD data to the RPU 120 without temporarily storing the CCD data in the original image data buffer 108a.

  However, conventionally, since the frequency of the drive clock of the image sensor 105 has been kept low to a constant value within a range not exceeding the maximum value of the drive clock frequency of the RPU 120, there has been a problem that the frame rate of the finder display is small. That is, since the low resolution CCD data thinned out from the image sensor 105 is output during the finder operation, the load on the RPU 120 is small, and high resolution CCD data is output from all the lines of the image sensor 105 during all pixel readout. Therefore, the load on the RPU 120 becomes approximately twice or more. However, since the drive clock frequency of the image sensor 105 is suppressed to a constant value within a range that does not exceed the maximum drive clock frequency allowed for the RPU 120 during both the all-pixel reading operation and the finder operation, The frame rate is reduced and the image quality of the video image displayed on the viewfinder is degraded.

  In the all-pixel reading operation, the transfer rate of pixel data from the image sensor is defined by the pixel transfer clock. When pixel interpolation processing is performed, pixel data for two pixels is obtained for each period of the pixel transfer clock. However, if the pixel transfer clock is used as it is for post-processing such as gamma correction processing, color space conversion processing, and contour enhancement processing, these post-processing must be performed on two pixels during one cycle of the pixel transfer clock. There's a problem.

  In view of the above problems, an object of the present invention is to provide an image processing circuit that can reliably execute image processing in real time to improve the reliability of image processing.

  In order to solve the above-described problem, an invention according to claim 1 is directed to an imaging sensor and a pixel transfer that executes a real-time pixel interpolation process on pixel data read from the imaging sensor and defines a transfer speed of the pixel data. An image processing unit including a pixel interpolation processing unit that outputs interpolation pixel data of a plurality of lines for each cycle of the clock, and a post-processing unit that further performs real-time image processing on the interpolation pixel data. The post-processing unit sets the frequency of the pixel processing clock that defines the processing speed of the pixel data to a value that is twice or more the frequency of the pixel transfer clock.

  According to a second aspect of the present invention, in the image processing circuit according to the first aspect, a plurality of lines of pixel data output from the pixel interpolation processing unit are multiplexed every one cycle of the pixel transfer clock to provide one line. And a buffer unit for storing pixel data output from the multiplexing unit, and the post-processing unit performs line sequential processing on the pixel data stored in the buffer unit. And executing image processing.

  The invention according to claim 3 is the image processing circuit according to claim 2, further comprising a DMA (direct memory access) controller for controlling direct transfer of pixel data from the pixel interpolation processing unit to the buffer unit. In addition.

  According to a fourth aspect of the present invention, there is provided the image processing circuit according to the first aspect, wherein the buffer unit stores a plurality of lines of pixel data output from the pixel interpolation processing unit for each period of the pixel transfer clock. And a DMA (Direct Memory Access) controller for controlling the channel so as to transfer the pixel data line-sequentially to the buffer unit.

  The invention according to claim 5 is the image processing circuit according to claim 1, wherein the image processing unit stores a plurality of lines of pixel data output from the pixel interpolation processing unit for each line. A second storage unit that stores pixel data output from the pixel interpolation processing unit for each line while the pixel data stored in the first storage unit by the post-processing unit is sequentially read out on a line basis. In addition, the first storage unit stores the pixel data output from the pixel interpolation processing unit for each line while the post-processing unit reads out the pixel data stored in the second storage unit sequentially in a line. Is.

  According to the image processing circuit of the first aspect, since the processing rate of the post-processing unit can be equal to or higher than the processing rate of the pixel interpolation processing unit, real-time image processing in the image processing unit is executed reliably and with high reliability. It becomes possible to do.

  According to the image processing circuit of the present invention, since the post-processing unit can sequentially process the pixel data rearranged in line order, the real-time image processing in the image processing unit can be reliably executed. It becomes possible.

  According to the image processing circuit of claim 3, the data transfer from the pixel interpolation processing unit to the buffer unit is executed efficiently and at high speed, so that the real-time image processing in the post-processing unit is reliably executed. It becomes possible to do.

  According to the image processing circuit of the fourth aspect, since the pixel data is stored in the pixel interpolation processing unit in line order by the channel control of the DMA controller, the post-processing unit reads out the pixel data and performs line sequential processing. Image processing can be executed.

  According to the image processing circuit of claim 5, the pixel data of the plurality of lines output from the pixel interpolation processing unit can be sequentially rearranged and output to the post-processing unit, and the post-processing unit can output the pixel data to the line Since processing can be performed sequentially, real-time image processing can be reliably executed.

  FIG. 1 is a schematic diagram showing an overall configuration of a digital still camera 1 according to an embodiment of the present invention. The digital still camera 1 includes an optical mechanism 11 having an AF (autofocus) function, an automatic aperture function, and the like, and a subject image is picked up by a CCD (charge coupled device) sensor 12 through the optical mechanism 11. At this time, if necessary, the light whose light amount is adjusted in synchronization with the photographing timing may be emitted from the strobe 30 to irradiate the subject. The captured original image data of the subject is taken into the analog signal processing circuit 13 and A / D converted into a digital image signal. The digital image signal is subjected to predetermined image processing such as pixel interpolation processing, color space conversion processing, contour correction processing, and filtering in real time processing (real time processing) in a real time processing unit (hereinafter abbreviated as RPU) 14. Applied. The image signal subjected to the image processing is displayed on the LCD 23 functioning as a viewfinder, or after being subjected to software processing such as JPEG compression processing in the CPU 17, transferred to the memory card 27 through the main bus 10 and stored, The data is output to an external device such as a personal computer through an interface (I / F) 28. The main memory 26 such as DRAM (Dynamic Random Access Memory), SDRAM (Synchronous DRAM), RDRAM (Rambus DRAM), etc. has a buffer area for temporarily storing data processed by the RPU 14 as will be described in detail later. In addition, a buffer area temporarily used when the CPU 17 executes software processing is provided.

  In FIG. 1, reference numeral 15 denotes a CCD drive circuit for driving the CCD sensor 12, 16 denotes a timing generator for regulating the operation timing of the RPU 14 and the CCD drive circuit 15, etc., 18 denotes a PLL oscillation circuit, and 19 denotes an auxiliary arithmetic unit for the CPU 17. Reference numeral 20 denotes a display module, 21 denotes a digital encoder 21, and 22 denotes an LCD driving circuit for driving the LCD 23. The clock generator 29 divides the clock signal supplied from the PLL transmission circuit 18 to generate drive clocks for all modules such as the RPU 14, the timing generator 16, the CPU 17, and the digital encoder 21.

  The main memory 26, the external interface 28, and the memory card 27 are connected to each other via the main bus 10 together with a DMA (direct memory access) controller 24 and a JPEG processing unit 25, and data transfer between these modules is performed. Can be directly executed through the main bus 10 without the CPU 17 under the control of the DMA controller 24. As a result, the load on the CPU 17 is reduced and the storage area of the main memory 26 can be used efficiently.

  When the viewfinder is operated, that is, when the subject image is displayed on the LCD (finder) 23, the CCD sensor 12 causes the CCD sensor 12 to output a plurality of lines of image signals obtained by thinning predetermined lines in accordance with the resolution of the viewfinder. It is driven as follows. The thinned CCD data is A / D converted by the analog signal processing circuit 13, subjected to image processing by the RPU 14, and then transferred to the display module 20 via the main bus 10. The image data output to the display module 20 is video-displayed on the LCD 23 by the digital encoder 21 and the LCD drive circuit 22. In this way, the photographer can adjust the composition and proper exposure of the subject while confirming the subject image displayed on the LCD 23.

  When the photographer issues a shooting command by pressing the shooting button or the like, the process proceeds to the all-pixel reading operation, and at the same time as the finder operation is canceled under the control of the CPU 17, the clock generator 29 drives the timing generator 16. Change the pulse. Next, the CCD sensor 12 is driven by the CCD driving circuit 15 so as to output signal charges of all pixels. The image signal read from the CCD sensor 12 is compressed after being subjected to real-time image processing by the RPU 14 as will be described later, and stored in a storage medium such as the memory card 27 through the card interface 27A, or the external interface 28. Or output to an external device such as a personal computer.

  The CCD sensor 12 may be an existing one, and generally includes a charge storage unit and a charge transfer unit, and an odd field and an even number consisting only of odd lines constituting the photosensitive unit during the all-pixel reading operation. It is driven by an interlace (interlaced scanning) system that reads out image signals from even-numbered fields consisting of only lines at different timings, and thinning out sequentially reads out image signals from a plurality of lines obtained by thinning out predetermined lines during the finder operation. Driven in a progressive manner. In this embodiment, the CCD sensor 12 is employed as the image sensor. However, the present invention is not limited to this, and a CMOS sensor may be used.

  Further, as shown in FIG. 2, the RPU 14 includes a single pixel processing unit (a single pixel processing block) 14a that processes a digital image signal in units of pixels and a pixel interpolation processing unit (Pixel) that performs pixel interpolation described in detail later. Interpolation Processing Block) 14b, Gamma Processing Block 14c for correcting gamma characteristics (gamma correction), Color Space Conversion & False Color Suppression Block 14d, A coring processing unit (Spatial Filter & Coring Block) 14e and an output unit (Resizing Block) 14f for outputting the image data processed by these units 14a to 14e to the main bus 10 are provided. Note that image processing that cannot be executed by the RPU 14 is executed by the CPU 17 using software. As a result, the processing speed can be improved several times to several tens of times compared with the case where all image processing is performed by the CPU 17, and the processing load on the CPU 17 can be reduced, so that the power consumption can be reduced. It becomes.

Embodiment 1 FIG.
3 and 4 are schematic configuration diagrams showing a flow of image signal processing according to the first embodiment in the digital still camera 1 having the above configuration. FIG. 3 is a diagram showing a flow of image signal processing during the viewfinder operation, and FIG. 4 is a diagram showing a flow of image signal processing during the all-pixel reading operation. FIG. 5 is a schematic diagram showing signal waveforms of various data during the viewfinder operation, and FIG. 6 is a schematic diagram showing signal waveforms of various data during the all-pixel reading operation.

  In the first embodiment, the frequency of the pixel transfer clock that drives the CCD sensor 12 and defines the transfer rate of pixel data during the finder operation is higher than the frequency of the pixel transfer clock during the all-pixel read operation. It is the feature that it is set to. The CCD sensor 12 at the time of the finder operation is driven by a pixel transfer clock having a signal waveform as shown in FIG. 5, and as described above, pixels from a plurality of lines obtained by thinning a predetermined horizontal line in accordance with the resolution of the finder (LCD 23). Controlled to output data. As shown in FIG. 3, a high-speed pixel transfer clock acts on the CCD sensor 12 during the finder operation period. The thinned CCD data is A / D converted by the analog signal processing circuit 13 and then input to the RPU 14 and subjected to image processing such as pixel interpolation processing, color space conversion processing, and contour enhancement processing in real time processing. Thereafter, it is transferred to the main memory 26 and temporarily stored in the processing data buffer 26a (step 1). At this time, there is no data signal input from the main memory 26 to the RPU 14 as shown in FIG. 5, and “R (red component)”, “G (green component)”, “G” after image processing as shown in FIG. Each data signal of “B (blue component)” is output from the RPU 14 to the main memory 26.

  Next, the image data stored in the processing data buffer 26a is read to the display module 20 via the main bus 10 and displayed on the LCD 23 having a finder function (step 2). The reason why the data processed by the RPU 14 is temporarily stored in the processing data buffer 26a is that the reading rate from the CCD sensor 12 and the reading rate of the display module 20 are different.

  Next, when the processing shifts during the all-pixel reading operation, a low-speed pixel transfer clock acts on the CCD sensor 12 as shown in FIG. The CPU 17 controls the clock generator 29 to cancel the finder operation of the CCD sensor 12, switches the pixel transfer clock frequency of the CCD sensor 12 to be equal to or less than the maximum value of the processing clock frequency of the RPU 14, and sets all the pixels of the CCD sensor 12. Control to read in interlaced mode. As shown in FIG. 4, the pixel data of the first field (either the even field or the odd field) read out first from the CCD sensor 12 is A / D converted by the analog signal processing circuit 13 and then the main memory 26. And is temporarily stored in the original image data buffer 26b provided in (Step 10).

  Next, the pixel data of the second field read from the CCD sensor 12 is A / D converted by the analog signal processing circuit 13 and then sequentially input to the RPU 14. At this time, the RPU 14 reads the first field temporarily stored in the original image data buffer 26b in synchronization with the reading of the second field, and executes image processing on the pixel data of both the first field and the second field. To do. The processed data is transferred and stored in the processed data buffer 26a provided in the main memory 26 via the main bus 10 (step 11).

  In steps 10 and 11, it is desirable that the data transfer between the original image data buffer 26b and the RPU 14 and between the RPU 14 and the processing data buffer 26a is directly performed without the CPU 17 under the control of the DMA controller 24. As a result, the load on the CPU 17 can be reduced, and the data processing speed can be increased.

  Image processing in the RPU 14 is as follows. The single pixel processing unit 14a performs one or both of multiplication and addition for each pixel of the digital image signal, thereby performing temporal averaging processing between a plurality of frames and shading correction processing within a single frame. Is a block that selectively performs any one of the above. In general, when a subject or the like is photographed by the CCD sensor 12, a phenomenon in which the peripheral light amount is reduced compared to the center position due to the optical properties of the lens used in the optical mechanism 11 is called shading. In the shading correction process, gain adjustment such as a luminance value in each pixel is executed in order to reduce the shading.

  The pixel interpolation processing unit 14 b and the gamma processing unit 14 c can fetch the image data once stored in the main memory 26 through the main bus 10 under the control of the DMA controller 24. Since image data can be directly input to the pixel interpolation processing unit 14b and the gamma processing unit 14c via the main bus 10 as well as the single pixel processing unit 14a in the first stage of the RPU 14, the image subjected to image processing by the CPU 17 Data can be directly input to the pixel interpolation processing unit 14b and the gamma processing unit 14c and processed without passing through the single pixel processing unit 14a.

Further, the color space conversion / color suppression processing unit 14d converts image data expressed in RGB three-color system or four-color system (such as YMCG system) into another color space coordinate system when the original signal is a color image signal. And a color suppression function for performing color suppression (chroma suppress: false color prevention) of bright and dark portions in an image. The destination of the coordinate system used in the color space conversion function, NTSC (National Television System Commitee) scheme YUV coordinate system adopted by such, YIQ coordinate system, it may be used and YC _b C _r coordinate system . For example, when using the YC _b C _r coordinate system to the color component conversion, the RGB components are converted into the coordinate system of the YC _b C _r component comprising a luminance signal Y and two color difference signals C _b, and C _r . Since the YC _{b Cr} component has a smaller correlation between the components than the RGB component, the image size can be compressed.

  In general, since dark portions appearing in an image are easily affected by various noises, suppressing color development in dark portions as much as possible leads to output of natural image quality. On the other hand, the bright part that appears in the image is a part that is likely to be modulated according to the characteristics of the image sensor that picked up the bright part and other various hardware components, and is a part that tends to get out of white balance. Suppresses the output of natural image quality. In consideration of these, the color suppression function suppresses the color development in the bright and dark areas appearing in the image.

  Further, the image signal processed by the spatial filter / coring processing unit 14e is output to the main bus 10 through the output unit 14f and temporarily stored in the processing data buffer 26a. Next, the processing data stored in the processing data buffer 26 a is transferred to the CPU 17. The CPU 17 uses the temporary storage data buffer 26c provided in the main memory 26, performs software processing such as JPEG compression on the image data transferred from the processing data buffer 26a, and then performs the memory card 27 and the external interface 28. Are transferred to and stored in the storage medium 31 of the external device connected via the terminal (step 12).

  As described above, the digital still camera 1 according to the first embodiment sets the pixel transfer clock at a high speed by setting the frequency of the pixel transfer clock at the time of the finder operation to the same level as the maximum clock frequency of the RPU 14 to read all pixels. During operation, the pixel transfer clock frequency is set to a value smaller than that during finder operation. Conventionally, as illustrated in the explanatory diagram of FIG. 7, the pixel transfer clock frequency does not exceed the maximum clock frequency (100 MHz) of hardware such as the RPU 14 throughout the finder operation period, that is, the thinning operation period and the entire pixel readout period. It was set to a constant value (50 MHz). Therefore, the RPU 14 processes the data read at the pixel transfer clock frequency during the finder operation period at the processing clock frequency 50 MHz, and on the other hand, when reading out the pixel data of the first field during the entire pixel reading period, Since the data is transferred to and stored in the main memory, the processing clock of the RPU 14 is 0 MHz, and when reading the pixel data of the second field, the RPU 14 is a pixel that combines the first field and the second field at a processing clock frequency of 100 MHz. The data was being processed. However, this has a problem that the frame rate of the finder display is low as described above.

  On the other hand, in the first embodiment, as illustrated in the explanatory diagram of FIG. 8, the pixel transfer clock frequency during the finder operation period is set to a maximum clock frequency of 100 MHz for driving hardware such as the RPU 14. In the all pixel readout period, the same pixel transfer clock frequency and processing clock frequency as those in the conventional example shown in FIG. 7 are set, and the pixel transfer clock frequency is switched between the finder operation period and the all pixel readout period. . As described above, during the finder operation period, the pixel transfer clock frequency can be set to the maximum value of the drive clock frequency of the RPU 14 to increase the frame rate of the finder display.

Embodiment 2. FIG.
Next, a second embodiment of the present invention will be described. There are various types of color filter arrays arranged on the pixels of the CCD sensor 12. For example, as shown in FIG. 9, the basic form of the Bayer primary color filter array is a color filter “R (red component) on each pixel. ) ”,“ G (green component) ”,“ R ”,“ G ”,..., And odd-numbered lines and color filters are assigned to“ G (green component) ”,“ B (blue component) ”,“ G , “B”,... Are evenly arranged in the vertical direction. A set of colors is expressed by “R”, “G”, “G”, and “B” of 2 × 2 = 4 pixels (pixels) surrounded by the thick line frame 33a. Although not shown, there is also a complementary color type (YMCG or YMCK) color filter array. As a YMCG color filter array, color filters are arranged in the order of “C (cyan color component)”, “M (magenta color component)”, “C”, “M”,. There are some in which even lines arranged in the order of “Y (yellow color component)”, “G (green component)”, “Y”, “G”,... Are alternately arranged in the vertical direction.

  As shown in FIG. 10, the pixel data output from the CCD sensor 12 having such a color filter array is transferred to each block of the line memories 41A, 41B,..., 41E provided in the pixel interpolation processing unit 14b of the RPU 14 for each line. Are stored sequentially. A pixel interpolation value in the central block P22 is calculated by applying a pixel interpolation unit (not shown) to pixel data of 5 × 5 = 25 pixels stored in the line memories 41A to 41E and surrounded by the thick line frame 40. The In the case of the Bayer-type primary color filter and YMCG color filter array, pixel data having the same color component as the pixel data stored in the block P22 among the blocks P00 to P44 to which the pixel interpolation unit is applied is hatched. The data is stored in the block with P00, P02, P04, P20, P24, P40, P42 and P44. For example, when there is a line passing through pixels corresponding to the blocks P02 and P42, an average value or an intermediate value of the pixel data stored in both the blocks P02 and P42 can be calculated and stored in the block P22 as a pixel interpolation value. Also, when there are lines passing through the blocks P20 and P24, lines passing through the blocks P00 and P44, and lines passing through the blocks P04 and P40, pixel interpolation values can be calculated in the same manner as described above.

  Further, instead of applying a pixel interpolation unit to 5 × 5 pixels as shown in FIG. 10, three line memories 41A, 41B, and 41C are provided in the pixel interpolation processing unit 14b as illustrated in FIG. A pixel interpolation unit (not shown) may be applied to 3 × 3 = 9 pixel data stored in the block P00 to P22 of the line memory and surrounded by the thick line frame 43. When pixel data is input to the line memories 41A to 41C from the CCD sensor having the primary color filter array shown in FIG. 9 or the YMCG color filter array, the blocks P00 and P02 in diagonal four directions with respect to the central block P11. , P20 and P22 have the same color component. For example, when the color component of the pixel data stored in the central block P11 is “B”, all the color components of the pixel data stored in the diagonal blocks P00, P02, P20, and P22 are “R”. The average value or intermediate value of the pixel data stored in the diagonally square blocks P00, P02, P20, and P22 can be stored in the central block P11 as a pixel interpolation value.

  Actually, a plurality of various pixel interpolation units as shown in FIGS. 10 and 11 are incorporated in the pixel interpolation processing unit 14b, and the pixel interpolation unit is appropriately set according to settings of a driver circuit, driver software, and the like for driving the RPU 14. Configured to be selected.

  In the second embodiment, during the finder operation period, a plurality of lines of pixel data obtained by thinning a predetermined horizontal line in accordance with the resolution of the finder (LCD 23) is output from the CCD sensor 12. The pixel data is sequentially stored for each line in the line memories 41A, 41B, 41C,... 41E provided in the pixel interpolation processing unit 14b of the RPU 14 as illustrated in FIG. By applying the pixel interpolation unit to the pixel data of 5 × 5 pixels (= 25 pixels) stored in such a line memory, the interpolation value of the pixel data of the central block is calculated.

  On the other hand, during the all-pixel readout period, as shown in FIG. 4, the CCD sensor 12 outputs one of the even field and odd field, and the first field is the original image data provided in the main memory 26. The data is temporarily stored in the buffer 26b (step 10). Next, the second field read from the CCD sensor 12 and the first field read from the original image data buffer 26b in synchronization with the reading of the second field are sent to the RPU 14 as illustrated in FIG. The stored line memories 41A to 41F are temporarily stored. When pixel data of the second field is buffered in the illustrated line memory 41A, the pixel data of the second field read out first is buffered in the line memories 41C and 41E and stored in the line memories 41B, 41D, and 41F. The first field pixel data is buffered.

  In the second embodiment, in one half period (T / 2) of one readout period (T) of the pixel transfer clock, the pixel interpolation unit is replaced with a thick line frame of the line memories 41A to 41E as shown in FIG. This is applied to the block surrounded by 40, and the remaining half cycle (T / 2) is surrounded by the thick line frame 40 of the line memories 41B to 41F shifted by one step in the vertical direction as shown in FIG. 13B. It is characterized by being applied to blocks. As a result, pixel interpolation processing can be executed in a time-sharing manner on the pixel data of the line memories 41C and 41D of the upper and lower two lines in the vertical direction, and the interpolation value of two pixels can be efficiently calculated in one readout cycle (T). It becomes. At this time, as shown in FIG. 6, data signals “R”, “G”, and “B” after pixel interpolation of the upper line and the lower line can be generated in one read cycle (T). In this way, when the processing speed of the RPU 14 is sufficiently high, the pixel interpolation unit can be shared in a time-sharing manner, so that the circuit configuration can be reduced, and the power saving and cost reduction of the apparatus can be achieved.

  In the second embodiment, the pixel interpolation unit is applied to pixel data of 5 × 5 pixels. However, the pixel interpolation unit may be applied to pixel data of 3 × 3 pixels instead. .

  Further, when the second embodiment is applied to the image processing circuit according to the first embodiment, the pixel transfer clock is set to a high speed during the finder operation period, and the pixel transfer clock is set to a low speed during the entire pixel reading period. Therefore, time division processing is facilitated, and pixel interpolation processing can be performed efficiently. In the all-pixel readout operation, the pixel data of the first field and the pixel data of the second field are input to the RPU 14 in synchronization with each readout cycle of the pixel transfer clock. Can be collectively subjected to pixel interpolation processing.

Embodiment 3 FIG.
Next, an image processing circuit according to Embodiment 3 of the present invention will be described. In the image processing method according to the second embodiment, the effect is likely to appear when the processing speed of the RPU 14 is high, but the effect may not increase when the processing speed of the RPU 14 is low. In order to deal with such a case, in the third embodiment, the RPU 14 has a plurality of pixel interpolation units, and each pixel data of a plurality of lines is provided with one-to-one pixel interpolation units for each pixel transfer clock read cycle (T). It is characterized by applying and performing a plurality of pixel interpolation processes in parallel. Hereinafter, the image processing method according to the third embodiment will be described in detail by taking as an example a case where two pixel interpolation units are incorporated in the RPU 14.

  During the finder operation period, as illustrated in FIG. 14, the same processing as that in the explanatory diagram shown in FIG. 12 is executed. That is, interpolation is performed by applying a pixel interpolation unit to pixel data of 5 × 5 pixels stored in the line memories 41A to 41E and surrounded by the thick line frame 50 during one readout cycle (T) of the pixel transfer clock. The obtained pixel data is calculated.

  Next, during the all-pixel readout period, as illustrated in FIG. 15, the RPU 14 is stored in the line memories 41A to 41E and surrounded by the thick line frame 50A during one readout cycle (T) of the pixel transfer clock. Pixel data A interpolated by applying the first pixel interpolation unit to the pixel data of × 5 pixels is calculated, and in parallel therewith, 5 is stored in the line memories 41B to 41F and surrounded by a dotted frame 50B. Pixel data B interpolated by applying the second pixel interpolation unit to the pixel data of x5 pixels is calculated. As a result, during one cycle (T) of the pixel transfer clock, pixel interpolation processing can be performed in parallel on the pixel data in the line memories 41C and 41D of the upper and lower two lines in the vertical direction.

  As described above, according to the third embodiment, parallel pixel interpolation processing for the number of pixel interpolation units incorporated in the RPU 14 can be executed during one readout cycle (T) of the pixel transfer clock, so that a high-speed image can be obtained. Processing can be realized. Further, in the combination of the first embodiment and the third embodiment, the pixel data of the first field and the pixel data of the second field are synchronized at every reading cycle of the pixel transfer clock during the all-pixel reading operation. Although it is input to the RPU 14, pixel data of both fields can be collectively subjected to pixel interpolation processing for each readout cycle.

  As described above, the interpolation pixel data calculated in the second and third embodiments during the all-pixel reading operation is output from the pixel interpolation processing unit 14b for two pixels every one cycle (T) of the pixel transfer clock. When the pixel transfer clock used in the pixel interpolation processing unit 14b is used as it is in the subsequent gamma processing unit 14c or later, two pixels are subjected to gamma correction processing, color space conversion processing, and contour enhancement processing during one cycle of the pixel transfer clock. There is a problem that the image processing such as. Image processing methods that can solve this problem will be described in Embodiments 4 to 6 below.

Embodiment 4 FIG.
In the image processing circuit according to the fourth embodiment, a pixel transfer clock frequency (f1) that drives the CCD sensor 12 and defines a transfer rate of pixel data, the gamma processing unit 14c, a color space conversion / color suppression processing unit, and the like. 14d, the pixel processing clock frequency (f2) that defines the processing speed of the pixel data is individually set in the post-processing unit including the spatial filter / coring processing unit 14e and the output unit 14f, and the pixel processing clock frequency (f2 ) Is set to a value more than twice the pixel transfer clock frequency (f1).

  Specifically, there are two types of pixel processing clocks used in the pixel interpolation processing unit 14b, depending on the type of CCD, the case where the pixel processing clock is equal to the pixel transfer clock and the case where the pixel processing clock is used in the post-processing unit. In the fourth embodiment, a case where the pixel processing clock used in the pixel interpolation processing unit 14b is equal to the pixel transfer clock will be described as an example. As shown in FIG. 16, a pixel transfer clock (frequency: f1) is used in the pixel interpolation processing unit 14b, and a pixel processing clock (frequency: f2 ≧ 2 × f1) is used in the post-processing units 14c to 14f. During the all-pixel reading operation, the pixel data of the first field stored in the original image data buffer 26b provided in the main memory 26 is DMA-transferred to the RPU 14 and simultaneously synchronized with the reading of the pixel data of the first field. The pixel data of the second field is read and input to the RPU 14. The pixel data of the first and second fields input to the RPU 14 are processed by the single pixel processing unit 14a (not shown) and then input to the pixel interpolation processing unit 14b. Next, the pixel data in the first and second fields are processed by the pixel interpolation method as in the second and third embodiments, and two pixels in two lines are provided during one period (1 / f1) of the pixel transfer clock. Data is output to the post-processing units 14c to 14f, and the post-processing units 14c to 14f execute image processing such as gamma correction processing one pixel at a time during one cycle (1 / f2) of the pixel processing clock. Pixel data output from the post-processing units 14 c to 14 f is DMA-transferred to a processing data buffer 26 a provided in the main memory 26.

  As described above, according to the fourth embodiment, since the processing rate of the post-processing units 14c to 14f is equal to or higher than the processing rate of the pixel interpolation processing unit 14b, real-time (real-time) image processing can be reliably performed. It becomes possible to improve the reliability of the.

Embodiment 5 FIG.
Next, FIG. 17 is a schematic configuration diagram showing a flow of image signal processing according to Embodiment 5 of the present invention. As in the fourth embodiment, a pixel transfer clock (frequency: f1) is used for the pixel interpolation processing unit 14b, and a pixel processing clock (frequency: f2 ≧ 2 × f1) is used for the post-processing units 14c to 14f. Yes. As described in the second and third embodiments, during the all-pixel reading operation, as shown in FIGS. 13 and 14, two pixel data are received from the line memories 41C and 41D in one cycle (T) of the pixel transfer clock. Output as line data.

  As shown in FIG. 17, the pixel data output from the two lines 1 and 2 are multiplexed by the multiplexer 51 and rearranged in series, and are transferred to the rate adjusting buffer 26d provided in the main memory 26 as one output. Transferred. More specifically, as shown in FIG. 18, the pixel data indicated by “L1” and “L2” stored in the line memories 41K and 41L provided in the pixel interpolation processing unit 14b are interpolated and then multiplexed. The data is multiplexed at 51 and DMA-transferred to a rate adjusting buffer 26d via a FIFO (Fast-In Fast-Out) memory (not shown), and the rate is set in an array of L1, L2, L1, L2, L1, L2,. It is stored in the adjustment buffer 26d.

  Next, the post-processing units 14c to 14f read the pixel data stored in the rate adjusting buffer 26d by addressing every other pixel. As a result, the post-processing units 14c to 14f can perform image processing by reading out pixel data line by line like L1, L1,..., L1, L2, L2,.

Embodiment 6 FIG.
FIG. 19 is a schematic diagram showing a flow of image signal processing according to Embodiment 6 of the present invention. As in the fourth embodiment, the pixel processing clock frequency used in the post-processing units 14c to 14f is set to a value that is twice or more the pixel transfer clock frequency used in the pixel interpolation processing unit 14b. At the time of reading all pixels, the pixel data of the first and second fields are input to the RPU 14 in synchronization with each other, processed by a single pixel processing unit 14a (not shown), and then input to the pixel interpolation processing unit 14b. To do. Two pixel data of two lines subjected to the interpolation by the pixel interpolation processing unit 14b during one cycle of the pixel transfer clock are output for each of the lines 1 and 2 shown in the figure, and the FIFO memory (not shown). The data is transferred to the rate adjusting buffer 26d via DMA. A channel of the DMA controller 24 is assigned to each output line, and the DMA controller 24 monitors and distributes the memory cycle, thereby transferring and storing the pixel data to the rate adjusting buffer 26d in line order. As a result, the pixel data is arranged and stored in the rate adjustment buffer 26d in line-sequential manner.

  The pixel data stored in the rate adjustment buffer 26d is read out in order from the beginning, DMA-transferred to a FIFO memory (not shown), and then output to the post-processing units 14c to 14f. After the post-processing units 14c to 14f perform color space conversion processing, edge enhancement processing, and the like on the input pixel data using the pixel processing clock, the processing data is stored in the processing data buffer 26a provided in the main memory 26. DMA transferred.

  Therefore, according to the sixth embodiment, the pixel data output from the pixel interpolation processing unit 14b is rearranged line-sequentially by the channel control of the DMA controller 24 and stored in the rate adjustment buffer 26d. In 14f, a series of image processing can be executed line by line.

Embodiment 7 FIG.
FIG. 20 is a schematic diagram showing the flow of image signal processing according to Embodiment 7 of the present invention. As in the fourth embodiment, the pixel processing clock frequency used in the post-processing units 14c to 14f is set to a value that is twice or more the pixel transfer clock frequency used in the pixel interpolation processing unit 14b. At the time of reading all pixels, the pixel data of the first and second fields are input to the RPU 14 in synchronization with each other, processed by a single pixel processing unit 14a (not shown), and then input to the pixel interpolation processing unit 14b. To do.

  The pixel data for two lines subjected to interpolation by the pixel interpolation processing unit 14b is output for each of the lines 1 and 2 shown in the figure, and is written to the first line memory 53 incorporated in the RPU 14 for each line. Next, the pixel data for two lines stored in the first line memory 53 is read out for each line at a speed twice as fast as the writing speed to the first line memory 53, and the lines are sequentially processed by the post-processing units 14c to 14f. And image processing such as gamma correction and color space conversion. In parallel with the reading of the pixel data from the first line memory 53, the pixel data for two new lines subjected to pixel interpolation is stored in the second line memory 54.

  After the pixel data stored in the first line memory 53 is read and the pixel data for two new lines are stored in the second line memory 54, the pixel data in the second line memory 54 is stored in the first line memory. The data is read out for each line at a speed twice as high as the writing speed to 53, and is sequentially input to the post-processing units 14c to 14f to be subjected to image processing such as gamma correction. In parallel, the first line memory 53 stores pixel data for two new lines subjected to pixel interpolation. Pixel data subjected to image processing by the post-processing units 14 c to 14 f is DMA-transferred to a processing data buffer 26 a provided in the main memory 26.

  In the seventh embodiment, two single-port memories 53 and 54 that can store pixel data for two lines are used, but a dual-port memory that can store pixel data for four lines is used instead. Good.

  As described above, in the seventh embodiment, the pixel data output from the pixel interpolation processing unit 14b in the RPU 14 can be rearranged in a line order and output to the post-processing units 14c to 14f.

Modifications of Embodiments 4 to 7
The image processing circuits according to Embodiments 4 to 7 can be applied to a CCD sensor that outputs two pixel data during one period of the pixel transfer clock. The CCD sensor 12 used in the above embodiment extracts one pixel data from one line during one cycle of the pixel transfer clock, but as a progressive CCD sensor, the pixel transfer clock is mainly used to improve the reading speed. There is also a CCD sensor that takes out two pixel data from two lines during one cycle. FIG. 21 is a schematic diagram showing the structure of the CCD sensor 12 (interline transfer CCD structure) used in the above embodiment. As shown in FIG. 21, vertical transfer CCDs 81, 81,... Are arranged between the columns of the charge storage units 80, 80,. One horizontal transfer CCD 82 is provided adjacent to the vertical transfer CCD 81. In FIG. 21, “R”, “G”, and “B” color component symbols are attached to the pixels constituting the charge storage unit 80. Accumulated charges in the photodiodes of the charge accumulation unit 80 are transferred to the vertical transfer CCD 81 for each horizontal line, and the vertical transfer CCD 81 transfers signal charges in the vertical transfer direction 83 in parallel. The horizontal transfer CCD 82 transfers the signal in the horizontal direction every time a signal for one line is input, and outputs the signal from one signal output terminal 85 to the outside via an output circuit (sense amplifier) 84.

  Next, FIG. 22 is a schematic diagram showing a CCD structure that outputs pixel data of two lines during one cycle of the pixel transfer clock. In FIG. 22, reference numeral 90 denotes a charge storage unit, 91 denotes a vertical transfer CCD arranged between the columns of the charge storage units 90,... 92A and 92B denote horizontal transfer CCDs. The stored charges in the photodiodes of the charge storage units 90,... Are transferred to the vertical transfer CCDs 91,... And then transferred in parallel in the vertical transfer direction 93 every two horizontal lines. The two lines of signals are simultaneously transferred in the horizontal direction by the first horizontal transfer CCD 92A and the second horizontal transfer CCD 92B, and output from the signal output terminals 95A and 95B via the output circuits (sense amplifiers) 94A and 94B. Since two lines of pixel data are output in one cycle of the pixel transfer clock in this way, the pixel data can be read out at twice the speed as compared with the CCD structure shown in FIG.

  Therefore, in the first modification, the two-line pixel data shown in FIG. 22 is input to the pixel interpolation processing unit 14b of the RPU 14 in the fourth embodiment instead of the pixel data of the first field and the second field. The pixel processing clock frequency (f2) used in the post-processing units 14c to 14f is set to a value more than twice the pixel transfer clock frequency (f1) for driving the CCD sensor having the structure of FIG. This makes it possible to efficiently perform real-time image processing by setting the processing rate of the post-processing units 14c to 14f to be higher than the processing rate of the pixel interpolation processing unit 14b.

  Furthermore, it is desirable to adopt the configuration of any of the above-described fifth to seventh embodiments by replacing the input data to the pixel interpolation processing unit 14b with the two lines of pixel data shown in FIG. As a result, the two lines of interpolated pixel data output from the pixel interpolation processing unit 14b can be converted into lines sequentially and post-processed.

Modification 2 of Embodiments 4-7
In addition, the image processing circuits according to Embodiments 4 to 7 described above can be applied to an image pickup device having a honeycomb pixel array described below. FIG. 23 is a schematic diagram for explaining a honeycomb pixel arrangement. In FIG. 23, “R”, “G”, and “B” color filter symbols are attached to each unit pixel cell. From such an image pickup device pixel data of the line along the imaginary line 70 _{1-70 5} is read. The pixel data of each line is further divided into a “G” line and other color component lines and stored in a line memory (register) as shown in FIG. That is, the pixel data of the line virtual line 70 ₁ shown in FIG. 23 is stored in the set 71 of the line memory shown in FIG. 24, "R", "B", "R", ... pixel color components Data is stored in the line memory 71A every other pixel. The symbol “I” means a block in which no pixel data is stored. On the other hand, pixel data of color components “G”, “G”,... Are stored in the line memory 71B every other pixel so as to be adjacent to the “I” block on the line memory 71A in the vertical direction. Pixel data on a line of another imaginary line 70 _{2-70 5} shown in FIG. 23 are also stored as in the phantom line 70 _1, respectively in a line memory 72-75 shown in FIG. 24.

  For the pixel data stored in such a form, pixel interpolation processing is executed every two lines for every cycle of the pixel transfer clock. An example of the pixel interpolation method will be described below with reference to FIG. First, as shown in FIG. 25A, for the “R” component block of the line memory 73A, the average or intermediate value of the “B” component on both sides in the horizontal direction is interpolated as “B ′ (blue component)”. For the “R” component, an average value or an intermediate value of the “G” components in the diagonal line memories 72B and 73B is interpolated as “G ′ (green component)”. Next, as shown in FIG. 25B, the average value of the same color components of “B”, “B”, “R”, and “R” in diagonal four directions or the “G” component block of the line memory 73B The intermediate values are interpolated as “R ′ (red component)” and “B ′ (blue component)”. Then, as shown in FIG. 25C, by calculating the average value or intermediate value of the color components interpolated in FIG. 25B, “R ″ (red) of the block in which no pixel data is accumulated. Component) "," G "(green component), and" B "(blue component)" are interpolated. With this type of pixel interpolation method, two pieces of interpolated pixel data are output from two line memories during one cycle of the pixel transfer clock.

  Therefore, when the pixel transfer clock used for the pixel interpolation processing is directly used for the subsequent image processing (gamma correction processing, color space conversion, color suppression processing, etc.) for the image pickup device having the honeycomb type pixel array, 1 of the pixel transfer clock is used. The above-described problem arises that two lines of interpolated pixel data must be processed simultaneously during a period. In order to solve this problem, the pixel interpolation processing unit 14b of the RPU 14 in the fourth embodiment outputs two lines output from the image sensor of the honeycomb type pixel array instead of the pixel data of the first field and the second field. And the pixel processing clock frequency (f2) used in the post-processing units 14c to 14f is set to a value more than twice the pixel transfer clock frequency (f1) for driving the image sensor of the honeycomb pixel array. can do. As a result, it is possible to perform image processing on the pixel data output from the imaging sensor having the honeycomb pixel array at a speed that can correspond to the processing rate of the pixel interpolation processing unit 14b by the post-processing units 14c to 14f. Furthermore, it is preferable to adopt the configuration of any of the above-described Embodiments 5 to 7 in order to convert the interpolated pixel data output in two lines into line-sequential and post-process.

It is the schematic which shows the whole structure of the digital still camera which concerns on embodiment of this invention. It is the schematic which shows the functional block which comprises the real-time processing unit (RPU) which concerns on embodiment of this invention. It is a schematic block diagram which shows the flow of the image signal process which concerns on Embodiment 1 of this invention. FIG. 3 is a schematic configuration diagram showing a flow of image signal processing according to the first embodiment. FIG. 3 is a diagram schematically illustrating a signal waveform of the image processing circuit according to the first embodiment. FIG. 3 is a diagram schematically illustrating a signal waveform of the image processing circuit according to the first embodiment. In the conventional image processing circuit, it is the prior art which illustrated the pixel clock frequency and the processing clock frequency of RPU in each period of a finder operation | movement period and all the pixel readout periods. In the image processing circuit according to the first embodiment, the pixel clock frequency and the RPU processing clock frequency in each of the finder operation period and the all pixel readout period are illustrated. It is a figure which shows the basic form of a color filter array of a Bayer system. 4 is a diagram illustrating an example of a pixel interpolation unit of the image processing circuit according to Embodiment 1. FIG. It is a figure which shows the other example of a pixel interpolation unit. It is the schematic for demonstrating the pixel interpolation process which concerns on Embodiment 2 of this invention. 10 is a schematic diagram for explaining pixel interpolation processing according to Embodiment 2. FIG. It is the schematic for demonstrating the pixel interpolation process which concerns on Embodiment 3 of this invention. 12 is a schematic diagram for explaining pixel interpolation processing according to Embodiment 3. FIG. It is the schematic for demonstrating the pixel interpolation process which concerns on Embodiment 4 of this invention. It is the schematic which shows the flow of the image signal process which concerns on Embodiment 5 of this invention. 10 is a schematic diagram for explaining image signal processing according to Embodiment 5. FIG. It is the schematic which shows the flow of the image signal process which concerns on Embodiment 6 of this invention. It is the schematic which shows the flow of the image signal process which concerns on Embodiment 7 of this invention. It is the schematic which shows the example of the CCD structure which outputs the pixel data of 1 line. It is the schematic which shows the example of the CCD structure which outputs the pixel data of 2 lines. It is the schematic for demonstrating a honeycomb type pixel arrangement | sequence. FIG. 24 is a schematic diagram showing a line memory group storing pixel data read from an image sensor having the pixel arrangement shown in FIG. 23. (A)-(c) is explanatory drawing which shows an example of the pixel interpolation method. It is a block diagram which shows schematic structure of a common digital still camera. It is the schematic which shows the example of the flow of the conventional image signal processing.

Explanation of symbols

1 Digital Still Camera 26 Main Memory 26a Processing Data Buffer 26b Original Image Data Buffer 26c Temporary Storage Data Buffer 31 Storage Medium

Claims (5)

An imaging sensor;
A pixel interpolation processing unit that performs real-time pixel interpolation processing on pixel data read from the image sensor and outputs interpolation pixel data of a plurality of lines for each cycle of a pixel transfer clock that defines a transfer speed of the pixel data And an image processing unit having a post-processing unit that further executes real-time image processing on the interpolated pixel data,
An image processing circuit, wherein the post-processing unit sets a frequency of a pixel processing clock that defines a processing speed of the pixel data to a value that is twice or more a frequency of the pixel transfer clock. The image processing circuit according to claim 1,
A multiplexing unit that multiplexes pixel data of a plurality of lines output from the pixel interpolation processing unit for each cycle of the pixel transfer clock and outputs the multiplexed data from one output line;
A buffer unit for storing pixel data output from the multiplexing unit;
The image processing circuit, wherein the post-processing unit reads out pixel data stored in the buffer unit in a line-sequential manner and executes image processing.   3. The image processing circuit according to claim 2, further comprising a DMA (direct memory access) controller that controls direct transfer of pixel data from the pixel interpolation processing unit to the buffer unit. The image processing circuit according to claim 1,
A buffer unit for storing pixel data of a plurality of lines output from the pixel interpolation processing unit for each cycle of the pixel transfer clock;
An image processing circuit further comprising: a DMA (direct memory access) controller that performs channel control so that the pixel data is sequentially transferred to the buffer unit on a line-by-line basis. The image processing circuit according to claim 1,
The image processing unit includes a first storage unit that stores a plurality of lines of pixel data output from the pixel interpolation processing unit for each line;
A second storage unit that stores pixel data output from the pixel interpolation processing unit for each line while the post-processing unit sequentially reads out the pixel data stored in the first storage unit line by line;
Image processing in which the first storage unit stores the pixel data output from the pixel interpolation processing unit for each line while the post-processing unit reads the pixel data stored in the second storage unit sequentially in a line. circuit.

Images