WO2005025232A1 - 撮像装置 - Google Patents
撮像装置 Download PDFInfo
- Publication number
- WO2005025232A1 WO2005025232A1 PCT/JP2004/012636 JP2004012636W WO2005025232A1 WO 2005025232 A1 WO2005025232 A1 WO 2005025232A1 JP 2004012636 W JP2004012636 W JP 2004012636W WO 2005025232 A1 WO2005025232 A1 WO 2005025232A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- full
- data
- color
- image
- color data
- Prior art date
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/80—Camera processing pipelines; Components thereof
- H04N23/84—Camera processing pipelines; Components thereof for processing colour signals
- H04N23/843—Demosaicing, e.g. interpolating colour pixel values
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
- H04N25/10—Circuitry of solid-state image sensors [SSIS]; Control thereof for transforming different wavelengths into image signals
- H04N25/11—Arrangement of colour filter arrays [CFA]; Filter mosaics
- H04N25/13—Arrangement of colour filter arrays [CFA]; Filter mosaics characterised by the spectral characteristics of the filter elements
- H04N25/134—Arrangement of colour filter arrays [CFA]; Filter mosaics characterised by the spectral characteristics of the filter elements based on three different wavelength filter elements
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N2209/00—Details of colour television systems
- H04N2209/04—Picture signal generators
- H04N2209/041—Picture signal generators using solid-state devices
- H04N2209/042—Picture signal generators using solid-state devices having a single pick-up sensor
- H04N2209/045—Picture signal generators using solid-state devices having a single pick-up sensor using mosaic colour filter
- H04N2209/046—Colour interpolation to calculate the missing colour values
Definitions
- the present invention relates to a digital imaging device, and more particularly, to a digital imaging device that generates an image having a smaller number of pixels than an imaging device mounted on the imaging device at high speed and with high image quality.
- the number of pixels of the output image is smaller than the number of pixels of the image sensor, the number of pixels to be used is limited in advance, or a plurality of pixels are averaged and read out in one clock.
- the amount of data transferred from the image sensor to the memory can be reduced, and the speed of the memory transfer can be improved.
- Japanese Patent Laying-Open No. 2001-016441 discloses an apparatus for thinning out data and correcting data distortion when the type of resolution is limited. This document discloses, in an embodiment, the creation of data equivalent to 400 dpi by a device having a resolution of 600 dpi. If the data of 600 dpi is thinned as it is, data distortion will occur. Therefore, pixel data for correcting the position distortion is generated by linear interpolation for the data power of 600 dpi.
- the present invention has been made in view of such a situation, and an object of the present invention is to reduce distortion due to magnification (size change) in which the time required to read image data from an image sensor is short. It is an object of the present invention to provide an imaging device capable of forming a few high-definition full-color images.
- An imaging device is an imaging device including an imaging device that photoelectrically converts an image formed by an optical system, wherein a color filter array (CFA) having a different spectral transmittance is mounted on the imaging device.
- CFA color filter array
- the luminance signal of each color of the CFA and the position information of the sampling are sampled to obtain an image signal, and when converting the pixel signal array to full color data, A means for generating a weighting factor for generating each color component of the full-color data based on the relative relationship between the position of the grid array of the full-color data to be generated and the positional information of the pixel signal, and having means for generating the full-color data; are doing.
- the present invention is directed, in part, to a method for converting full color image data.
- the pixel signal at each position of the input image data has information of one component constituting the full-color data, and is generated when the pixel signal array is converted into the full-color data.
- a weighting factor for generating each color component of the full-color data based on the relative relationship between the position of the grid array of the full-color data and the positional information of the pixel signal, and generating the full-color data. .
- FIG. 1 schematically shows a configuration of an imaging device according to a first embodiment of the present invention.
- FIG. 2 shows an example of an averaging readout mode.
- FIG. 3 shows a relationship between a pixel position and a barycenter position in averaging readout.
- FIG. 4 shows a configuration of an imaging device according to a second embodiment of the present invention.
- FIG. 5 shows a thinning-out reading rule corresponding to a 75% reduction in the second embodiment.
- FIG. 6 shows data read out by thinning out two pixels out of eight pixels in the horizontal direction on the R'G line on the image sensor on which the color filter array is mounted in the second embodiment, as R of six pixels.
- FIG. 9 schematically shows how pixel data is converted to pixel data at equal intervals having G color information.
- FIG. 7 schematically illustrates processing of distortion correction in the vertical direction and full-color conversion in the second embodiment.
- FIG. 8 schematically shows a configuration of a pipeline for performing distortion correction and color complementation processing in the second embodiment, in which data of five pixels R, G, R, G, and R is reading Showed the state of being out and out.
- FIG. 9 shows a state of the pipeline after a lapse of one clock unit from the state of FIG. 8, and data of five pixels of G, R, G, R, and G are read out.
- FIG. 10 shows a state of the pipeline after a lapse of one clock unit from the state of FIG.
- FIG. 11 shows a state of the pipeline after a lapse of one clock unit from the state of FIG.
- FIG. 12 shows a state of the pipeline after a lapse of one clock unit from the state of FIG.
- FIG. 13 shows a state of the pipeline after a lapse of one clock unit from the state of FIG.
- FIG. 14 schematically shows the configuration of a pipeline that performs vertical complementation of the G channel.
- FIG. 15 schematically shows a state in which only distortion correction processing is performed in the horizontal direction of the R'G line to generate a Bayer array at equal intervals.
- FIG. 16 shows a configuration of an imaging device according to a third embodiment of the present invention.
- FIG. 17 shows, in the third embodiment, one of two frames temporally adjacent to each other in reading by repeating 6/8 thinning-out reading.
- FIG. 18 shows the other of two frames temporally adjacent to each other in reading by repeating 6/8 thinning-out reading in the third embodiment.
- FIG. 19 shows a method of converting the averaged sampled data to image data according to the magnification.
- Fig. 20 shows a flow of converting the thinned-out sampled data into image data according to a scaling factor after performing distortion correction and full color processing.
- FIG. 21 shows a complementing process between consecutive frames.
- FIG. 22 shows details of horizontal distortion correction and colorization processing in the flow of FIG. 20.
- FIG. 23 shows details of the distortion correction process in the vertical and horizontal directions in the flow of FIG.
- FIG. 24 shows a procedure for calculating the contents of the LUT in FIG.
- FIG. 25 shows a procedure for calculating the contents of the LUT in FIG.
- FIG. 1 shows the configuration of the imaging apparatus according to the first embodiment of the present invention.
- the imaging apparatus 100 includes an imaging optical system 110 that forms an optical image of a subject, and an imaging device 120 that outputs an image signal of a predetermined region of the optical image formed by the imaging optical system 110.
- the imaging device 120 includes an area-shaped photoelectric conversion element (imaging element) 122 that photoelectrically converts an optical image formed by the imaging optical system 110 to obtain digital image data (a set of pixel data).
- a read control unit 124 that reads the image data obtained by the element 122 in accordance with the supplied read rule.
- the photoelectric conversion element 122 includes, for example, a color filter array having a Bayer array, and acquires RGB data of an optical image formed by the imaging optical system 110.
- the photoelectric conversion element 122 is capable of averaging reading, and may be, for example, an image sensor using CMOS (hereinafter simply referred to as CMOS) or a CCD.
- CMOS complementary metal-oxide-semiconductor
- CCD complementary metal-oxide-semiconductor
- the imaging device 100 further includes an image scaling ratio selection unit 132 for selecting a scaling factor of an image to be output, and image data read by the readout control unit 124 from the photoelectric conversion element 122 in accordance with the selected image scaling factor. And a read-out method selection unit 134 for selecting one of the read-out methods.
- the readout method selection unit 134 selects an averaged readout method or an all-pixels readout method according to the selected image scaling factor. More specifically, the reading method selection unit 134 selects the all-pixel reading method when the image reduction ratio is large (for example, when the image scaling ratio is 90% or more), and when the image reduction ratio is less than that, the averaged reading method Select a method.
- the imaging device 100 further includes an all-pixels reading unit 142 and an averaging reading unit 144 that are activated in accordance with the reading method selected by the reading method selection unit 134.
- the all-pixel reading unit 142 supplies a reading rule corresponding to the all-pixel reading method to the reading control unit 124
- the averaging reading unit 144 supplies a reading rule corresponding to the averaging reading method to the reading control unit 124.
- the read control unit 124 reads pixel data from the photoelectric conversion element 122 according to a read rule.
- the image capturing apparatus 100 further includes a resizing / full-color processing unit 152 and a look-up table unit 154 for storing a plurality of filter coefficients.
- the size change / full-color conversion processing unit 152 performs size change and full-color processing on the image data read from the photoelectric conversion element 122 by filter processing.
- the look-up table unit 154 supplies an appropriate filter coefficient to the size change / full color conversion processing unit 152 according to the magnification selected by the image magnification selection unit 132.
- the imaging apparatus 100 further includes a video signal processing unit 172 that performs predetermined processing (such as white balance, gradation conversion, and edge enhancement) on the image signal output from the size change 'full color processing unit 152.
- a video signal processing unit 172 that performs predetermined processing (such as white balance, gradation conversion, and edge enhancement) on the image signal output from the size change 'full color processing unit 152.
- An image display unit 174 that displays an image in accordance with the image signal output from the video signal processing unit 172, and an image recording unit 176 that records an image in accordance with the image signal output from the video signal processing unit 172 You.
- the photoelectric conversion element 122 has a Bayer array color filter array, and acquires RGB data at each position. In the addition or averaging sampling, data of the R channel and the G channel of a plurality of pixels are simultaneously obtained in a line where the R-G channels are adjacent in the horizontal direction. In the line where G'B channel is adjacent, data of G channel and B channel of multiple pixels are simultaneously obtained
- FIG. 2 shows an example of an averaging reading (averaging sampling) mode.
- the two images of the G channel are acquired (addition or averaging).
- data of the same color (R_R, G_G, B-B) located at every other pixel in the horizontal direction is read out at the same time to obtain average data of a plurality of pixels.
- FIG. 2 is a schematic diagram corresponding to a 4Z3 magnification process.
- P101 and P108 indicate positions where two pixels of G channel are read out on the image sensor.
- P101 and P108 indicate a pixel group including two pixels that are added or averaged and simultaneously read.
- G101 G108 be the sampling data.
- a position corresponding to the center of gravity of the position of the pixel included in the photoelectric conversion data is defined as a sampling position.
- the pixel arrangement position of the full-color data is determined according to the image magnification set by the image magnification selecting section 132.
- P301 to P308 indicate the grid positions (grid arrangement) of full-color data in consideration of scaling.
- a weighting factor for each averaged sampling data is set according to the position of the center of gravity of each averaged sampling and the pixel array position of the full-color data, and full-color data is generated by weight addition.
- a position P201- P208 (indicated by ⁇ ) of full-color data having the same size as the averaging sampling is set.
- the data of G-chan talent at the positions of # 202, # 203, # 206, and # 207 are G202, G203, G206, and G207, respectively.
- G202 0.5196 G101 + 0.1039 G102 + 0.2354-G103 + 0.1414 G104
- G203 0.1414 G101 + 0.2354 G102 + 0.1039 G103 + 0.5196 G104
- G206 0.5196G105 + 0.1039 G106 + 0.2354 G107 + 0.1414 G108
- G207 0.1414-G105 + 0.2354 G106 + 0.1039 G107 + 0.5196 G108 [0029]
- P201-208 are distributed on a grid at equal intervals in the horizontal and vertical directions.
- the grid points become 3/4, so the position P306 of the full-color image after resizing is calculated using the bilinear interpolation of G202, G203, G206, and G207. It can be expressed as follows.
- G306 0.25 (0.25 .G202 + 0.75 .G203) + 0.75 ⁇ (0.25 .G206 + 0.75 'G207)
- the bilinear interpolation of the G channel in the k-th row and the first column as the G301 reference point can be expressed as follows.
- [x] represents the fractional part of the rational number x.
- G -G -G -G is P
- the relationship between the positions of the centers of gravity of the sampling is unequally spaced, but has a repetitive structure.
- the relative position of the center of gravity of the R'G channel on which the averaging sampling was performed is such that the distance between adjacent ones is one pixel and the distance between them is three pixels repeatedly.
- the first-order transformation shown in 1) is cyclic, and the number of coefficients used is four.
- the look-up table unit 154 stores a weighting coefficient corresponding to the image scaling factor obtained from Expressions (1) and (3), and supplies this to the size change / full-color processing unit 152. .
- the size change processing unit 152 performs weight addition to generate full color data whose size has been changed.
- the size change 'full-color processing unit 152 The processing of enlargement complementation and full colorization is performed by the data processing.
- the output signal from the resizing / full-color conversion processing unit 152 is subjected to gradation conversion, color balance, color enhancement, and edge enhancement processing in the video signal processing unit 172.
- the signal processed by the video signal processing unit 172 is displayed on the image display unit 174, and is recorded by the image recording unit 176 as necessary.
- FIG. 4 shows the configuration of the imaging device according to the second embodiment of the present invention.
- elements indicated by the same reference numerals as those of the imaging device 100 of the first embodiment are equivalent elements, and detailed description thereof will be omitted in the description that follows to avoid duplication.
- the imaging device 200 includes an imaging optical system 110 that forms an optical image of a subject, and an imaging device 220 that outputs an image signal of a predetermined region of the optical image formed by the imaging optical system 110.
- the imaging device 220 includes an area-shaped photoelectric conversion element (imaging element) 222 that photoelectrically converts an optical image formed by the imaging optical system 110 to obtain digital image data (a set of pixel data), A read control unit 224 that reads out the image data acquired by the element 222 according to the supplied read rule.
- the photoelectric conversion element 222 includes, for example, a color filter array having a Bayer array, and acquires RGB data of an optical image formed by the imaging optical system 110.
- the imaging apparatus 200 further includes an image scaling ratio selection unit 132 that selects a scaling factor of an image to be output, and a reading rule setting that selects a reading rule of the photoelectric conversion element 222 according to the selected image scaling factor. It has a part 240.
- the reading rule setting unit 240 provides data relating to the reading rule to the reading control unit 224 that controls reading from the photoelectric conversion element 222 according to the selected image scaling factor.
- the imaging apparatus 200 further includes a distortion correction / full-color processing section 252, a filter coefficient selection section 254, and a look-up table section 256.
- the distortion correction / full color processing unit 252 performs distortion correction and full color processing on the digital image data read from the photoelectric conversion element 222 by filter processing.
- the look-up tape storage unit 256 stores a plurality of filters used for the filtering process in the distortion correction / full-color processing unit 252 as a look-up table.
- the filter coefficient selection unit 254 selects an appropriate filter coefficient according to the image magnification selected by the image magnification selection unit 132, and supplies the selected filter coefficient from the lookup table unit 256 to the distortion correction / full-color processing unit 252. .
- the imaging device 200 further includes a video signal processing unit 172 that performs predetermined processing (such as white balance, gradation conversion, and edge enhancement) on the output signal from the distortion correction / full-color conversion processing unit 252;
- a video signal processing unit 172 that performs predetermined processing (such as white balance, gradation conversion, and edge enhancement) on the output signal from the distortion correction / full-color conversion processing unit 252;
- An image display unit 174 displays an image according to an output signal from the processing unit 172, and an image recording unit 176 records an image according to an output signal from the video signal processing unit 172.
- the reading rule setting unit 240 provides the reading control unit 224 with a thinning reading rule for an image scaling ratio of less than 100%, that is, for image reduction. For example, in a 75% reduction process, the read control unit 224 is given a rule of reading out six pixels out of eight horizontal and vertical pixels. The read control unit 224 reads pixel data in a corresponding range in the pixel array in the photoelectric conversion element 222 based on the thinning read rule provided from the read rule setting unit 240. As a result, the thinned image data is output from the imaging device 220.
- the data of the photoelectric conversion element 222 read according to such a thinning-out reading rule is supplied to the distortion correction / full-color processing unit 252.
- the data of the reading rule setting unit 240 is provided to the filter coefficient selecting unit 254, and the filter coefficient selecting unit 254 provides the filter coefficient corresponding to the reading rule to the distortion correction / full color processing unit 252. .
- the filter coefficient may be calculated by an operation according to the readout rule set by 40. Filter coefficient selection using the LUT requires a lot of memory to store the LUT, but requires less computational burden. On the other hand, filter-one-coefficient selection without using a LUT requires a large amount of computation, but does not require a large amount of memory.
- the photoelectric conversion element 222 is, for example, a CMOS or a CCD, and can perform a thinning-out reading operation, and can thinly read image data at an arbitrary position. With the thinning-out read operation, pixels in a specific area can be read with a smaller number of pixels than when all pixels are read.
- the imaging device 120 when the photoelectric conversion element 222 is a CMOS, the imaging device 120 is a CMOS, the imaging device 120
- the i-th element in the j-th line is defined as C (i, j), and the pixels in the horizontal direction are represented by
- the photoelectric conversion element 222 is a CCD
- the CCD reads out while shifting charges in the horizontal direction. Therefore, the imaging device 120 reads out all pixels in the horizontal direction and thins out pixels in the vertical direction. Can be read.
- the distortion correction 'full-color processing section 252 complements missing information, performs magnification conversion, and performs color complementation on the digital image data thus decimated and read.
- the photoelectric conversion element 222 includes a color filter array in a Bayer array, and acquires RGB data at each position. Further, an image data acquisition area on the image sensor is set in accordance with the image scaling ratio selection unit 132, and pixels of the image data acquired by the photoelectric conversion element are thinned out and read as necessary. The position of the image data read in this way The position is set in advance. On the other hand, the pixel array position of the generated full-color data is determined according to the image magnification set by the image magnification selection unit 132.
- the look-up table unit 256 stores a weighting coefficient of the pixel data of the thinning-out reading according to the image scaling factor in accordance with the positional relationship between the pixel position of the thinning-out reading and the pixel arrangement position of the full-color data.
- FIG. 5 shows a thinning-out reading rule corresponding to the scaling of 75% reduction.
- the readout rule shown in Fig. 5 is repeated on an image sensor with a CFA (color filter array).
- Figure 6 shows how the data read out by thinning out 2 pixels out of 8 pixels in the horizontal direction on a part of the R'G line is converted to equally-spaced pixel data with R and G color information of 6 pixels. This is shown schematically.
- the transformation shown in Figure 6 can be expressed in matrix form as:
- Rc ⁇ , Gc f represents the respective color data of the i th pixel, which is converted at regular intervals.
- Fig. 6 based on the position of R, eight pixels are regarded as one block, and the pixels of the left block are denoted by G, and the pixels of the right block are denoted by R.
- R'G as above If correction of thinning distortion is performed on a line, pixels having R'G data will be arranged at equal intervals in the horizontal direction in the corresponding line, and the R'G line and the corrected R'G line will be aligned in the vertical direction. G'B lines alternate.
- FIG. 7 schematically shows the process of vertical distortion correction and full-color conversion.
- two of the eight lines are read out by thinning them out, and this data is converted into six lines of full-color data at equal intervals in the vertical direction.
- G-channel data is present in all the lines that are being read. To data.
- the matrix representation of this transformation can be expressed as:
- R'B has data only on either the even-numbered lines or the odd-numbered lines, so it can be expressed as follows using the same conversion coefficients used for horizontal distortion correction and color interpolation.
- Gc, Rc, and Be represent G, R, and B data on the 0th line, respectively. Less than
- Gc, Rc, and Be represent the data of G, R, and B on the i-th line, respectively.
- FIG. 8 to FIG. 13 schematically show a configuration of a pipeline for performing the processing of distortion correction and color complementation shown in equation (4).
- the image signal 600 is on the R'G line, and in the order of reading from right to left, R, G, R, G, R
- FIG. 8 to FIG. 13 show the state of each clock unit.
- This pipeline includes a 5-tap shift register 611, a selector 622 to which the data of the first and third registers of the shift register 611 are input, and a data of the third and fifth registers of the shift register 611. 621, a multiplier 631 that multiplies the output data of the selector 622 by a coefficient of k4, a multiplier 632 that multiplies the output data of the selector 621 by a coefficient of k3, and a multiplier.
- a calorie calculator 641 for calculating the calorie of the output data of 631 and the output data of the multiplier 632, a multiplier 633 for multiplying the data of the second register of the shift register 611 by a coefficient of k2, A multiplier 634 that multiplies the data of the fourth register by the coefficient of kl, and the output data of the multiplier 633 and the output of the multiplier 634 And an adder 642 for adding the force data.
- Both the selector 621 and the selector 622 have an input inl and an input in2, and selectively output one of the data input to the input inl and the input in2.
- the states of the selector 621 and the selector 622 can be changed in each clock unit, and are switched in conjunction with each other. In FIG. 8 to FIG. 13, the selector 621 and the selector 622 output the data input to the input inl and the input in2 on the side indicated by the ellipse.
- FIG. 8 shows a state in which data of five pixels R, G, R, G, and R is being read.
- Multiplier 631 multiplies R by 1Z2, multiplier 632 multiplies R by 1/2, and adder 641
- Multiplier 633 multiplies G by 1, and multiplier 634 generates G
- Multiply 17 by 0 and adder 642 adds them and outputs.
- FIG. 9 shows a state after a lapse of one clock unit from the state of FIG. Therefore, the data input to the shift register 611 is updated by one clock unit, and G, R, G, and R are updated.
- FIGS. 10 to 13 show states after a lapse of one clock unit with respect to the immediately preceding state (the state of the figure with the figure number being one smaller). The same operation is performed in each of the states shown in FIGS.
- the states shown in FIGS. 8 to 13 correspond to the operations for generating Rc, Gc,..., Rc, and Gc in equation (4).
- the load addition process is performed for each clock, it can be obtained by the output power pipeline process corresponding to the left side of equation (4).
- the color channels output by the adder are alternated every clock unit. Therefore, when storing time-series data in memory for each channel, a selector should be placed after the adder. Table 1 shows the transition states of such pipeline processing.
- Equation (6) performs distortion correction and color complementation on the data of the R′B channel complemented by each line in the vertical direction, and the processing configuration is the same as in FIG. 8 and FIG. What is necessary is just to add a line memory to the object so that vertical complementation processing can be performed.
- This pipeline includes a 3-tap shift register 811, a selector 822 to which the data of the first and second registers of the shift register 81 1 are inputted, and a data of the second and third registers of the shift register 61 1.
- An adder 841 for adding the output data and the output data of the multiplier 832 is provided.
- FIG. 15 schematically shows a state in which only the distortion correction processing is performed in the horizontal direction of the R′G line to generate an equally-spaced Bayer array.
- Such a conversion can be expressed as follows by using a matrix notation similar to that of Equation (4).
- the two G signals are (Gc + Gc) Z2. This means that the original read signals G, G, and G
- Equation (4) shows the case where the distortion correction processing and the color interpolation processing are performed simultaneously, and Equation (10) shows the case where the distortion correction processing and the color interpolation processing are performed separately.
- Equation (10) shows the case where the distortion correction processing and the color interpolation processing are performed separately.
- the coefficient in one row of the matrix indicates the weighting coefficient of the first-order complement for generating one corresponding pixel data.The larger the coefficient, the greater the contribution of distant pixels, so the spatial band loss It means there is.
- Equation (4) has a smaller coefficient spread than equation (10), so it can be seen that the band loss before performing color complementation is smaller.
- a high-definition image can be obtained by simultaneously performing the distortion correction and full-color processing in consideration of the positions of the pixels for which color complementation and full-color conversion are performed and the sampled pixel positions.
- FIG. 16 shows the configuration of the imaging device according to the third embodiment of the present invention.
- elements indicated by the same reference numerals as those of the imaging device 200 of the second embodiment are equivalent elements, and detailed description thereof will be omitted in the description that follows to avoid duplication.
- the imaging device 300 continuously converts an image forming optical system 110 that forms an optical image of a subject and an image signal of a predetermined region of the optical image formed by the image forming optical system 110 into an image. And an imaging device 320 for outputting. That is, the image signal output from the imaging device 320 is a moving image signal, which is composed of image data of a plurality of frames that are continuous in time series.
- the imaging device 320 includes an area-shaped photoelectric conversion element 322 that photoelectrically converts an optical image formed by the imaging optical system 110 to obtain digital image data (a set of pixel data); And a read control unit 324 for thinning out the image data acquired in 322 as necessary and reading out continuously.
- the photoelectric conversion element 322 includes a Bayer array color filter array, and acquires RGB data of an optical image formed by the imaging optical system 110.
- the imaging device 300 further includes an image scaling ratio selection unit 132 that selects a scaling ratio of an image to be output, a reading rule setting unit 340 that sets a pixel thinning-out reading rule of a reading control unit 324, and a reading device that reads from the imaging device 320 And a full-color processing section 252 for performing distortion correction and full colorization processing on the output image data.
- the details of the distortion correction / full-color processing unit 252 are as described in the second embodiment.
- the imaging device 300 further includes a readout phase control unit 336 that changes a reference position of a range (readout range) of image data read out from the photoelectric conversion element 322 by the readout control unit 324 for each frame, and a readout phase control unit 336. Based on the reference position of the readout range set for each frame, the image data output from the distortion correction And an image range selection processing section 360 for selecting a range common to all frames of the image data.
- a readout phase control unit 336 that changes a reference position of a range (readout range) of image data read out from the photoelectric conversion element 322 by the readout control unit 324 for each frame
- a readout phase control unit 336 Based on the reference position of the readout range set for each frame, the image data output from the distortion correction And an image range selection processing section 360 for selecting a range common to all frames of the image data.
- the reading rule setting unit 340 In response to the reading phase control unit 336 changing the reference position of the reading range for each frame and the image range selection processing unit 360 selecting a range common to all frames, the reading rule setting unit 340 The pixel thinning-out readout rule is set so that the readout control unit 324 reads out image data over a wider range than the image area set by the image scaling ratio selection unit 132.
- the read control unit 324 in the imaging device 320 performs photoelectric conversion based on the read rule set by the read rule setting unit 340 and the reference position of the read range set by the read phase control unit 336.
- the image data (pixel data for one frame) in the corresponding range in the pixel array in 322 is continuously read.
- the imaging device 320 outputs a moving image signal composed of image data of a plurality of frames that are continuous in time series.
- the imaging device 300 further includes three cyclic frame memories 362, 364, and 366 that temporarily store image data of a plurality of frames, and a plurality of frames stored in the frame memories 362, 364, and 366. And an inter-frame arithmetic processing unit 368 that performs arithmetic processing on this image data to generate new image data.
- the thinning position for each frame For example, in a 75% reduction process, as shown in Fig. 5, the power to read out 6 pixels out of 8 pixels, the readout position of 6 pixels is made different between frames, and distortion correction and full correction in each frame are performed. If addition processing is performed between frames after the colorization processing, a smooth image can be obtained. It is effective to change the reading rule for each frame as described above, but since the reading rule is different for each frame, it is necessary to make the coefficient of the distortion correction process different for each frame. This leads to an increase in the size of the hardware.
- the readout rule is the same for all frames, but the readout range is shifted between two consecutive frames.
- the imaging device 300 of the present embodiment complements each other's missing pixel data in two consecutive frames.
- the readout phase control unit 336 changes the reference position of the pixel data range (readout range) read out by the readout control unit 324 by thinning out the pixels of the photoelectric conversion element 322 for each frame. More specifically, the readout phase control unit 336 periodically changes the reference position of the readout range for each frame according to a certain rule.
- the readout phase control unit 336 sets the reference position of the range of the pixel data read out by the readout control unit 324 so that the image data of a plurality of continuous frames does not have the missing pixel data as a whole. Shift. More preferably, the readout phase control unit 336 shifts the reference position of the readout range so that image data of two consecutive frames does not have pixel data that is commonly missing. The shift amount is preferably about 418 pixels.
- inter-frame arithmetic processing unit 368 performs a process of complementing missing pixel data with respect to image data of continuous frames stored in the frame memories 362, 364, and 366. For example, 1/2: 1/2 addition is performed on image data of two consecutive frames. Alternatively, 1Z4: 1Z2: 1/4 may be added to image data of three consecutive frames.
- FIG. 17 and FIG. 18 show the reading in the reading by repeating the 6/8 thinning-out reading.
- the shift of the reference position in the range is schematically shown.
- [X, y] represents a pixel position in the pixel array of the photoelectric conversion element 322, and (x, y) represents a pixel data array in the readout range.
- FIG. 17 shows one of two temporally adjacent frames (hereinafter, referred to as an A frame), and FIG. 18 shows the other frame (hereinafter, referred to as a B frame). I have.
- the number of pixels of the photoelectric conversion element 322 is k pixels in the horizontal direction and 1 pixel in the vertical direction. Therefore, the position of the upper left pixel of the photoelectric conversion element 322 can be expressed as [0, 0], and the position of the lower right pixel can be expressed as [k, 1].
- the number of pixels in the readout range of one frame is m pixels in the horizontal direction and n pixels in the vertical direction. Therefore, the reading start position at the upper left of the frame can be expressed as (0, 0), and the reading end position at the lower right can be expressed as (m, n).
- the reading range of the B frame in FIG. 18 is shifted by +2 pixels in the horizontal direction and +2 pixels in the vertical direction with respect to the reading range of the A frame in FIG.
- the image range selection processing unit 360 selects a range common to the A frame in FIG. 17 and the B frame in FIG. In other words, the image range selection processing unit 360 selects a rectangular range having the diagonal vertices of (2, 2) and (m, n) for the frame A in FIG. For, a rectangular range with diagonal vertices from (0, 0) to (m-2, n-2) is selected.
- the range selected by the image range selection processing unit 360 always has (m-2) X (n-2) pixel data.
- the imaging device according to the present embodiment has the same advantages as those of the imaging device according to the second embodiment.
- the imaging apparatus according to the present embodiment has a frame By reading out the image data by changing the readout rule every time, it is possible to avoid a loss of pixel data at a specific position in the photoelectric conversion element. Thereby, the imaging device of the present embodiment can form a high-definition moving image with little distortion.
- the present embodiment is directed to a method for converting image data according to a scaling factor from averaged and sampled data.
- FIG. 19 shows a procedure of a method of converting image data according to the magnification from the averaged sampled data.
- the original image data assumes a Bayer array.
- S191 two horizontal pixels are averaged and assigned to one pixel.
- R represents the image data after the averaging process
- r represents the original pixel data.
- the subscripts are as follows in order to explicitly represent the pixel positions after averaging.
- the position of the center of gravity of the pixel can be expressed as follows in consideration of the sampling position of the original data shown in Expression (11).
- R CO g (i, j) Cog (r (i, j), r (i, j + 2)) ⁇ ⁇ ⁇ (13)
- R (i, j) represents the position of the center of gravity of pixel R (i, j), and Cog (X, Y) calculates the center of gravity of X and Y.
- the actual position may be obtained from the subscript (i, j + 1) in consideration of the pitch of the actual pixel array.
- the averaging process is not limited to the horizontal two-pixel averaging process shown in Expression (11), but may be a horizontal or vertical averaging process.
- the grid position of the full-color data is calculated according to the scaling factor. As shown in Fig. 2, the grid position of the full-color data has a relative relationship to the sampling position of the original data, and the grid position of the full-color data of the output image and the pixel data used for the weighted average Distance L can be calculated.
- the load coefficient W is inversely proportional to the distance L, and is calculated so that the sum of the weighted averages becomes 1.
- the load addition processing for calculating the output image data is expressed in a matrix format.
- W is a weight addition coefficient
- R is a matrix of pixel data. Assuming Bayer array image data, as shown in equation (11), the weighted addition coefficient in equation (13) is different for each channel since the center of gravity when averaged sampling is different for each of the RGB channels. become.
- the present embodiment is directed to an imaging apparatus that performs a thinning process in advance according to a scaling factor of an image size as needed, and performs a distortion correction and a full color process.
- FIG. 20 and FIG. 25 show that, in the fifth embodiment of the present invention, when certain image data (Bayer array) is converted into full-color image data of different sizes, the image data (number of pixels) is adjusted in advance in accordance with the scaling factor. It shows the procedure for performing thinning processing, and performing distortion correction and full color processing.
- FIG. 20 shows the overall processing flow.
- S202 horizontal distortion correction processing and two-channel colorization are performed in accordance with the input image data (after the thinning processing) and the magnification.
- S203 vertical processing and full color conversion are performed.
- Two-channel colorization means for example, in the case of a Bayer array, there are horizontal lines in line with R'G.R'G and lines in parallel with G.B'G'B. Then, as shown in Figure 6- Figure 13, R'G two-channel data is generated at the same pixel position.
- RG (i, j) corresponds to a state in which R′G has two channels at the same pixel position.
- the output G of the detector 642 is collectively represented as RG (ij).
- Figure 22 shows the details of the horizontal distortion correction and colorization processing.
- the image data that is continuous in the horizontal direction is stored in a FIFO (First In First Out) register. This corresponds to the 5-tap register shown in FIGS.
- the input image data has been thinned out in advance and has luminance data and pixel position data.
- the LUT indicates the number of repetition pixels of the complementing process determined according to the scaling factor, the load addition coefficient, and the position of the selector. For the operation of the selector, refer to FIGS.
- the repeating pixel unit is the same as that shown in FIGS.
- the relative position of the pixel is obtained from the data of the position of the input image and the repeated pixel unit.
- the repetition pixel unit corresponds to an interval between pixels using the same complement coefficient and selector state when performing the variable magnification complement process.
- the state of the selector is obtained from the repetition pixel unit and the pixel position of the input image.
- the configuration is such that the pixel unit of the repetition, the weight addition coefficient, and the state of the selector are calculated in advance and can be referred to by the LUT.
- Fig. 24 shows the procedure for calculating the contents of such an LUT.
- the unit of repetition is calculated based on the scaling factor and the position of the pixel thinning-out, based on the scaling factor and the pixel thinning-out position. Since the state of the data is obtained, the load addition coefficient as shown in the equation (4) can be obtained.
- the coefficient (state) of the selector is referred from the relative position of the pixel obtained in S222.
- the weight coefficient is referred to from the relative position of the pixel.
- the read position of the register held in step S221 is determined from the selector coefficient obtained in step S221, and the weighted addition process is performed using the weighted addition coefficient obtained in step S224.
- FIG. 23 shows details of the vertical distortion correction processing.
- the image data continuous in the vertical direction is stored in the vertical shift register.
- the relative position of the pixel is obtained from the data of the position of the input image and the repeated pixel unit.
- FIG. 25 shows a procedure for calculating the contents of the LUT in FIG.
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Physics & Mathematics (AREA)
- Spectroscopy & Molecular Physics (AREA)
- Color Television Image Signal Generators (AREA)
- Transforming Light Signals Into Electric Signals (AREA)
Abstract
Description
Claims
Priority Applications (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US11/366,734 US20060158537A1 (en) | 2003-09-04 | 2006-03-02 | Imaging apparatus |
US11/745,302 US7948543B2 (en) | 2003-09-04 | 2007-05-07 | Imaging apparatus provided with image scaling function and image data thinning-out readout function |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2003312722A JP3942569B2 (ja) | 2003-09-04 | 2003-09-04 | 撮像装置および画像データの変換方法 |
JP2003-312722 | 2003-09-04 |
Related Child Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US11/366,734 Continuation US20060158537A1 (en) | 2003-09-04 | 2006-03-02 | Imaging apparatus |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2005025232A1 true WO2005025232A1 (ja) | 2005-03-17 |
Family
ID=34269747
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/JP2004/012636 WO2005025232A1 (ja) | 2003-09-04 | 2004-09-01 | 撮像装置 |
Country Status (3)
Country | Link |
---|---|
US (2) | US20060158537A1 (ja) |
JP (1) | JP3942569B2 (ja) |
WO (1) | WO2005025232A1 (ja) |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2007110829A2 (en) | 2006-03-27 | 2007-10-04 | Nxp B.V. | Method and arrangement for generating a color video signal |
US20100303349A1 (en) * | 2009-05-28 | 2010-12-02 | Bechtel Jon H | Digital image processing and systems incorporating the same |
Families Citing this family (19)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
KR100657145B1 (ko) * | 2005-02-24 | 2006-12-13 | 매그나칩 반도체 유한회사 | 스케일러를 구비한 이미지센서 및 이미지센서의 이미지 스케일링 방법 |
JP4759293B2 (ja) * | 2005-03-15 | 2011-08-31 | キヤノン株式会社 | 撮像素子 |
JP2007124295A (ja) * | 2005-10-28 | 2007-05-17 | Pentax Corp | 撮像手段駆動装置、撮像手段駆動方法、および信号処理装置 |
US20080260291A1 (en) * | 2007-04-17 | 2008-10-23 | Nokia Corporation | Image downscaling by binning |
GB0709026D0 (en) * | 2007-05-10 | 2007-06-20 | Isis Innovation | High speed imaging with slow scan cameras using pixel level dynami shuttering |
US20090077359A1 (en) * | 2007-09-18 | 2009-03-19 | Hari Chakravarthula | Architecture re-utilizing computational blocks for processing of heterogeneous data streams |
JP5342796B2 (ja) * | 2008-03-27 | 2013-11-13 | 株式会社東芝 | 三次元画像表示方法および装置 |
WO2010032720A1 (ja) * | 2008-09-19 | 2010-03-25 | コニカミノルタホールディングス株式会社 | 画像歪み補正方法及び画像処理装置 |
EP2175632A1 (en) * | 2008-10-10 | 2010-04-14 | Samsung Electronics Co., Ltd. | Image processing apparatus and method |
JP5241429B2 (ja) | 2008-10-24 | 2013-07-17 | キヤノン株式会社 | 画像形成装置およびその制御方法 |
JP5254740B2 (ja) * | 2008-10-24 | 2013-08-07 | キヤノン株式会社 | 画像処理装置および画像処理方法 |
JP5254739B2 (ja) | 2008-10-24 | 2013-08-07 | キヤノン株式会社 | 画像形成装置およびその制御方法 |
US8565554B2 (en) * | 2010-01-09 | 2013-10-22 | Microsoft Corporation | Resizing of digital images |
JP2012175600A (ja) | 2011-02-24 | 2012-09-10 | Sony Corp | 撮像装置、および撮像装置制御方法、並びにプログラム |
US9769430B1 (en) | 2011-06-23 | 2017-09-19 | Gentex Corporation | Imager system with median filter and method thereof |
US9041838B2 (en) | 2012-02-14 | 2015-05-26 | Gentex Corporation | High dynamic range imager system |
JP6127368B2 (ja) * | 2012-03-15 | 2017-05-17 | 株式会社ニコン | 撮像装置 |
JP6872956B2 (ja) * | 2017-04-13 | 2021-05-19 | キヤノン株式会社 | 撮像システム及び撮像システムの制御方法 |
CN114422766B (zh) * | 2018-08-03 | 2024-06-04 | 杭州海康威视数字技术股份有限公司 | 一种图像采集设备 |
Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2001086522A (ja) * | 1999-08-20 | 2001-03-30 | Neucore Technol Inc | 画像処理装置 |
JP2002084547A (ja) * | 2000-09-06 | 2002-03-22 | Nikon Corp | 画像データサイズ変換処理装置、電子スチルカメラ、および画像データサイズ変換処理用記録媒体 |
Family Cites Families (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP4047943B2 (ja) * | 1995-01-31 | 2008-02-13 | 富士フイルム株式会社 | ディジタル画像データ記録装置および方法ならびにディジタル画像データ再生装置および方法ならびに固体電子撮像装置およびその信号電荷読出し方法 |
JP3392564B2 (ja) * | 1995-02-27 | 2003-03-31 | 三洋電機株式会社 | 単板式カラービデオカメラ |
JP3901878B2 (ja) | 1999-07-02 | 2007-04-04 | 株式会社リコー | 画像処理装置 |
JP4136255B2 (ja) | 2000-02-29 | 2008-08-20 | キヤノン株式会社 | 画像処理装置及びその方法 |
JP2003087514A (ja) * | 2000-11-15 | 2003-03-20 | Nippon Sheet Glass Co Ltd | 画像読取り装置、その光源およびシェーディング補正装置 |
JP4307780B2 (ja) * | 2002-03-07 | 2009-08-05 | 富士フイルム株式会社 | 固体撮像装置およびその信号読出し方法 |
US7120297B2 (en) * | 2002-04-25 | 2006-10-10 | Microsoft Corporation | Segmented layered image system |
-
2003
- 2003-09-04 JP JP2003312722A patent/JP3942569B2/ja not_active Expired - Fee Related
-
2004
- 2004-09-01 WO PCT/JP2004/012636 patent/WO2005025232A1/ja active Application Filing
-
2006
- 2006-03-02 US US11/366,734 patent/US20060158537A1/en not_active Abandoned
-
2007
- 2007-05-07 US US11/745,302 patent/US7948543B2/en not_active Expired - Fee Related
Patent Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2001086522A (ja) * | 1999-08-20 | 2001-03-30 | Neucore Technol Inc | 画像処理装置 |
JP2002084547A (ja) * | 2000-09-06 | 2002-03-22 | Nikon Corp | 画像データサイズ変換処理装置、電子スチルカメラ、および画像データサイズ変換処理用記録媒体 |
Cited By (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2007110829A2 (en) | 2006-03-27 | 2007-10-04 | Nxp B.V. | Method and arrangement for generating a color video signal |
WO2007110829A3 (en) * | 2006-03-27 | 2007-12-06 | Koninkl Philips Electronics Nv | Method and arrangement for generating a color video signal |
US20100303349A1 (en) * | 2009-05-28 | 2010-12-02 | Bechtel Jon H | Digital image processing and systems incorporating the same |
US8463035B2 (en) * | 2009-05-28 | 2013-06-11 | Gentex Corporation | Digital image processing for calculating a missing color value |
Also Published As
Publication number | Publication date |
---|---|
US20070206207A1 (en) | 2007-09-06 |
US7948543B2 (en) | 2011-05-24 |
US20060158537A1 (en) | 2006-07-20 |
JP3942569B2 (ja) | 2007-07-11 |
JP2005086247A (ja) | 2005-03-31 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US7948543B2 (en) | Imaging apparatus provided with image scaling function and image data thinning-out readout function | |
JP4555775B2 (ja) | 撮像装置 | |
JP4142340B2 (ja) | 撮像装置 | |
US8040385B2 (en) | Image pickup apparatus | |
JP3771054B2 (ja) | 画像処理装置及び画像処理方法 | |
JP4445870B2 (ja) | 撮像装置 | |
JP4285948B2 (ja) | 撮像装置 | |
US6917383B2 (en) | Image sensing device | |
JP2004193911A (ja) | 画像処理装置および画像処理方法、デジタルスチルカメラ | |
KR100719988B1 (ko) | 화상 신호 처리 장치 | |
JP2006262382A (ja) | 画像処理装置 | |
JP4412446B2 (ja) | 解像度変換方法および解像度変換装置 | |
JP2920380B2 (ja) | 色信号処理装置 | |
JP3767338B2 (ja) | 撮像装置 | |
JP3651477B2 (ja) | 画像信号処理装置 | |
JP2006054583A (ja) | 撮像装置 | |
JP3551670B2 (ja) | 電子スチルカメラ | |
JP4478934B2 (ja) | 画像信号処理装置 | |
JP2012099870A (ja) | 撮像装置及び撮像方法 | |
JP2002095000A (ja) | 撮像装置 |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AK | Designated states |
Kind code of ref document: A1 Designated state(s): AE AG AL AM AT AU AZ BA BB BG BR BW BY BZ CA CH CN CO CR CU CZ DE DK DM DZ EC EE EG ES FI GB GD GE GH GM HR HU ID IL IN IS KE KG KP KR KZ LC LK LR LS LT LU LV MA MD MG MK MN MW MX MZ NA NI NO NZ OM PG PH PL PT RO RU SC SD SE SG SK SL SY TJ TM TN TR TT TZ UA UG US UZ VC VN YU ZA ZM ZW |
|
AL | Designated countries for regional patents |
Kind code of ref document: A1 Designated state(s): BW GH GM KE LS MW MZ NA SD SL SZ TZ UG ZM ZW AM AZ BY KG KZ MD RU TJ TM AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HU IE IT LU MC NL PL PT RO SE SI SK TR BF BJ CF CG CI CM GA GN GQ GW ML MR NE SN TD TG |
|
121 | Ep: the epo has been informed by wipo that ep was designated in this application | ||
WWE | Wipo information: entry into national phase |
Ref document number: 11366734 Country of ref document: US |
|
WWP | Wipo information: published in national office |
Ref document number: 11366734 Country of ref document: US |
|
122 | Ep: pct application non-entry in european phase |