JP5244533B2 - Imaging device - Google Patents

Description

  The present invention relates to an imaging apparatus applied to, for example, a digital still camera and a camcorder.

  In recent years, imaging devices represented by digital still cameras and camcorders include CCD (Charge Coupled Device) image sensors (hereinafter referred to as CCD) and CMOS (Complementary Metal Oxide Semiconductor) image sensors (hereinafter referred to as CMOS). A representative solid-state image sensor is mounted, but a subject image (hereinafter referred to as a photographed image) output from an effective pixel area of these solid-state image sensors has vertical stripes along the column direction of the solid-state image sensor. It is known that a fixed pattern noise (hereinafter referred to as vertical stripe noise) occurs. The vertical stripe noise may be expressed as V scratch in CCD, column noise in CMOS, or column FPN (Fixed Pattern Noise).

  As a method of removing such vertical stripe noise component, a method of storing the address of the defective line of the solid-state imaging device and removing the vertical stripe noise component of the CCD according to the address of the defective line is known (for example, Similarly, a method of removing a vertical stripe noise component in a CMOS is also known (see, for example, Patent Document 2).

  In addition, by utilizing the fact that vertical streak noise components are also generated in the area of the solid-state image sensor that is shielded from light, an image output from the light-shielded area of the solid-state image sensor (hereinafter referred to as a light-shielded image) is used. The vertical streak noise component is extracted by resetting the capacitor included in the image, and correlated double sampling (hereinafter referred to as CDS) is performed using the captured image, and the vertical streak noise component is removed from the captured image. How to do is known.

  When extracting vertical stripe noise components from a light-shielded image, an optical black region (hereinafter referred to as an OB region) in which a solid-state image sensor is subjected to a light-shielding process so as not to be exposed to light, for example, by an aluminum thin film or the like. A method of removing a vertical stripe noise component from a captured image by providing a vertical stripe noise component from the OB area in advance is known. Also, before shooting the subject, the shutter is closed in advance, a light-shielded image is taken in a situation where the solid-state image sensor is shielded from light, and the vertical streak noise component is extracted from the light-shielded image. A method for removing the streak noise component is also known.

Also, if random noise components due to pixel variations of solid-state image sensors are included in the light-shielded image, random noise components are extracted after averaging the random noise temporally using multiple shooting frames. However, there is a drawback that it takes time to extract the vertical streak noise component by averaging over time. In order to solve this problem, the conventional technology (for example, refer to Patent Document 3) extracts the vertical stripe noise component by quickly converging the filter calculation by changing the feedback coefficient of the IIR (Infinite Impulse Response) filter. How to do is shown.
JP 2006-148439 A JP 2006-135423 A JP 2006-25146 A

  However, although the method disclosed in Patent Document 3 has succeeded in reducing the time for extracting the vertical streak noise component by changing the feedback coefficient of the IIR filter in the middle of the filter calculation, However, since the filter calculation cannot be completely converged, the vertical stripe noise component with poor accuracy is removed from the photographed image, and the original noise removal effect cannot be obtained. Furthermore, the method of extracting vertical stripe noise components after temporally averaging random noise using a plurality of shooting frames requires time to average random noise, so that it is difficult to speed up continuous shooting. There was also a problem.

  The present invention has been made in view of the above-described points, and extracts a vertical stripe noise component generated in a solid-state imaging device with high accuracy at high speed from a light-shielded image, and removes a vertical stripe noise component from a captured image with high accuracy at high speed. An object is to provide an imaging device.

  The present invention provides a solid-state image sensor, a noise component extraction unit that extracts a noise component from light-shielded image data output from a light-shielded region of the solid-state image sensor, and a non-light-shielded region of the solid-state image sensor. A noise subtractor that performs a process of subtracting the noise component from the captured image data. The noise component extractor includes an initialization coefficient multiplier that multiplies the shading image data by an initialization coefficient, and a cumulative value. A storage unit for storing, a feedback coefficient multiplication unit for multiplying a cumulative value stored in the storage unit by a feedback coefficient, and an addition unit, and a value obtained by multiplying the shading image data by the initialization coefficient, The addition unit adds the accumulated value acquired from the storage unit, and executes the first mode in which the addition result is stored in the storage unit as a new accumulated value, whereby the initial value of the accumulated value is stored in the storage unit. The Next, the adding unit adds a value obtained by multiplying the accumulated value acquired from the storage unit by the feedback coefficient to the shading image data, and stores the addition result as a new accumulated value in the storage unit. The imaging device is characterized in that the second mode is executed.

  According to the present invention, in the first mode, the noise component extraction unit is configured such that a product of the number of lines of the shaded image data and the initialization coefficient is equal to a filter amplification factor calculated by the feedback coefficient. The cumulative value is initialized so that the image pickup apparatus has the following characteristics.

Further, in the present invention, the noise component extraction unit includes a horizontal average value calculation unit that calculates an average value in the horizontal direction of the light-shielded image data, and the average calculated by the horizontal average value calculation unit in the first mode. The imaging apparatus is characterized in that a value is multiplied by the initialization coefficient .

  In the present invention, the noise component extraction unit executes the first mode, omits the execution of the second mode, and the noise subtraction unit subtracts the noise component from the captured image data. It is an imaging device characterized by performing.

  According to the present invention, the noise component extraction unit executes the first mode for the shading image data, executes the second mode for the shading image data acquired thereafter, and the noise. The subtracting unit is an imaging apparatus that performs a process of subtracting the noise component from the captured image data.

  According to the present invention, the number of lines of the shading image data acquired after that used in the second mode is smaller than the number of lines of the shading image data used in the first mode. Device.

  According to the present invention, when the imaging apparatus performs initialization in consideration of the filter amplification factor of the IIR filter in the first mode and shifts to the second mode, the number of lines acquired from the light-shielded image is small. Even so, the vertical streak noise component can be extracted from the light-shielded image with high accuracy and at high speed, and by removing the vertical streak noise component from the captured image with high speed and accuracy, a good captured image can be obtained at high speed. it can.

  Next, embodiments of the present invention will be described in detail with reference to the drawings. FIG. 1 is a block diagram of an imaging apparatus according to the present embodiment. 1, the imaging apparatus includes an optical lens 101, a diaphragm 102, a shutter 103, a solid-state imaging device 104, a vertical stripe noise correction unit 105, a preprocessing unit 106, an image processing unit 107, and a display IF (InterFace). ) Unit 108, display panel 109, card IF (InterFace) unit 110, recording medium 111, data bus 112, memory controller 113, memory 114, and CPU (Central Processing Unit) 115.

  The optical lens 101 refracts incident light from a subject. The diaphragm 102 passes a part of the incident light and projects the incident light from the subject onto the solid-state image sensor 104. As a result, the subject image is formed on the solid-state image sensor 104. In addition, when the shutter 103 is open, incident light is projected onto the solid-state image sensor 104, but when the shutter 103 is closed, the incident light can be blocked and a light-shielded image can be created on the solid-state image sensor 104. Can do. The solid-state image sensor 104 is realized by, for example, a CCD or a CMOS.

  The vertical stripe noise correction unit 105 includes a vertical stripe noise subtraction unit 140 and a vertical stripe noise component extraction unit 150. The vertical streak noise component extraction unit 150 acquires a light-shielded image from the solid-state image sensor 104 and extracts a vertical streak noise component from the light-shielded image. The vertical streak noise subtraction unit 140 subtracts the extracted vertical streak noise component from the captured image, thereby removing the vertical streak noise component from the captured image.

  FIG. 2 shows the solid-state imaging device 104 in the case where the OB region is provided. From the OB area, a light-shielded image for extracting the vertical stripe noise component is output to the vertical stripe noise component extraction unit 150 for each line, and from the effective area 202, the shot image is a vertical stripe noise subtraction unit for each line. 140 is output. The OB area is not necessarily limited to the arrangement shown in FIG. 2. For example, the OB area is arranged on the effective area 202 (OB area 201), arranged below the effective area 202 (OB area 203), and the effective area. It may be arranged above and below 202 (OB region 201 and OB region 203). Therefore, the vertical streak noise component extraction unit 150 can extract the vertical streak noise component by using the light-shielded image output from the OB region 201, the OB region 203, or both of the OB regions.

  When the OB area is not provided in the solid-state imaging device 104, the CPU 115 closes the shutter 103 so that the effective area 202 where the incident light is shielded can be used as the OB area. When the shutter 103 is closed, the vertical streak noise component extraction unit 150 extracts a vertical streak noise component using the light-shielded image output for each line of pixels from the effective area 202 light-shielded by the shutter 103. can do. Further, if the CPU 115 opens the shutter 103 to project incident light onto the effective area 202, the vertical streak noise subtraction unit 140 can acquire a captured image for each line of pixels arranged from the effective area 202. .

  Therefore, when using the solid-state imaging device 104 provided with the OB area, the vertical streak noise subtraction unit 140 subtracts the extracted vertical streak noise component from the subject image based on the light-shielded image and the captured image that are simultaneously captured. Thus, the vertical stripe noise component is removed from the subject image. When using a light-shielded image output from the effective area 202 when the shutter 103 is closed, first, a light-shielded image is acquired from the effective area 202 shielded by the shutter 103, and then the shutter 103 is opened to display the effective area 202. Captured images from Thereby, the vertical stripe noise subtraction unit 140 subtracts the extracted vertical stripe noise component from the subject image, thereby removing the vertical stripe noise component from the subject image. Even when the solid-state imaging device 104 provided with the OB area is used, the light-shielded image may be acquired from the effective area 202 light-shielded by the shutter 103.

  Further, the vertical streak noise subtracting unit 140 may perform a process of subtracting the vertical streak noise component from the entire captured image, or may perform a process of subtracting the vertical streak noise component from a part of the captured image. good. For example, subtraction processing may be performed on coordinates (columns) where the level of the vertical streak noise component is large in the solid-state image sensor 104, or coordinates (columns) where the level of the vertical streak noise component is large in advance in the solid-state image sensor 104. And processing for subtracting the vertical streak noise component may be performed. In the case of CMOS, variations in characteristics in the column direction tend to occur in the entire column, so that the process of subtracting the vertical streak noise component from the entire column may be performed. In the case of a CCD, a vertical stripe noise component tends to be generated in a specific column (column) due to a defect in the vertical CCD. Therefore, a process of subtracting the vertical stripe noise component from only a specific column (column) is performed. It is also good.

  FIG. 3 shows the vertical stripe noise component extraction unit 150. The vertical stripe noise component 150 includes an initialization coefficient multiplier 151, a selector (hereinafter referred to as SEL) 152, a cumulative adder 153, a SEL 156, a feedback coefficient multiplier 155, a line buffer 154, and a correction coefficient multiplication. 157 and an operation mode switching unit 158. The operation mode switching unit 158 counts the number of lines of the input light-shielded image data, and uses the SEL 152 and the SEL 156, and the cumulative addition mode and the IIR filter mode, which are the operation modes of the vertical streak noise component extraction unit 150. And the correction mode. Details of the operation mode will be described later.

  The initialization coefficient multiplier 151 multiplies the shading image data acquired from the solid-state image sensor 104 by the initialization coefficient Kx. Here, the initialization coefficient Kx is a coefficient used as an initial value for filter calculation. The SEL 152 is a selector that switches the flow of light-shielded image data under the control of the operation mode switching unit 158. The cumulative adder 153 cumulatively adds the shaded image data and outputs it to the line buffer 154. Assume that the value 0 is stored in the line buffer 154 before starting the cumulative addition mode operation as an example. The line buffer 154 stores light-shielded image data cumulatively added by the cumulative adder 153. It should be noted that a plurality of line buffers 154 may be provided, and the shaded image data for a plurality of lines (columns) may be stored.

  The feedback coefficient multiplier 155 multiplies the cumulatively added shading image data stored in the line buffer 154 by the feedback coefficient Ky. The SEL 156 is a selector that switches the flow of image data based on the control of the operation mode switching unit 158. The correction coefficient multiplier 157 multiplies the cumulatively added shading image data stored in the line buffer 154 by the correction coefficient Kz.

  FIG. 4 shows the flow of shaded image data and captured image data in the vertical streak noise component extraction unit 150 shown in FIG. The operation modes of the vertical streak noise component extraction unit 150 include a cumulative addition mode, an IIR filter mode, and a correction mode. When shaded image data is input to the vertical streak noise component extraction unit 150, the vertical streak noise component extraction unit 150 first operates in the cumulative addition mode. FIG. 4A shows the flow of shaded image data in the cumulative addition mode. The SEL 152 and the SEL 156 switch the data flow of the vertical stripe noise component extraction unit 150 so as to execute the cumulative addition mode under the control of the operation mode switching unit 158.

  The shading image data input from the solid-state imaging device 104 to the vertical streak noise component extraction unit 150 is multiplied by the initialization coefficient Kx by the initialization coefficient multiplier 151 and input to the cumulative adder 153 via the SEL 152. The cumulative adder 153 adds the light-shielded image data obtained by multiplying the input light-shielded image data by the initialization coefficient Kx and the light-shielded image data accumulated from the line buffer 154 until the previous time, and accumulates this time. The added light shielding image data is stored in the line buffer 154.

  When the vertical streak noise component extraction unit 150 operates in the cumulative addition mode, the light-shielded image data Sn obtained by cumulatively adding n lines (n is an integer equal to or greater than 1) is expressed by [Equation 1]. Here, Dn is the light-shielded image data of the nth line, and Kx is an initialization coefficient.

  When the vertical streak noise component extraction unit 150 operates in the cumulative addition mode, the vertical streak noise component extraction unit 150 multiplies the input shading image data by the initialization coefficient Kx and repeats the cumulative addition to obtain [Equation 1]. The value indicated by Sn is stored in the line buffer 154.

  [Equation 2] shows the degree of amplification of the cumulative value with respect to the input data when the n lines of the light-shielded image are cumulatively added. That is, the degree of amplification of the accumulated value indicates how much the accumulated value of the light-shielded image data is amplified when n lines of input light-shielded image data are cumulatively added.

  Next, the operation mode switching unit 158 counts how many lines of light-shielded image data are input. When the count value reaches a predetermined number of lines, the SEL 152 and the SEL 156 are switched to switch the vertical stripes. The noise component extraction unit 150 is switched to the IIR filter mode.

  In the IIR filter mode, the vertical streak noise component extraction unit 150 operates as an IIR filter that multiplies the light-shielded image data stored in the line buffer 154 by the feedback coefficient Ky and adds this to the input light-shielded image data. The shaded image data stored in the line buffer 154 by the operation in the cumulative addition mode performed until the operation in the IIR filter mode is performed as the initial value of the line buffer 154 when the operation in the IIR filter mode is started. It shall be taken over as it is.

  FIG. 4B shows the flow of shaded image data in the IIR filter mode. The shaded image input to the vertical streak noise component extraction unit 150 is input to the cumulative adder 153 via the SEL 152. The cumulative adder 153 adds the input light-shielded image data to the light-shielded image data that has been cumulatively added up to the previous time multiplied by the feedback coefficient Ky by the feedback coefficient multiplier 155, and as the light-shielded image data that has been cumulatively added this time. The data is stored in the line buffer 154.

  When the vertical streak noise component extraction unit 150 operates in the IIR filter mode, the light-shielded image data Sn obtained by accumulating n lines is expressed by [Expression 3]. Here, Dn is the light-shielded image data of the nth line, and Ky is the feedback coefficient (0 <Ky <1).

When the vertical streak noise component extraction unit 150 operates in the IIR filter mode, the vertical streak noise component extraction unit 150 has been input by the previous time (n−1) when the shading image data of the nth line is input. The shaded image data Sn _{-1 obtained} by cumulatively adding the lines is multiplied by the feedback coefficient Ky, and the shaded image data Dn of the nth line input this time is added to this to operate as a vertical IIR filter. , [Equation 3] Sn is stored in the line buffer 154.

  [Equation 4] shows the filter amplification factor for the input light-shielded image data when the light-shielded image data of n lines is cumulatively added. Here, the filter amplification factor indicates how much the input shaded image data is amplified by the IIR filter.

  Further, when n is infinite, [Equation 4] matches [Equation 5]. For example, when the feedback coefficient Ky = 63/64, the filter amplification factor becomes 64 according to [Equation 5].

The (n−1) th power of the feedback coefficient Ky is referred to as a gain for the nth line when the shading image data is cumulatively added in the IIR filter mode. That is, as shown in FIG. 4B, [Equation 3], when the light-shielded image data Dn is input, the (n-1) lines accumulated in the line buffer 154 until the previous input are accumulated. The value S _n−1 is multiplied by the feedback coefficient Ky, and the currently input shading image data Dn and the multiplication result are added by the cumulative adder 153 and input to the line buffer 154 as Sn. The gain for the nth line when cumulatively adding in the mode is the (n-1) th power of the feedback coefficient Ky.

  Therefore, gain is weighting in filter calculation, and a line having a large gain has a large influence on noise component subtraction. FIG. 5 and FIG. 6 show the relationship between the number of cumulative addition lines and the gain when the cumulative addition mode is switched to the IIR filter operation mode. In FIGS. 5 and 6, the cumulative number of added lines is the number of lines added by the cumulative adder 153 in the cumulative addition mode and the IIR filter mode executed next to the cumulative addition mode.

  As an example, the calculation conditions shown in FIG. 5F are the initialization coefficient Kx = 1, the feedback coefficient Ky = 63/64, the number of initialization lines = (filter amplification factor) / (initialization coefficient Kx) = (1 / The relationship between the cumulative number of added lines (horizontal axis) and the gain (vertical axis) when (1-Ky)) / Kx = 64 is shown in FIGS. Here, the number of initialized lines indicates the number of lines accumulated in the cumulative addition mode. That is, “initialization” means that the vertical streak noise component extraction unit 150 determines the initial value of the line buffer 154 by executing cumulative addition in the cumulative addition mode. “Initialization” means vertical initialization. This means that the streak noise component extraction unit 150 has shifted to the IIR filter operation after executing the cumulative addition in the cumulative addition mode, and “no initialization” means that the vertical line noise component extraction unit 150 performs the cumulative addition in the cumulative addition mode. This means that the operation has shifted to the IIR filter operation without executing. In FIG. 5 and FIG. 6, in order to easily compare the cases of “with initialization” and “without initialization”, the gain for each line when an infinite number of lines are cumulatively added is indicated by a broken line.

  FIG. 5A shows the relationship between the number of cumulative addition lines and the gain in the state where the initialization in the cumulative addition mode is completed. The area of the shaded portion indicated by the rectangle with the initialization coefficient Kx = 1 as the height and the number of initialization lines = 64 as the base is 64, and the initialization coefficient Kx = 1 and n = 64 in [Equation 2]. The area can also be obtained by substituting. That is, the area of the shaded portion in FIG. 5A is the degree of amplification of the cumulative value with respect to the input data, and indicates Sn stored in the line buffer 154 in a state where initialization is completed.

  FIG. 5B shows a state in which the vertical streak noise component extraction unit 150 shifts to the IIR filter mode and cumulatively adds the light-shielded image data, and the cumulative addition line when the light-shielded image data is cumulatively added up to the 50th. The relationship between the number and gain is shown. The shaded portion 400 is an area indicating the gain for each line of the light-shielded image data input in the IIR filter mode, and every time the light-shielded image data is input to the vertical streak noise component extraction unit 150, the line buffer up to the previous time. Since the accumulated value stored in 154 is multiplied by the feedback coefficient Ky = 63/64, the gain for each line has a gradient indicated by the shaded portion 400. The area of the shaded portion 400 is calculated by substituting n = 50 into [Equation 4], and is about 35.

  A shaded portion 401 is an area indicating the gain that the initial value of the line buffer 154 determined in the cumulative addition mode has for each line in the IIR filter mode. In consideration of starting the IIR filter mode using the initial value of the line buffer 154 determined in the cumulative addition mode, the feedback coefficient from the first shading image data input in the IIR filter mode is applied to the initial value of the line buffer 154. Ky = 63/64 is multiplied. Therefore, the area of the shaded portion is obtained by increasing the gain (feedback coefficient Ky = (63/64) to the 50th power) when the light-shielded image data is cumulatively added up to the 50th in the IIR filter mode, and the number of initialization lines (64 This is represented by a rectangular area with the bottom of the shaded portion being about 29, and is about 29. Therefore, the total area of the shaded portion 400 and the shaded portion 401 is 64.

  FIG. 5C shows the relationship between the cumulative number of added lines and gain when the shading image data is cumulatively added up to the 100th line, and FIG. 5D similarly shows the cumulative shading image data up to the 150th line. The relationship between the number of cumulative addition lines and the gain when added is shown. Similarly, the area of each shaded portion is 64.

  FIG. 5E shows the relationship between the cumulative number of added lines and the gain when the shading image data is cumulatively added up to the infinite number. The area of the shaded portion is found to be 64 by substituting the feedback coefficient Ky = 63/64 into [Equation 5].

  Similarly, the calculation conditions are shown in FIG. 6F. The initialization coefficient Kx = 2, the feedback coefficient Ky = 63/64, the number of initialization lines = (filter amplification factor) / (initialization coefficient Kx) = (1 / The relationship between the cumulative number of added lines (horizontal axis) and the gain (vertical axis) when (1-Ky)) / Kx = 32 is shown in FIGS.

  FIG. 6A shows the relationship between the number of cumulative addition lines and the gain when the initialization in the cumulative addition mode is completed. As an example, since the shading image data with the number of initialization lines = 32 is input with the initialization coefficient Kx = 2, the initialization coefficient Kx = 2 is the height, and the number of initialization lines is the base. The area of the shaded portion indicated by a rectangle is 64. That is, the number of initialization lines is set to 32, which is smaller than that in the case of FIG. 5A, and instead of shortening the time required for accumulation, the initialization coefficient Kx is set to 2 so that the initial value of the line buffer 154 is reduced. 64.

  FIG. 6B shows a state in which the vertical streak noise component extraction unit 150 is in the process of accumulating and adding light-shielded image data to the IIR filter mode, and the cumulative number of lines added when light-shielded image data is cumulatively added up to the 50th. And the gain. The area of the shaded portion is 64 by the same calculation as in the case of FIG.

  FIG. 6C shows the relationship between the number of accumulated addition lines and the gain when the shading image data is cumulatively added up to the 100th, and FIG. 6D similarly shows the cumulative shading image data up to the 150th. The relationship between the number of cumulative addition lines and the gain when added is shown. Similarly, the area of each shaded portion is 64.

  FIG. 6E shows the relationship between the cumulative number of added lines and the gain when the shading image data is cumulatively added up to the infinite number. The area of the shaded portion is found to be 64 by substituting the feedback coefficient Ky = 63/64 into [Equation 5].

  Therefore, the shaded areas of FIGS. 5A to 5E are equal. The same applies to FIG. That is, the vertical streak noise component extraction unit 150 performs initialization in the cumulative addition mode (FIG. 5 (A), FIG. 6 (A)) and shifts to the IIR filter mode ((FIG. 5 (B) 6 (B))), even if the cumulative addition of shading image data is stopped with a small number of lines (for example, FIG. 5 (C), FIG. 6 (C)), the infinite number of lines can be reduced in the IIR filter mode. Since the vertical streak noise component can be calculated using the same filter amplification factor (the area of each shaded portion in FIGS. 5 and 6) as in the case of the operation, the vertical streak noise component is obtained from the light-shielded image data. Can be extracted with high accuracy.

  Further, when the cumulative addition of the shading image data is stopped with a small number of lines, the vertical stripe noise component extraction unit 150 can extract the vertical stripe noise component at high speed without requiring time for the filter calculation. The IIR filter mode may be omitted and the correction mode may be shifted directly from the cumulative addition mode.

  In addition to the cumulative adder 153, a cumulative adder (not shown) that does not receive an input from the initialization coefficient multiplier 151 may be provided. In this case, the operation mode switching unit 158 can switch the data flow regardless of the SEL 152 by selecting a cumulative adder (not shown) according to the operation mode instead of switching the data flow by the SEL 152. In addition to the cumulative adder 153, a cumulative adder (not shown) that does not receive an input from the feedback coefficient multiplier 155 may be provided. In this case, the operation mode switching unit 158 can switch the data flow regardless of the SEL 156 by selecting a cumulative adder (not shown) according to the operation mode instead of switching the data flow by the SEL 156.

  FIG. 4C shows the flow of captured image data in the correction mode of the vertical streak noise component extraction unit 150. The vertical streak noise component extraction unit 150 extracts the vertical streak noise component in the cumulative addition mode and the IIR filter mode in which the cumulative addition is performed up to a predetermined number of lines, and then shifts to the correction mode.

  The correction coefficient multiplier 157 acquires the accumulated value of the light-shielded image data from the line buffer 154, and outputs a value obtained by multiplying the correction coefficient Kz to the vertical streak noise subtraction unit 140 as a vertical streak noise component. Here, the correction coefficient Kz may be set, for example, to the inverse of the filter amplification factor in order to return the light-shielded image data amplified by the filter to the level before amplification. Alternatively, the value of the correction coefficient Kz may be determined according to the characteristics of the solid-state imaging element 104, the temperature, the shooting conditions set by the user in the imaging device, and the like. For example, the correction coefficient Kz may be set smaller for the purpose of preventing overcorrection, or the correction coefficient Kz may be set larger for the purpose of enhancing correction.

  The vertical streak noise component extraction unit 150 may have the following configuration in addition to the configuration illustrated in FIG. 3. The vertical streak noise component extraction unit 150 shown in FIG. 7 has a horizontal average value calculator 301 added to the configuration shown in FIG. The horizontal average value calculator 301 acquires light-shielded image data from the solid-state image sensor 104.

  FIG. 8 shows the flow of shaded image data and captured image data in the vertical streak noise component extraction unit 150 shown in FIG. 7 for each operation mode. The operation modes of the vertical streak noise component extraction unit 150 include a cumulative addition mode 1, a cumulative addition mode 2, an IIR filter mode, and a correction mode. The horizontal average value calculator 301 includes a cumulative adder 302, a flip-flop (FF) 303, an average value calculator 304, and a SEL 305.

  FIG. 8A1 shows the flow of light-shielded image data in the cumulative addition mode 1. When shaded image data is input to the vertical streak noise component extraction unit 150, the vertical streak noise component extraction unit 150 operates in the cumulative addition mode 1. The operation mode switching unit 158 switches the data flow of the vertical streak noise component extraction unit 150 using the SEL 152 and the SEL 156.

  The horizontal average value calculator 301 calculates, for example, the average value of the light-shielded image in the horizontal direction of the solid-state image sensor 104 for one entire line.

  Next, the initialization coefficient multiplier 151 acquires the average value of the light-shielded image data from the horizontal average value calculator 301, multiplies it by the initialization coefficient Kx, and outputs it to the SEL 152. The SEL 152 outputs the average value of the light-shielded image data to the cumulative adder 153, and the cumulative adder 153 stores the cumulative value of the light-shielded image data in the line buffer 154.

  When the shading image data is input again to the vertical streak noise component extraction unit 150, the horizontal average value calculator 301 calculates the average value of the shading image in the horizontal direction of the solid-state imaging device 104 for the entire line. Next, the initialization coefficient multiplier 151 acquires the average value of the light-shielded image data from the horizontal average value calculator 301, multiplies it by the initialization coefficient Kx, and outputs it to the SEL 152. The SEL 152 outputs the average value of the light-shielded image data to the cumulative adder 153, and the cumulative adder 153 acquires the cumulative value of the light-shielded image data from the line buffer 154 via the SEL 156, and adds the average value of the light-shielded image to this. The result is stored in the line buffer 154.

  Next, the vertical streak noise component extraction unit 150 shifts to the cumulative addition mode 2 (FIG. 8 (A2)). In the cumulative addition mode 2, the horizontal average value calculator 301 does not particularly process the light-shielded image data, and transfers the light-shielded image data to the initialization coefficient multiplier 151. The operation of the vertical streak noise component extraction unit 150 after the cumulative addition mode 2 is the same as that of the cumulative addition mode shown in FIG.

  Next, the vertical streak noise component extraction unit 150 shifts to the IIR filter mode (FIG. 8B), and performs the same operation as the IIR filter mode shown in FIG. 4B described above. Furthermore, the vertical streak noise component extraction unit 150 shifts to the correction mode (FIG. 8C), and performs the same operation as the correction mode shown in FIG. 4C described above. Note that after the initialization in the cumulative addition mode 1, the mode may be shifted to the IIR filter mode. Further, after initialization in the cumulative addition mode 1, further initialization may be performed in the cumulative addition mode 2, and then the mode may be shifted to the IIR filter mode.

 That is, the vertical streak noise component extraction unit 150 illustrated in FIG. 3 calculates the cumulative value in the vertical direction with respect to the light-shielded image data, but the vertical streak noise component extraction unit 150 illustrated in FIG. The difference is that after calculating the average value in the horizontal direction for the data, the cumulative value is calculated in the vertical direction.

  The vertical streak noise component extraction unit 150 shown in FIG. 3 initializes the filter by increasing the initialization coefficient Kx when the number of lines in the OB region is small. There is a problem of being easily affected by random noise. However, since the vertical streak noise component extraction unit 150 shown in FIG. 7 performs initialization after using the average value in the horizontal direction, even if it is initialized with data having a small number of lines, it is difficult to be affected by random noise. Effective in terms.

  FIG. 9 shows the input data from the solid-state imaging device 104 to the vertical streak noise correction unit 105, the operation mode of the vertical streak noise component extraction unit 150, and the vertical streak noise subtraction unit 140 in the moving image shooting state or the moving image shooting standby state. The relationship with the operation will be described. Each value shown here is an example.

  Light-shielded image data is input to the vertical streak noise extraction unit 150 from the OB region 201 or OB region 203 or the OB region 201 and OB region 203 of the solid-state image sensor 104. The vertical streak noise extraction unit 150 executes the operation in the cumulative addition mode with the number of initialization lines = 8 and the initialization coefficient Kx = 4, and the initialization is completed. The vertical streak noise subtraction unit 140 is not operating (step S1).

  Captured image data is input from the effective area 202 of the solid-state image sensor 104 to the vertical streak noise subtraction unit 140. In order for the vertical streak noise subtracting unit 140 to subtract the vertical streak noise component from the captured image data, the vertical streak noise extracting unit 150 has a value obtained by multiplying the initialization line number = 8 and the initialization coefficient Kx = 4 by 32. Therefore, for example, the operation in the correction mode with the correction coefficient Kz = 1/32 is performed on the light-shielded image data acquired in step S <b> 1, and the vertical stripe noise component is extracted and output to the vertical stripe noise subtraction unit 140. The vertical stripe noise subtracting unit 140 subtracts the vertical stripe noise component from the captured image data (step S2).

  Light-shielded image data is input to the vertical streak noise extraction unit 150 from the OB region 201 or OB region 203 or the OB region 201 and OB region 203 of the solid-state image sensor 104. The vertical streak noise extraction unit 150 performs the IIR filter mode operation on eight lines with the feedback coefficient Ky = 31/32. The vertical streak noise subtraction unit 140 is not operating (step S3).

  In step S4, the same operation as in step S2 is performed. In step S5, the same operation as in step S3 is performed. In step S6, the same operation as in step S2 is performed. Continue movie shooting standby.

  As described above, the vertical streak noise extraction unit 150 acquires the light-shielded image data from the OB area 201 and the OB area 203, and has been initialized in step S1, so that IIR is applied to the light-shielded image data after step S3. The filter mode operation can be performed, and the vertical stripe noise component can be extracted without using the light-shielded image data by the shutter 103 in the moving image shooting state or the moving image shooting standby state.

  FIG. 10 shows the input data from the solid-state imaging device 104 to the vertical streak noise correction unit 105, the operation mode of the vertical streak noise component extraction unit 150, and the operation of the vertical streak noise subtraction unit 140 in the still image shooting state. The relationship is shown. Each value shown here is an example.

  The CPU 115 closes the shutter 103. The shaded image data is input from the effective area 202 of the solid-state image sensor 104 to the vertical streak noise extraction unit 150. In this case, the shaded image data may be input from a plurality of lines. The vertical streak noise extraction unit 150 executes the operation in the cumulative addition mode with the number of initialization lines = 32 and the initialization coefficient Kx = 2, and the initialization is completed. The vertical streak noise subtraction unit 140 is not operating (step S10).

  The CPU 115 inputs the shaded image data from the effective area 202 of the solid-state imaging device 104 to the vertical streak noise extraction unit 150 while the shutter 103 is closed. The vertical streak noise extraction unit 150 performs the IIR filter mode operation on eight lines with the feedback coefficient Ky = 63/64. The vertical streak noise subtraction unit 140 is not operating (step S11).

  The CPU 115 opens the shutter 103. Captured image data is input from the effective area 202 of the solid-state image sensor 104 to the vertical streak noise subtraction unit 140. The vertical streak noise extraction unit 150 executes the operation of the correction mode with the correction coefficient Kz = 1/64 on the shading image data acquired up to step S11, extracts the vertical streak noise component, and extracts the vertical streak noise subtraction unit. Output to 140. The vertical stripe noise subtraction unit 140 subtracts the vertical stripe noise component from the captured image data (step S12).

  As described above, the vertical streak noise extraction unit 150 acquires the light-shielded image data from the shutter 103, so that the vertical streak noise component in the still image shooting state can be obtained even when the solid-state image sensor 104 does not have the OB region 201 and the OB region 203. Extraction can be performed.

  FIG. 11 shows the input data from the solid-state imaging device 104 to the vertical streak noise correction unit 105, the operation mode of the vertical streak noise component extraction unit 150, and the operation of the vertical streak noise subtraction unit 140 in the still image shooting state. Regarding the relationship, a procedure different from the procedure of FIG. 10 is shown. Each value shown here is an example.

  The CPU 115 closes the shutter 103. The shaded image data is input from the effective area 202 of the solid-state image sensor 104 to the vertical streak noise extraction unit 150. In this case, the shaded image data may be input from a plurality of lines. The vertical streak noise extraction unit 150 executes the operation in the cumulative addition mode with the number of initialization lines = 64 and the initialization coefficient Kx = 1, and the initialization is completed. The vertical streak noise subtraction unit 140 is not operating (step S20).

  The CPU 115 opens the shutter 103. Captured image data is input from the effective area 202 of the solid-state image sensor 104 to the vertical streak noise subtraction unit 140. The vertical streak noise extraction unit 150 performs a correction mode operation on the light-shielded image data acquired in step S20 with the correction coefficient Kz = 1/64, extracts a vertical streak noise component, and extracts the vertical streak noise subtraction unit 140. Output to. The vertical stripe noise subtraction unit 140 subtracts the vertical stripe noise component from the captured image data (step S21).

  As described above, the vertical streak noise extraction unit 150 acquires the light-shielded image data from the shutter 103 and performs the vertical streak noise component in the still image shooting state without executing the operation in the IIR filter mode after initialization in the cumulative addition mode. It is also possible to perform extraction.

  FIG. 12 shows the input data from the solid-state imaging device 104 to the vertical streak noise correction unit 105, the operation mode of the vertical streak noise component extraction unit 150, and the operation of the vertical streak noise subtraction unit 140 in the continuous shooting state of still image shooting. And the relationship. Each value shown here is an example.

  The CPU 115 closes the shutter 103. The shaded image data is input from the effective area 202 of the solid-state image sensor 104 to the vertical streak noise extraction unit 150. In this case, the shaded image data may be input from a plurality of lines. The vertical streak noise extraction unit 150 executes the operation in the cumulative addition mode with the number of initialization lines = 64 and the initialization coefficient Kx = 1, and the initialization is completed. The vertical streak noise subtraction unit 140 is not operating (step S30).

  The CPU 115 opens the shutter 103. Captured image data is input from the effective area 202 of the solid-state image sensor 104 to the vertical streak noise subtraction unit 140. The vertical streak noise extraction unit 150 performs a correction mode operation on the light-shielded image data acquired in step S20 with the correction coefficient Kz = 1/64, extracts a vertical streak noise component, and extracts the vertical streak noise subtraction unit 140. Output to. The vertical stripe noise subtraction unit 140 subtracts the vertical stripe noise component from the captured image data (step S31).

  The CPU 115 closes the shutter 103 and the light-shielded image data is input to the vertical streak noise extraction unit 150 from the effective area 202 of the solid-state image sensor 104. The vertical streak noise extraction unit 150 performs the IIR filter mode operation on 16 lines with the feedback coefficient Ky = 63/64. The vertical streak noise subtraction unit 140 is not operating (step S32).

  In step S33, the same operation as in step S31 is performed. In step S34, the same operation as in step S32 is performed. In step S35, the same operation as in step S31 is performed. Continue shooting continuously.

  As described above, since the vertical streak noise extraction unit 150 performs the operation in the IIR filter mode for a small number of lines, the vertical streak noise component can be extracted at high speed, and high-speed continuous shooting for still image shooting can be performed. Is advantageous.

  The vertical streak noise subtracting unit 140 acquires a captured image from the solid-state image sensor 104 and a vertical streak noise component from the vertical streak noise component extracting unit 150, and pre-processes data obtained by subtracting the vertical streak noise component from the captured image. The data is output to 106.

  The preprocessing unit 106, the image processing unit 107, the display IF unit 108, and the card IF unit 110 are connected by a data bus 112, and the activation timing and operation mode of these blocks are controlled by the CPU 115. .

  The pre-processing unit 106 performs processing such as shading correction on the captured image from which the vertical streak noise component has been subtracted, for example, so that the entire image has a uniform brightness on average. The captured image data is output to.

  The image processing unit 107 performs image processing such as luminance chroma (YC) generation processing on the captured image data as necessary, and outputs the captured image data to the memory controller 113. The memory controller 113 stores the captured image data in the memory 114. The card IF unit 110 acquires captured image data from the memory 114 and stores the captured image data in the recording medium 111. The display IF unit 108 acquires captured image data from the memory 114 and displays it on the display panel 109.

  As described above, the vertical streak noise component extraction unit 150 performs initialization in consideration of the filter amplification factor of the IIR filter in the cumulative addition mode, and shifts to the IIR filter mode. Therefore, the number of lines acquired from the light-shielded image is small. Even in this case, the vertical stripe noise component can be extracted from the light-shielded image with high accuracy and at high speed, and a good captured image can be obtained at high speed.

  The embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and includes designs and the like that do not depart from the gist of the present invention. For example, when a solid-state image sensor is rotated 90 degrees and placed in an imaging device, or vertical line noise may appear as horizontal line noise due to image signal processing, image signal recording processing, or the like, vertical line noise The direction may not be limited to the vertical direction.

  Further, the initialization in the cumulative addition mode may not be limited so that the product of the number of initialization lines and the initialization coefficient becomes the filter amplification factor in the IIR filter mode.

  The noise component extraction unit according to the present invention corresponds to the vertical stripe noise component extraction unit 150, the noise subtraction unit corresponds to the vertical stripe noise subtraction unit 140, and the initialization coefficient multiplication unit performs initialization coefficient multiplication. The storage unit corresponds to the line buffer 154, the feedback coefficient multiplier corresponds to the feedback coefficient multiplier 155, the adder corresponds to the cumulative adder 153, and the horizontal average value calculator , Corresponding to the horizontal average value calculator 301.

  Also, a program for realizing the steps shown in FIGS. 9, 10, 11, and 12 is recorded on a computer-readable recording medium, and the program recorded on the recording medium is read into a computer system. Execution processing of the communication terminal may be performed by executing. Here, the “computer system” may include hardware such as an OS (Operating System) and peripheral devices.

  Further, the “computer system” includes a homepage providing environment (or display environment) if a WWW system is used. The “computer-readable recording medium” means a flexible disk, a magneto-optical disk, a ROM, a writable nonvolatile memory such as a flash memory, a portable medium such as a CD-ROM, a hard disk built in a computer system, etc. This is a storage device.

Further, the “computer-readable recording medium” refers to a volatile memory (for example, DRAM (Dynamic) in a computer system serving as a server or a client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line. Random Access Memory)) that holds a program for a certain period of time is also included.
The program may be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. Here, the “transmission medium” for transmitting the program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line.
The program may be for realizing a part of the functions described above.
Furthermore, what can implement | achieve the function mentioned above in combination with the program already recorded on the computer system, and what is called a difference file (difference program) may be sufficient.

It is a block diagram of the imaging device by one embodiment. This is a solid-state imaging device 104 when an OB region is provided. This is a vertical stripe noise component extraction unit 150. It is a figure which shows the flow of the shading image data and picked-up image data in the vertical stripe noise component extraction part 150 shown in FIG. FIG. 6 is a relationship diagram between the number of cumulative addition lines and gain when switching from the cumulative addition mode to the IIR filter operation mode. FIG. 6 is a relationship diagram between the number of cumulative addition lines and gain when switching from the cumulative addition mode to the IIR filter operation mode. This is a vertical stripe noise component extraction unit 150. It is a figure which shows the flow of the shading image data and picked-up image data in the vertical stripe noise component extraction part 150 shown in FIG. FIG. 6 is a relationship diagram between input data from the solid-state imaging device 104 to the vertical streak noise correction unit 105, an operation mode of the vertical streak noise component extraction unit 150, and an operation of the vertical streak noise subtraction unit 140 in a moving image shooting state or the like. . FIG. 6 is a relationship diagram between input data from the solid-state imaging device 104 to the vertical streak noise correction unit 105, an operation mode of the vertical streak noise component extraction unit 150, and an operation of the vertical streak noise subtraction unit 140 in a still image shooting state. . FIG. 6 is a relationship diagram between input data from the solid-state imaging device 104 to the vertical streak noise correction unit 105, an operation mode of the vertical streak noise component extraction unit 150, and an operation of the vertical streak noise subtraction unit 140 in a still image shooting state. . Relationship between the input data from the solid-state imaging device 104 to the vertical streak noise correction unit 105, the operation mode of the vertical streak noise component extraction unit 150, and the operation of the vertical streak noise subtraction unit 140 in the continuous shooting state of still image shooting. FIG.

Explanation of symbols

DESCRIPTION OF SYMBOLS 101 ... Optical lens 102 ... Aperture 103 ... Shutter 104 ... Solid-state image sensor 105 ... Vertical stripe noise correction part 106 ... Pre-processing part 107 ... Image processing part 108 ... Display IF part 109 ... Display panel 110 ... Card IF part 111 ... Recording medium DESCRIPTION OF SYMBOLS 112 ... Data bus 113 ... Memory controller 114 ... Memory 115 ... CPU 140 ... Vertical stripe noise subtraction part 150 ... Vertical stripe noise component extraction part 151 ... Initialization coefficient multiplier 152 ... SEL 153 ... Accumulation adder 154 ... Line buffer 155 ... Feedback coefficient multiplier 156 ... SEL 157 ... Correction coefficient multiplier 158 ... Operation mode switching unit 201 ... OB area 202 ... Effective area 203 ... OB area 301 ... Horizontal average value calculator 302 ... Cumulative adder 303 ... Flip-flop (FF) 304 ... Average value calculation Vessel 305 ... SEL 400 ... shaded 401 ... shaded

Claims (6)

A solid-state image sensor;
A noise component extraction unit that extracts a noise component from light-shielded image data output from a light-shielded region of the solid-state imaging device;
A noise subtracting unit that performs a process of subtracting the noise component from captured image data output from an unshielded region of the solid-state imaging device;
With
The noise component extraction unit
An initialization coefficient multiplier for multiplying the shading image data by an initialization coefficient;
A storage unit for storing cumulative values;
A feedback coefficient multiplier for multiplying a cumulative value stored in the storage by a feedback coefficient;
An adder;
With
A first mode in which the addition unit adds a cumulative value acquired from the storage unit to a value obtained by multiplying the shading image data by the initialization coefficient, and stores the addition result as a new cumulative value in the storage unit. To store the initial value of the cumulative value in the storage unit,
Next, the addition unit adds a value obtained by multiplying the accumulated value acquired from the storage unit by the feedback coefficient to the shaded image data, and stores the addition result as a new accumulated value in the storage unit. Run the mode
An imaging apparatus characterized by that. In the first mode, the noise component extraction unit
A product of the number of lines of the shading image data and the initialization coefficient;
A filter amplification factor calculated from the feedback coefficient;
The image pickup apparatus according to claim 1, wherein the accumulated value is initialized so that the two are equal. The noise component extraction unit
A horizontal average value calculating unit for calculating an average value in the horizontal direction of the shading image data,
The imaging apparatus according to claim 1, wherein in the first mode, the initialization value is multiplied by the average value calculated by the horizontal average value calculation unit. The noise component extraction unit
Execute the first mode, omit the execution of the second mode,
The noise subtraction unit
The imaging apparatus according to claim 1, wherein a process of subtracting the noise component from the captured image data is performed. The noise component extraction unit
Executing the first mode on the shading image data;
The second mode is executed on the shading image data acquired thereafter,
The noise subtraction unit
The imaging apparatus according to claim 1, wherein a process of subtracting the noise component from the captured image data is performed.   The number of lines of the shading image data acquired after that used in the second mode is smaller than the number of lines of the shading image data used in the first mode. Imaging device.

Images