JP2022053481A - Signal processing device, photoelectric conversion device, photoelectric conversion system, control method of signal processing device and program - Google Patents

Abstract

To provide a technique advantageous for improving the accuracy of correction for a signal output from an imaging device in a signal processing device.SOLUTION: A signal processing device processing image data output from a photoelectric conversion unit having a light receiving area and a light blocking area, is provide including: a control data generation unit that outputs control data used to generate correction data for correction of the image data by using a learned model generated by machine learning; and a signal processing unit that generates the correction data on the basis of the light blocking image data which is the image data of the light blocking data and the control data, and corrects the light receiving image data which is the image data of the light receiving area of the image data, without applying the learned model according to the correction data.SELECTED DRAWING: Figure 4

Description

The present invention relates to a signal processing device, a photoelectric conversion device, a photoelectric conversion system, a control method and a program of the signal processing device.

In an image pickup device using an image pickup element such as a CMOS image sensor, it is necessary to correct noise caused by a dark current generated in each pixel in order to improve the image quality. In Patent Document 1, the imaging region is divided into a plurality of blocks, the size and division position of the blocks are changed according to the imaging conditions, and various imaging conditions are used by using the correction values according to the division pattern stored in advance. It has been shown to make appropriate corrections to.

Japanese Unexamined Patent Publication No. 2017-034315

In order to further improve the image quality, it is necessary to more appropriately correct the dark current generated in each pixel of the image sensor and the noise caused by the circuit for reading the signal from the pixel.

An object of the present invention is to provide a technique advantageous for improving the accuracy of correction for a signal output from an image pickup device in a signal processing device.

In view of the above problems, the signal processing device according to the embodiment of the present invention is a signal processing device that processes image data output from a photoelectric conversion unit including a light receiving region and a light shielding region, and is generated by machine learning. A control data generation unit that outputs control data used to generate correction data for correction of the image data using the trained model, a light-shielding image data that is image data of the light-shielding region among the image data, and the light-shielding image data. Signal processing that generates the correction data based on the control data and corrects the light receiving image data, which is the image data of the light receiving region among the image data, according to the correction data without applying the trained model. It is characterized by including a part and.

According to the present invention, it is possible to provide a technique advantageous for improving the accuracy of correction for a signal output from an image pickup device in a signal processing device.

The figure which shows the structural example of the photoelectric conversion apparatus which includes the signal processing apparatus of this embodiment. The conceptual diagram of the output signal of the photoelectric conversion part of the photoelectric conversion apparatus of FIG. The conceptual diagram of the OB clamp processing of the signal processing apparatus of this embodiment. The flow diagram of the control data generation used for the correction of the signal processing apparatus of this embodiment. The figure of the machine learning model of the signal processing apparatus of this embodiment. The figure which shows the structural example of the photoelectric conversion apparatus which includes the signal processing apparatus of this embodiment. The figure which shows the structural example of the photoelectric conversion apparatus which includes the signal processing apparatus of this embodiment. The figure which shows the structural example of the photoelectric conversion apparatus which includes the signal processing apparatus of this embodiment. The flow diagram of the control data generation used for the correction of the signal processing apparatus of this embodiment. The figure which shows the arrangement example of the photoelectric conversion apparatus of FIG. The figure which shows the structural example of the image pickup apparatus which incorporated the photoelectric conversion apparatus of FIG. The figure which shows the structural example of the photoelectric conversion system including the photoelectric conversion apparatus of FIG.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. The following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiment, not all of the plurality of features are essential for the invention, and the plurality of features may be arbitrarily combined. Further, in the attached drawings, the same or similar configurations are given the same reference numbers, and duplicate explanations are omitted.

First Embodiment With reference to FIGS. 1 to 5, the signal processing apparatus according to the first embodiment of the present disclosure will be described. FIG. 1 is a diagram showing a configuration example of a photoelectric conversion device 100 including the signal processing device 150 of the present embodiment. The photoelectric conversion device 100 includes a signal processing device 150, and a photoelectric conversion unit 101 having a light receiving region and a light blocking region, which will be described later, and outputting image data processed by the signal processing device 150. The photoelectric conversion device 100 may further include a vertical scanning unit 103, a control unit 104, a reading circuit unit 105, an AD conversion unit 106, a memory unit 107, and a horizontal scanning unit 108.

The signal processing device 150 includes a machine learning unit 151 and a signal processing unit 152. When the machine learning unit 151 generates correction data for correcting image data in the signal processing unit 152 using the trained model 160 based on the trained model 160 generated by machine learning and the image data. Includes a control data generation unit 161 that outputs control data used in the above. The signal processing unit 152 generates correction data based on the image data and the control data generated by the control data generation unit 161 and corrects the image data output from the photoelectric conversion unit 101 according to the correction data. In the present embodiment, an example in which the signal processing device 150 is mounted on the photoelectric conversion device 100 is shown, but the present invention is not limited to this, and the signal processing device 150 is a separate body from the photoelectric conversion device 100. May be good.

The photoelectric conversion unit 101 is composed of (m + 1) × (n + 1) pixels in which m + 1 pixels 102 are arranged in the row direction (horizontal direction in FIG. 1) and n + 1 pixels are arranged in the column direction (vertical direction in FIG. 1). .. The vertical scanning unit 103 is connected to the m + 1 pixels 102 arranged for each row by the row selection line 110 (V (n)), and selects a row for reading a signal. In the selected row, the signals output from the m + 1 pixels 102 included in the selected row are simultaneously read out to the read circuit unit 105 via the vertical output line 111 (H (m)). The read circuit unit 105 may include an amplifier or the like and amplify a signal output from the pixel 102. The signal output from the read circuit unit 105 is converted from an analog signal to a digital signal (AD conversion) by the AD conversion unit 106, and is temporarily held in the memory unit 107. After that, the digital signals addressed by the horizontal scanning unit 108 are sequentially read out to the signal processing unit 152 of the signal processing device 150, and the signal processing unit 152 performs digital signal processing. The control unit 104 may include, for example, a timing generator and may generate control signals for each configuration of the photoelectric conversion device 100. Further, the control unit 104 acquires setting information such as imaging conditions at the time of imaging of the photoelectric conversion device 100 by communicating with the outside of the photoelectric conversion device 100, and corresponds to each configuration included in the photoelectric conversion device 100 according to the conditions. It can supply control signals. Based on this setting information, the control unit 104 controls the vertical scanning unit 103, the reading circuit unit 105, the AD conversion unit 106, the memory unit 107, the horizontal scanning unit 108, and the signal processing unit 152.

The signal processing unit 152 performs a process of reducing reset noise generated in a switch element (for example, a MOS transistor) included in each pixel 102 of the photoelectric conversion unit 101. Further, the image data output from the photoelectric conversion unit 101 has variations (FPN: fixed pattern noise) caused by the dark current generated from the photodiode included in the pixel 102 and the circuit-induced differences such as power supply impedance and signal delay. ) Can be included. In the present specification, the state in which the FPN varies from row to row and column to column with a certain rule is referred to as shading. After reducing the reset noise component, the signal processing unit 152 reduces the FPN component, the shading component, and the like, and performs a process of extracting the signal component in which the noise component is suppressed from the image data. In the present specification, the process in which the signal processing unit 152 reduces the noise component of the image data is referred to as an OB clamp process.

FIG. 2 is a conceptual diagram in the case where the signal output from the photoelectric conversion unit 101 is broken line by line in the present embodiment. The photoelectric conversion unit 101 includes a light receiving region 202 that receives light incident on the optical system such as a lens and a light shielding region 201 (which may also be referred to as an optical black (OB) region) that optically shields the incident light. .. The light-shielding region 201 is a reference region for determining a so-called black level reference in the image data obtained by the photoelectric conversion unit 101. Here, in the light-shielding area 201, as shown in FIG. 2, the area in which the pixels 102 are optically light-shielded over the entire row at the upper part of the screen is referred to as a VOB area 203. Further, a region located on the left side of the light receiving region 202 and optically shielding the pixels 102 over the entire row is referred to as a HOB region 204.

In order to reduce the FPN component for each row and each column in the OB clamping process, the signal processing unit 152 generates correction data from the light-shielded image data which is the image data of the light-shielding region 201 among the image data. Next, the signal processing unit 152 corrects the received light image data, which is the image data of the light receiving area 202 among the image data, according to the correction data. For example, the signal processing unit 152 averages the signal level (signal value) of the light-shielded image data row by row and column by column to generate correction data. Next, the signal processing unit 152 may generate data for image display by subtracting the corresponding signal level of the correction data from each signal level of the received light image data. Here, an arbitrary area of the light-shielding area 201 can be set as the area of the light-shielding area 201 for acquiring the average value for each row and each column of the signal level. This region is referred to as a clamp value generation region. Further, the average value obtained from the clamp value generation area is referred to as a clamp value.

3 (a) to 3 (c) are conceptual diagrams of the OB clamp process. 3A shows the photoelectric conversion unit 101 shown in FIG. 2, FIG. 3B is a conceptual diagram before the OB clamping process is performed, and FIG. 3C shows the OB clamping process. It is a conceptual diagram after the implementation. FIG. 3B shows the time on the vertical axis and the signal level of the light-shielding region 201 on the horizontal axis, and shows an example of the average value of the signal level corresponding to the level of the FPN component. FIG. 3B shows a conceptual diagram in which the FPN component increases row by row, and shows a clamp value 301 when the entire region of the light-shielding region 201 is used as a clamp value generation region. The result of subtracting the clamp value 301 from the signal level of each row is shown in FIG. 3 (c). In FIG. 3C, by subtracting the clamp value 301 from the signal level obtained in the light-shielding region 201, the FPN component and the shading component are suppressed from the image data obtained by the photoelectric conversion unit 101 including the light-receiving region 202. The state in which the signal component was taken out is shown. That is, an example is shown in which the OB clamping process by the signal processing unit 152 is appropriately performed.

However, in the OB clamping process, if the average value of each row of the light-shielding image data in the light-shielding area 201 of the image data is used as it is for subtraction, the average value may vary due to the influence of random noise or the like that differs for each pixel. There is sex. Therefore, in the OB clamping process, the shading component cannot be appropriately reduced, and the accuracy of the OB clamping process may be lowered.

Therefore, for example, by using an LPF (Low Pass Filter) for the average value of the light-shielded image data obtained for each row, the clamp value can be made resistant to row variation and the accuracy of the OB clamping process can be improved. Conceivable. Therefore, when the signal processing unit 152 performs the OB clamping process, it is necessary to provide the signal processing unit 152 with control data for performing the OB clamping process. The signal processing unit 152 generates the above-mentioned correction data based on the light-shielding image data and the control data, which are the image data of the light-shielding region 201 among the image data. In addition to the LPF feedback gain (following to shading) information, the control data for OB clamping processing includes information on the size of the clamp value generation area used for OB clamping processing and the clamp value among the light-shielding image data. Information on the location of the generation area, etc. can be mentioned. The control data generation unit 161 arranged in the machine learning unit 151 of the signal processing device 150 generates these control data by using the trained model 160. The signal processing unit 152 generates correction data based on the shading image data and the information of these control data, and for example, subtracts the signal level of the correction data from the signal level of the received light image data as described above. Can generate data for image display.

Next, the process for generating control data will be described with reference to FIG. First, in S401, image data for generating control data is prepared. The image data for generating the control data may be image data acquired by conventional imaging, or may be image data acquired under the same imaging conditions. These image data may be supplied from the outside of the signal processing device 150 or the photoelectric conversion device 100, or as shown in FIG. 1, a memory 162 is arranged in the machine learning unit 151 of the signal processing device 150, and the memory It may be stored in 162. Further, as the image data for generating the control data, the light-shielding image data which is the data of the light-shielding region 201 among the image data acquired in real time may be used.

When the image data for generating the control data is prepared, the step transitions to S402 to S404. In S402, as the control data, the control data generation unit 161 determines the size of the region (clamp value generation region) for generating the correction data in the light-shielded image data, that is, the average parameter when generating the clamp value. .. The average parameter for generating the clamp value may be determined, for example, from the magnitude of the noise amount of the FPN component in the light-shielding region 201. For example, when the amount of noise is large, the clamp value generation area is set large in order to suppress large variations, and when the amount of noise is small, the clamp is clamped so as not to be unnecessarily affected by stains and unevenness existing in the light-shielding area 201. The size of the value generation area is set small.

Further, in S403, as control data, the control data generation unit 161 determines the location of a region (clamp value generation region) for generating correction data in the light-shielded image data. The location of the clamp value generation region is determined from the shading shape in the light-shielding region 201 and the location of the pixel that outputs an abnormal signal value. The pixel that outputs an abnormal signal may be, for example, a pixel in a region that causes spots or unevenness, or a pixel that causes scratches. Spots and unevenness can appear by a set of pixels that output a signal level that is relatively larger or smaller than other pixels. Further, the scratched pixel may be a pixel that always outputs the same signal level, such as a white scratch or a black dot. If stains or unevenness are included in the clamp value generation area, the clamp value may contain components other than the desired FPN component, and the FPN component may not be reduced, so that appropriate OB clamping processing cannot be performed. There is a risk. Therefore, when the light-shielding image data of the light-shielding area 201 has spots or unevenness, the clamp value generation area can be set by removing the area where the spots or unevenness appear.

Further, in S404, as control data, the control data generation unit 161 determines the followability to shading. For example, the followability to shading may be determined from the size of the shading component. For example, when the shading amount is large, the followability of the clamp value is set to be large, and the clamp value is set so as to be able to follow the large shading amount. When the shading amount is small, the followability is set small so that the clamp value is not affected by unintended variations.

Each step of S402 to S404 may be carried out in any order. Further, for example, the steps S402 to S404 may be performed in parallel at the same time. Further, for example, all the steps of S402 to S404 may not be executed. By executing at least one step and outputting the control data from the control data generation unit 161 to the signal processing unit 152, the accuracy of the OB clamping process performed by the signal processing unit 152 can be improved.

The control data generation unit 161 includes control data including at least one of the information on the size of the clamp value generation area, the information on the location of the clamp value generation area, and the information on the followability to shading determined in S402 to S404. Is output to the signal processing unit 152. Next, in S405, the signal processing unit 152 generates correction data based on the control data determined in S402 to S404 and the light-shielding image data of the light-shielding area 201, and the light-receiving image data of the light-receiving area 202 is the correction data. The OB clamping process that corrects according to the above is performed. As a result, in the present embodiment, the clamp value according to the imaging condition is calculated as the correction data, and an appropriate OB clamp process capable of reducing the FPN component and the shading component becomes possible. As a result, it is possible to obtain an image with good image quality.

Here, the image data from the photoelectric conversion unit 101 of the photoelectric conversion device 100 may have the same imaging conditions depending on the scratches generated during the manufacturing process of the photoelectric conversion unit 101 and the placement location of the chips in the lot. Each chip may have different FPN components and unevenness. Further, for example, even within the same lot, the thickness of the interlayer film differs between the chips, and the size of the wiring capacity varies among the chips. If the size of the wiring capacity is different, the amount of shading and the shape may vary between the chips. When the chip size of the photoelectric conversion unit 101 is large, for example, as an image sensor used in a digital single-lens reflex camera, it is more affected by such variations. Therefore, the control data generation unit 161 tunes the trained model 160 for generating the control data before shipment in order to carry out the OB clamping process suitable for the FPN component and unevenness that vary from chip to chip. It becomes very important.

Further, even in the same photoelectric conversion unit 101, the FPN component changes depending on the imaging conditions such as the amount of dark current generated depending on the accumulation time during imaging and the amplification factor of the FPN component depending on the ISO setting. It is a possible value. Further, even if it is not at the time of shipment, the image output from the photoelectric conversion unit 101 may have stains or unevenness due to aging. Based on these effects, if an appropriate OB clamping process cannot be performed each time, the FPN component and shading component cannot be reduced, and the accuracy of the OB clamping process may not be maintained.

Therefore, in the present embodiment, the control data generation unit 161 inputs the image data (light-shielding image data) into the trained model 160 and outputs the control data for performing the OB clamping process. That is, the control data generation unit 161 outputs control data for reducing the FPN component by using a trained model in which shading shapes, stains, and unevenness that differ for each imaging condition, lot, and chip are machine-learned.

In the present embodiment, the control data generation unit 161 causes the trained model 160 to output the control data for generating the clamp value, instead of outputting the clamp value itself. Further, the signal processing unit 152 generates correction data based on the control data and the light-shielding image data of the light-shielding region 201, and does not apply the trained model 160 to the light-receiving image data of the light-receiving region 202 according to the correction data. The OB clamping process is performed to correct the data. By doing so, for example, it is possible to reduce the area of the memory 162 as compared with the case where the clamp value itself is held in the memory 162. Further, when the clamp value itself is held in the memory 162, it is necessary to access the memory 162 for each frame. However, in the present embodiment, when it is not necessary to change the control data, for example, when continuous shooting is performed without changing the shooting conditions, it is not necessary to access the memory 162, and shooting at a high frame rate becomes possible. ..

Further, the machine learning unit 151 may further include a learning unit 163 that updates the trained model 160 by machine learning using image data. By repeating the imaging with the photoelectric conversion device 100, the learning unit 163 may, for example, capture the statistical characteristics of the acquired image data with respect to the imaging conditions and update the trained model 160. Thereby, the trained model 160 can be adapted to, for example, the occurrence of spots and unevenness due to aging. Machine learning by the learning unit 163 in this case can be called unsupervised learning.

FIG. 5 is a schematic diagram of a neural network of a machine learning model in this embodiment. The neural network may include an input layer having a plurality of nodes, an intermediate layer having a plurality of nodes, and an output layer having one node. Image data is input to each node of the input layer. Each node in the middle layer is connected to each node in the input layer. Each element of the input value input to each node of the intermediate layer is used for the operation at each node of the intermediate layer. Each node of the intermediate layer calculates a calculated value by using, for example, an input value input from each node of the input value, a predetermined weighting coefficient, and a predetermined bias value. Each node in the intermediate layer is connected to the output layer, and the calculated calculated value is output to the node in the output layer. For the nodes of the output layer, the calculated values are input from each node of the intermediate layer. The machine learning model (intermediate layer) derives control data from the input image data so that the FPN component can be reduced under various imaging conditions. If the OB clamping process is performed based on the control data derived in this way, it is possible to perform the OB clamping process with high accuracy adapted to not only the FPN component and shading component that differ depending on the imaging conditions and chips but also the secular variation. As a result, in the present embodiment, it is possible to acquire an image having good image quality.

Second Embodiment With reference to FIG. 6, the signal processing apparatus according to the second embodiment of the present disclosure will be described. FIG. 6 is a diagram showing a configuration example of a photoelectric conversion device 600 including the signal processing device 150 of the present embodiment. The signal processing device 150 of the present embodiment is different from the above-mentioned first embodiment in that the signal processing device 150 of the present embodiment is the machine learning unit 651 with respect to the machine learning unit 151 of the signal processing device 150 shown in FIG. Compared with the machine learning unit 151, the machine learning unit 651 has an additional signal line for acquiring a signal from the outside of the photoelectric conversion device 600. Other than the configuration of the machine learning unit 651, it may be the same as the first embodiment described above. Therefore, the points different from the first embodiment described above will be mainly described, and the same may be applied. Will omit the description as appropriate.

In the first embodiment described above, the trained model 160 is used, and the control data generation unit 161 derives control data suitable for each chip and imaging conditions from the image data, so that appropriate OB clamping processing can be performed. Indicated. Further, in the first embodiment, it has been explained that the learning unit 163 may perform unsupervised learning. On the other hand, in the present embodiment, the learning unit 163 updates the trained model 160 by performing machine learning using the teacher data. Then, as in the first embodiment described above, the control data generation unit 161 generates the control data used by the signal processing unit 152 in the OB clamping process using the trained model 160.

More specifically, in the present embodiment, data from which the FPN component and the shading component are removed from the image data is provided from the outside of the signal processing device 150 (photoelectric conversion device 100). For example, the user provides data from which the FPN component and the shading component have been removed. The learning unit 163 performs machine learning using data in which noise is reduced from image data provided from the outside of the signal processing device 150 as teacher data.

For example, noise is generated from the correction data generated by the signal processing unit 152 based on the control data generated by the control data generation unit 161 using the trained model 160, and the image data obtained under the same image data and the same imaging conditions. The learning unit 163 compares the reduced teacher data with the reduced teacher data. As a result, the learning unit 163 can perform machine learning so that the control data generation unit 161 can generate control data suitable for the chip and the imaging conditions by using the trained model 160. Based on this comparison result, the learning unit 163 updates the trained model 160. Further, for example, the learning unit 163 performs machine learning using data corresponding to the control data input from the outside of the signal processing device 150 and output by the control data generation unit 161 capable of reducing noise of the image data as teacher data. You may.

The machine learning performed by the learning unit 163 may be performed, for example, before the photoelectric conversion device 100 is shipped. Further, after the photoelectric conversion device 100 is shipped, the learning unit 163 may perform machine learning, for example, at the time of starting or ending the photoelectric conversion device 100, or while charging the battery. Further, the learning unit 163 may perform machine learning at any time in response to a request from a user or a photoelectric conversion system described later.

Further, in the first embodiment and the present embodiment described above, the learning unit 163 is arranged in the machine learning unit 651, but the present invention is not limited to this. For example, the learning unit 163 may be arranged in the signal processing unit 152, or may be arranged separately from the machine learning unit 151 and the signal processing unit 152.

In this embodiment, machine learning is performed using teacher data. As a result, if the OB clamping process is performed based on the control data generated by the control data generation unit 161 using the trained model 160 that is updated as appropriate, the FPN component and shading component that differ depending on the imaging conditions and chips can be reduced. It can be carried out with high accuracy. Furthermore, it is possible to perform OB clamping processing with high accuracy that adapts to aging. As a result, in the present embodiment, it is possible to acquire an image having good image quality.

Third Embodiment With reference to FIG. 7, the signal processing apparatus according to the third embodiment of the present disclosure will be described. FIG. 7 is a diagram showing a configuration example of a photoelectric conversion device 700 including the signal processing device 150 of the present embodiment. The signal processing device 150 of the present embodiment is different from the above-mentioned first embodiment in that the signal processing device 150 of the present embodiment is the machine learning unit 751 with respect to the machine learning unit 151 of the signal processing device 150 shown in FIG. Compared with the machine learning unit 151, the machine learning unit 751 has an additional signal line for acquiring a signal from the signal processing unit 152. Further, the machine learning unit 751 further includes an analysis unit 750. Other than the configuration of the machine learning unit 751, it may be the same as the first embodiment described above. Therefore, the points different from the first embodiment described above will be mainly described, and the same may be applied. Will omit the description as appropriate.

In the second embodiment described above, it has been described that the learning unit 163 performs machine learning and updates the trained model 160 using the data received from the outside of the signal processing device 150 (photoelectric conversion device 100) as the teacher data. .. In the present embodiment, the signal processing unit 152 performs OB clamping processing using the control data generated by the control data generation unit 161 and uses the result as teacher data for learning. More specifically, the analysis unit 750 analyzes the degree of noise reduction of the correction data generated by the signal processing unit 152 based on the control data generated by the control data generation unit 161. The analysis unit 750 analyzes the correction data generated by the signal processing unit 152, feeds back the result of determining the accuracy of the OB clamp processing (the degree of noise reduction of the correction data) as teacher data, and feeds it back to the learning unit. 163 performs machine learning. That is, in the present embodiment, unlike the second embodiment described above, the teacher data is generated inside the signal processing device 150. The signal line from the signal processing unit 152 to the machine learning unit 751 shown in FIG. 7 is arranged to transfer the corrected image data to which the OB clamping process has been performed by the signal processing unit 152 to the analysis unit 750.

Machine learning by the learning unit 163 in this embodiment can be called reinforcement learning. Machine learning may be performed every time the signal processing unit 152 corrects the received light image data according to the correction data and generates the corrected image data. As the number of teacher data increases, the accuracy of the OB clamping process may become higher.

An example of the method of analyzing the degree of removal of the FPN component by the OB clamping process carried out by the analysis unit 750 is shown below. With respect to the input image data, the analysis unit 750 sets an arbitrary area in the light-shielding image data of the light-shielding area 201. Here, this area is referred to as an FPN shape calculation area. Next, the analysis unit 750 obtains the average value of the signal levels for each row and each column in the FPN shape calculation area, and calculates an approximate curve with respect to the average value. This approximate curve becomes a function indicating the FPN shape. Here, the ideal state in which the FPN component can be completely removed is defined as the state in which the slope of the approximate curve is zero. By obtaining the correlation between the approximate curve calculated by the analysis unit 750 and the function in which the slope of the approximate curve in the ideal state is zero, it is possible to determine the degree of reduction of the FPN component. The degree of noise reduction analyzed by the analysis unit 750 is input to the learning unit 163 and utilized as teacher data.

In the present embodiment, the learning unit 163 and the analysis unit 750 are arranged in the machine learning unit 751, but the present invention is not limited to this. For example, the learning unit 163 and the analysis unit 750 may be arranged in the signal processing unit 152, or may be arranged separately from the machine learning unit 151 and the signal processing unit 152, respectively. Further, the learning unit 163 and the analysis unit 750 may be integrated. When the user uses the photoelectric conversion device 100 under various imaging conditions, a wide range of teacher data can be obtained, and it is expected that the learning accuracy will be improved in a shorter period of time.

In the present embodiment, teacher data is generated in the signal processing device 150 and machine learning is performed. As a result, if the OB clamping process is performed based on the control data generated by the control data generation unit 161 using the trained model 160 that is updated as appropriate, the FPN component and shading component that differ depending on the imaging conditions and chips can be reduced. It can be carried out with high accuracy. Furthermore, it is possible to perform OB clamping processing with high accuracy that adapts to aging. As a result, in the present embodiment, it is possible to acquire an image having good image quality.

Fourth Embodiment With reference to FIGS. 8 and 9, the signal processing apparatus according to the fourth embodiment of the present disclosure will be described. FIG. 8 is a diagram showing a configuration example of a photoelectric conversion device 800 including the signal processing device 150 of the present embodiment. The signal processing device 150 of the present embodiment is different from the above-mentioned first embodiment in that the signal processing device 150 of the present embodiment is the machine learning unit 851 with respect to the machine learning unit 151 of the signal processing device 150 shown in FIG. The machine learning unit 851 further includes an extraction unit 850 as compared with the machine learning unit 151. Other than the configuration of the machine learning unit 851, it may be the same as the first embodiment described above. Therefore, the points different from the first embodiment described above will be mainly described, and the same may be applied. Will omit the description as appropriate.

In the first embodiment described above, it has been described that image data is input to each node of the input layer of the neural network of the trained model 160. In the present embodiment, the extraction unit 850 is arranged in the machine learning unit 851, the feature amount for improving the accuracy of the control data is extracted from the image data, and the extracted feature amount is used in the neural network of the trained model 160. Input to each node of the input layer. Instead of inputting the image data to the neural network of the trained model 160 as in each of the above-described embodiments, the information of the feature amount extracted from the image data is input to the neural network. As a result, the accuracy of the control data output by the control data generation unit 161 to the signal processing unit 152 using the trained model 160 can be improved, and the accuracy of the OB clamp processing can be improved. Here, it is described that only the feature amount is input to the trained model 160, but both the image data and the feature amount of the image data may be input to the neural network of the trained model 160.

FIG. 9 is a process for generating control data in the present embodiment. Compared with the process shown in FIG. 4, the step of S901 for extracting the feature amount of the image data is added. Since the steps other than this may be the same as the steps shown in FIG. 4, the description thereof will be omitted as appropriate.

When the image data is prepared in S401, the step shifts to S901, and the extraction unit 850 extracts the feature amount for improving the accuracy of the control data from the image data. The feature amount may be, for example, the amount of FPN noise in the light-shielding image data of the light-shielding region 201 for determining the size of the clamp value generation region. Further, for example, the feature amount may be a shading shape in the light-shielding image data of the light-shielding region 201 for determining the location of the clamp value generation region. Further, for example, the feature amount may be the size of the shading component in the light-shielding image data of the light-shielding region 201 for determining the shading followability. Further, as described above, if there are stains, unevenness, or scratches, the accuracy of the clamp value may decrease. Further, even if there is a step in the clamp value generation region, the accuracy of the clamp value may be similarly lowered. Here, the "step" in the present embodiment may be a state in which the signal value has an offset with a certain row or a certain column as a boundary. Therefore, the feature amount may be, for example, the information of the pixel 102 that outputs an abnormal signal value in the light-shielding image data of the light-shielding region 201. The extraction unit 850 inputs at least one of these pieces of information into the trained model 160 as a feature amount.

Here, an example of a method for deriving the noise amount of FPN in the light-shielding image data of the light-shielding region 201 is shown. Prepare a plurality of image data acquired under the same imaging conditions. Each pixel value of these image data is averaged in the time direction. This averaging process makes it possible to remove random noise components. An arbitrary area is set in the light-shielded image data for the image data averaged in the time direction, and the standard deviation in this area is calculated. This standard deviation is the amount of FPN noise in the light-shielding image data of the light-shielding region 201.

The method of deriving the shading shape and the size of the shading component in the shading image data of the light-shielding region 201 is the same as the analysis performed by the analysis unit 750 in the third embodiment described above. Therefore, the description thereof is omitted here.

Next, an example of a method of deriving the location of the pixel 102 that outputs an abnormal signal value such as a spot, unevenness, scratch, or the above-mentioned step in the light-shielding image data of the light-shielding region 201 will be shown. The light-shielding image data of the light-shielding area 201 is divided into arbitrary areas. For each of the divided regions, the average value of the signal levels in the region is acquired. Among them, one area is selected, and the average value of the signal level of each area is standardized by using the average value of the signal level of the selected area. At that time, it can be determined that the region where the average value deviates from the other regions is the region where the pixels 102 that cause spots, unevenness, scratches, and steps are arranged. Further, for example, the information of the pixel 102 that outputs an abnormal signal value such as a spot, unevenness, scratch, or step known at the time of shipment may be stored in a memory 162 or the like. Here, the information of the pixel 102 that outputs the abnormal signal value stored in the memory 162 may be the position information of the pixel 102 that outputs the abnormal signal value, or the signal value of the pixel 102 that outputs the abnormal signal value. Level may be used. Further, even after shipment, the information of the pixel 102 that outputs the abnormal signal value extracted by the extraction unit 850 or the like may be stored in the memory 162. As a result, the amount of calculation in the extraction unit 850 when extracting the feature amount can be suppressed.

When the extraction unit 850 extracts the feature amount, the feature amount is input to the trained model 160. Next, in S402 to S404, the control data generation unit 161 transfers the control data used when the signal processing unit 152 performs the OB clamping process to the signal processing unit 152 by using the trained model 160 in which the feature amount is input. Output. Based on these information derived from the image data, each set value required for the OB clamping process is determined. Next, in S405, the signal processing unit 152 generates correction data based on the control data determined in S402 to S404 and the light-shielding image data of the light-shielding area 201, and the light-receiving image data of the light-receiving area 202 is according to the correction data. The OB clamping process for correction is performed.

In the present embodiment, the learning unit 163 and the extraction unit 850 are arranged in the machine learning unit 851, but the present invention is not limited thereto. For example, the learning unit 163 and the extraction unit 850 may be arranged in the signal processing unit 152, or may be arranged separately from the machine learning unit 151 and the signal processing unit 152, respectively.

In the present embodiment, the feature amount extracted from the image data is input to the neural network of the trained model 160 to improve the accuracy of the control data generation unit 161 to generate the control data. This enables more appropriate OB clamping processing by the signal processing unit 152. That is, also in this embodiment, it is possible to accurately reduce the FPN component and the shading component that differ depending on the imaging conditions and the chip. Furthermore, it is possible to perform OB clamping processing with high accuracy that adapts to aging. As a result, in the present embodiment, it is possible to acquire an image having good image quality.

The first to fourth embodiments described so far may be combined. For example, the trained model 160 may be updated using data input from the outside of the signal processing device 150 as teacher data, and the feature amount of the image data may be input to the updated trained model 160. Further, for example, even if the trained model 160 is updated, the data input from the outside of the signal processing device 150 and the data analyzed by the signal processing unit 152 for performing the OB clamping process are used as teacher data, respectively. good. Further, for example, the trained model 160 is updated and updated with the data input from the outside of the signal processing device 150 and the data analyzed by the signal processing unit 152 for performing the OB clamping process as teacher data, respectively. The feature amount of the image data may be input to the trained model 160.

In this way, by combining each of the above-described embodiments, a trained model 160 that is adapted with higher accuracy to FPN components, shading components, and secular variation, which differ depending on imaging conditions, chips, and the like, can be obtained. This makes it possible to realize a signal processing device 150 capable of performing OB clamping processing with high accuracy. Further, in the photoelectric conversion devices 100, 600, 700, 800 incorporating the signal processing device 150 of the present disclosure, it is possible to obtain an image with high image quality.

Other Embodiments The signal processing apparatus 150 that carries out the OB clamping process described in each of the above-described embodiments may be provided inside the photoelectric conversion device 100 as described above. However, it is not limited to this. For example, the signal processing device 150 may be arranged separately from the photoelectric conversion device 100 and the photoelectric conversion unit 101. The signal processing device 150 may be a computer such as a personal computer including a processor (for example, a CPU or MPU), which is arranged separately from the photoelectric conversion device 100. Further, for example, the signal processing device 150 may be a circuit such as an ASIC that realizes the above-mentioned functions.

FIG. 10 is a diagram showing an arrangement example when each block of the photoelectric conversion device 100 shown in FIG. 1 is arranged on a substrate such as a semiconductor. The photoelectric conversion device 100 may include a substrate 1001 and a substrate 1002 using a semiconductor such as silicon. Each configuration such as a signal processing device 150, a vertical scanning unit 103, a control unit 104, a reading circuit unit 105, an AD conversion unit 106, a memory unit 107, and a horizontal scanning unit 108 is arranged on the substrate 1001. On the substrate 1002, a photoelectric conversion unit 101 in which pixels 102 having a light receiving region 202 and a light shielding region 201 are arranged in an array is arranged. As shown in FIG. 10, at least a part of the substrate 1001 and the substrate 1002 may be laminated. With this configuration, in the analog unit including the photoelectric conversion unit 101 and the logic unit including the signal processing device 150, it becomes possible to select a suitable process when manufacturing the photoelectric conversion device 100. By using an appropriate manufacturing process for each configuration, it may be possible to obtain good characteristics in each configuration included in the photoelectric conversion device 100. As a result, the photoelectric conversion device 100 with improved image quality can be obtained.

FIG. 11 shows the configuration of the image pickup device 1100 as an example of the photoelectric conversion devices 100, 600, 700, and 800 in which the above-mentioned signal processing device 150 is incorporated. The image pickup apparatus 1100 includes a signal generation unit 1101, a signal correction unit 1102, a CPU 1103, an external input unit 1104, an optical system 1105, an image display unit 1106, a recording unit 1107, and a drive system 1108.

The signal correction unit 1102 may be the signal processing device 150 described above. Further, the signal generation unit 1101 includes a photoelectric conversion unit 101 including the above-mentioned pixels 102, a vertical scanning unit 103, a control unit 104, a reading circuit unit 105, an AD conversion unit 106, a memory unit 107, a horizontal scanning unit 108, and the like. Can include. Therefore, the configuration including the signal generation unit 1101 and the signal correction unit 1102 may be the above-mentioned photoelectric conversion device 100, 600, 700, 800. In this configuration, the signal generation unit 1101 may also be referred to as a photoelectric conversion device by itself in the signal generation unit 1101. That is, as described above, the signal processing device 150 may be arranged separately from the photoelectric conversion device.

In response to the light incident on the optical system 1105 for incident light on the photoelectric conversion unit 101 of the signal generation unit 1101, the signal generation unit 1101 performs photoelectric conversion to generate an analog image signal, and AD conversion is performed. , Output image data. The output image data is corrected by the signal correction unit 1102 so that it can be output and stored in the video display unit 1106 and the recording unit 1107. The image display unit 1106 displays an image using the display image data after the correction process. In addition, the recording unit 1107 stores the display image data. The CPU 1103 controls each configuration in the image pickup apparatus 1100 described above. The drive system 1108 is arranged, for example, to operate the focus and aperture of the optical system 1105. The external input unit 1104 may be various buttons for inputting and operating by the user, such as imaging conditions and shutter operation. A touch panel may be arranged as the image display unit 1106, and the image display unit 1106 may function as (a part of) the external input unit 1104.

In addition to the configuration shown in FIG. 11, the image pickup apparatus 1100 may have a configuration for obtaining information on other environments such as an image obtained by the signal generation unit 1101, such as a thermometer. For example, the signal processing device 150 may perform machine learning using information such as temperature information that cannot be obtained by the signal generation unit 1101 (photoelectric conversion device) to construct the trained model 160. By using various environmental parameters of the environment in which the image pickup apparatus 1100 is used, it may be possible to carry out the OB clamping process appropriately corresponding to the imaging conditions. As a result, the image quality of the image obtained by the image pickup apparatus 1100 can be improved.

FIG. 12 includes a plurality of photoelectric conversion devices 100, 600, 700, 800 and a communication unit 1202 for communicating with an external server 1201 (for example, a cloud server) of the photoelectric conversion devices 100, 600, 700, 800. It is a figure which shows the structural example of the photoelectric conversion system 1200 provided. The communication unit 1202 may be realized by having the photoelectric conversion devices 100, 600, 700, 800 have functions such as a wireless LAN and Bluetooth (registered trademark) for communicating. Further, the communication unit 1202 may be realized by the Internet or the like in which various data that can be acquired by the user with the photoelectric conversion devices 100, 600, 700, 800 are uploaded or downloaded to the server 1201. The communication unit 1202 may be in any form as long as it can exchange data between the photoelectric conversion devices 100, 600, 700, 800 and the server 1201 arranged outside the photoelectric conversion devices 100, 600, 700, 800. May be.

For example, the trained model 160 of the signal processing device 150 of the photoelectric conversion devices 100, 600, 700, 800 may be shared among a plurality of photoelectric conversion devices 100, 600, 700, 800 via the server 1201. For example, when the photoelectric conversion devices 100, 600, 700, and 800 are connected to the server 1201 via the communication unit 1202, the learning unit 163 of the signal processing device 150 performs machine learning to obtain the trained model 160. You may update it.

Further, for example, as described above, the photoelectric conversion device may not include the signal processing device 150, and the signal processing device 150 may be arranged in the server 1201. That is, the trained model 160 generated by machine learning may be arranged on the server 1201.

By providing such a configuration, the photoelectric conversion system 1200 can update the trained model 160 by using information from a plurality of photoelectric conversion devices (for example, photoelectric conversion devices 100, 600, 700, 800). It will be possible. That is, a photoelectric conversion device used by a user who performs imaging under many wide-ranging imaging conditions is connected to the server 1201 of the photoelectric conversion system 1200. This makes it possible to acquire a large amount of teacher data in a short period of time, for example. As a result, the accuracy of the OB clamping process is further improved, and the image quality captured by each photoelectric conversion device can be further improved.

The present invention is also realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiment is supplied to a system or device via a network or various storage media, and one or more processors (for example, a CPU or a CPU) in the computer of the system or device. MPU.) Is a process to read and execute a program. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

The invention is not limited to the above embodiment, and various modifications and modifications can be made without departing from the spirit and scope of the invention. Therefore, a claim is attached to publicize the scope of the invention.

150: Signal processing device, 152: Signal processing unit, 160: Trained model, 161: Control data generation unit, 201: Light-shielding area, 202: Light-receiving area

Claims (17)

A signal processing device that processes image data output from a photoelectric conversion unit having a light receiving area and a light blocking area.
A control data generation unit that outputs control data used to generate correction data for correction of the image data using the trained model generated by machine learning.
The correction data is generated based on the light-shielding image data which is the image data of the light-shielding region and the control data of the image data, and the light-receiving image data which is the image data of the light-receiving region of the image data is used. A signal processing unit that corrects without applying the trained model according to the correction data,
A signal processing device comprising. The signal processing device according to claim 1, wherein the image data is input to the trained model. It further includes an extraction unit that extracts a feature amount for improving the accuracy of the control data from the image data.
The signal processing device according to claim 1 or 2, wherein the feature amount is input to the trained model. The feature amount is the amount of fixed pattern noise in the light-shielding image data, the shape of shading in the light-shielding image data, the size of the shading component in the light-shielding image data, and the pixel that outputs an abnormal signal value in the light-shielding image data. The signal processing apparatus according to claim 3, wherein the signal processing apparatus includes at least one of the information. The signal processing apparatus according to any one of claims 1 to 4, further comprising a learning unit that updates the trained model by machine learning using the image data. The claim is characterized in that the learning unit updates the trained model by performing machine learning using data obtained by reducing noise from the image data provided from the outside of the signal processing device as teacher data. 5. The signal processing apparatus according to 5. The learning unit updates the trained model by performing machine learning using data corresponding to the control data provided from the outside of the signal processing device, which can reduce the noise of the image data, as teacher data. The signal processing apparatus according to claim 5 or 6. It further includes an analysis unit that analyzes the degree of noise reduction of the correction data generated based on the control data.
Claims 5 to 7 are characterized in that the learning unit updates the trained model by performing machine learning using the degree of noise reduction of the correction data analyzed by the analysis unit as teacher data. The signal processing device according to any one item. The signal processing apparatus according to claim 8, wherein the machine learning is performed every time the signal processing unit generates the correction data. The control data includes information on the size of an area of the light-shielded image data for generating the correction data, information on the location of the area of the light-shielded image data for generating the correction data, and the correction data. The signal processing apparatus according to any one of claims 1 to 9, which includes at least one of information on followability to shading when generating the data. The signal processing device according to any one of claims 1 to 10.
A photoelectric conversion unit having a light receiving area and a light blocking area and outputting image data processed by the signal processing device, and a photoelectric conversion unit.
A photoelectric conversion device characterized by being provided with. The photoelectric conversion device includes a first substrate and a second substrate, and includes the first substrate and the second substrate.
The photoelectric conversion device according to claim 11, wherein the signal processing device is arranged on the first substrate, and the photoelectric conversion unit is arranged on the second board. The photoelectric conversion device according to claim 12, wherein at least a part of the first substrate and the second substrate are laminated. The photoelectric conversion device according to any one of claims 11 to 13.
A communication unit for communicating with an external server of the photoelectric conversion device, and
It is a photoelectric conversion system equipped with
A photoelectric conversion system characterized in that the trained model is shared with a trained model of a photoelectric conversion device different from the photoelectric conversion device via the server. The signal processing device according to any one of claims 1 to 10.
A plurality of photoelectric conversion devices each having a light receiving area and a light blocking area and each having a photoelectric conversion unit for outputting image data processed by the signal processing device.
A communication unit that communicates between the signal processing device and the plurality of photoelectric conversion devices, and
Photoelectric conversion system including. It is a control method of a signal processing device that processes image data output from a photoelectric conversion unit having a light receiving area and a light blocking area.
A process of outputting control data used for correction of the image data using a trained model generated by machine learning, and
A step of generating correction data based on the light-shielding image data which is the image data of the light-shielding region and the control data among the image data,
A step of correcting the light-receiving image data, which is the image data of the light-receiving region among the image data, according to the correction data, and
A method for controlling a signal processing device, which comprises. A program for causing a computer to execute each step of the control method of the signal processing apparatus according to claim 16.

Images