JP2019004358A - Imaging apparatus and imaging system - Google Patents

Abstract

To provide an imaging system or the like which acquires an image of an excellent image definition.SOLUTION: At a front side of an imaging element, an optical element is provided for rotating a polarization axis of an incident light at a predetermined angle. The optical element comprises a liquid crystal element and a polarization filter. A pixel array that the imaging element includes is divided into multiple pixel groups, the liquid crystal element is used to rotate the polarization axis at an angle that is different for each pixel group and then, an incident light passing the polarization filter is imaged continuously for each pixel group. During the continuous imaging, a reset operation is not performed in the middle. Imaging information for each pixel group is read out collectively after the end of the continuous imaging. An arithmetic section including a neural network performs comparison processing and combination processing of the pixel groups.SELECTED DRAWING: Figure 1

Description

One embodiment of the present invention relates to an imaging device and an imaging system.

Note that one embodiment of the present invention is not limited to the above technical field. One embodiment of the invention disclosed in this specification and the like relates to an object, a method, or a manufacturing method. Alternatively, one embodiment of the invention disclosed in this specification and the like relates to a process, a machine, a manufacture, or a composition (composition of matter).

More specifically, examples of the technical field of one embodiment of the present invention disclosed in this specification and the like include a display device (a liquid crystal display device, a light-emitting display device, and the like), a projection device, a lighting device, an electro-optical device, and a power storage device. A memory device, a semiconductor circuit, an imaging device, an electronic device, a driving method thereof, or a manufacturing method thereof. As an example, a semiconductor device, a display device, a liquid crystal display device, a light emitting device, a lighting device, a power storage device, a memory device, an imaging device, a driving method thereof, or a manufacturing method thereof can be given.

Note that in this specification and the like, a semiconductor device refers to any device that can function by utilizing semiconductor characteristics. A transistor and a semiconductor circuit are one embodiment of a semiconductor device. In addition, a display device, a projection device, a lighting device, an electro-optical device, a power storage device, a memory device, a semiconductor circuit, an imaging device, an electronic device, and the like are sometimes referred to as semiconductor devices. Alternatively, it may be said that these include semiconductor devices.

An optical device in which an optical element in which a liquid crystal element and a polarizing filter are combined is provided in front of an image pickup element, and a plurality of images are acquired by rotating a polarization axis of incident light by a predetermined angle to reduce reflection due to reflected light. Is known (Patent Document 1).

JP 2011-40839 A

In the method described in Patent Document 1, if the state of a subject or reflected light changes during a period during which a plurality of images are acquired (during continuous imaging), the reflected light detection accuracy decreases, and a sufficient reflection removal effect is obtained. I can't get it. Therefore, the method described in Patent Document 1 requires a reduction in imaging interval. However, in the conventional imaging method, since it is necessary to perform a reset operation and a readout operation for each imaging, it is difficult to shorten the imaging interval.

An object of one embodiment of the present invention is to provide an imaging device or the like that can operate at high speed. Another object is to provide an imaging system or the like that can operate at high speed. Another object is to provide an imaging device or the like that can acquire an image with excellent image quality. Another object is to provide an imaging system or the like that can acquire an image with excellent image quality. Another object is to provide an imaging system or the like in which the quality of captured images is favorable. Another object is to provide an imaging device with low power consumption. Another object is to provide an imaging system with low power consumption. Another object is to provide a highly reliable imaging device or the like. Another object is to provide a highly reliable imaging system or the like. Another object is to provide a novel imaging device or the like. Another object is to provide a novel imaging system or the like. Another object is to provide a novel semiconductor device or the like.

Note that the description of these problems does not disturb the existence of other problems. Note that one embodiment of the present invention does not have to solve all of these problems. Issues other than these will be apparent from the description of the specification, drawings, claims, etc., and other issues can be extracted from the descriptions of the specification, drawings, claims, etc. It is.

An optical element that rotates the polarization axis of incident light by a predetermined angle is provided in front of the imaging element. The optical element has a liquid crystal element and a polarizing filter. Divide the pixel array of the image sensor into multiple pixel groups, rotate the polarization axis at different angles for each pixel group using liquid crystal elements, and then continuously capture the incident light that has passed through the polarization filter for each pixel group To do. During continuous imaging, the reset operation is not performed halfway. The imaging information for each pixel group is read collectively after the continuous imaging is completed. Comparing processing and synthesizing processing for each pixel group are performed by an arithmetic unit including a neural network.

One embodiment of the present invention includes an optical element, an imaging element, and a calculation unit, and the optical element transmits a light in a first state or a second polarization axis that transmits light having a first polarization axis. The imaging device is operable in both two states, the imaging device has a first pixel group and a second pixel group, and the arithmetic unit is an imaging device having a function of executing learning or inference by a neural network.

The liquid crystal element included in the optical element operates in, for example, a TN mode. The relative angle between the first polarization axis and the second polarization axis is preferably less than 180 °.

The neural network is a deep neural network, a convolutional neural network, a recursive neural network, a self-encoder, a deep Boltzmann machine, or a deep belief network.

The pixel groups are preferably arranged in a checkered pattern.

According to another aspect of the present invention, there is provided an imaging system using the imaging apparatus, wherein the first imaging information is acquired by the first pixel group when the optical element is in the first state; An imaging system comprising: a step of acquiring second imaging information with the second pixel group in two states; and a step of generating third imaging information with a computing unit using the first imaging information and the second imaging information. is there.

According to one embodiment of the present invention, an imaging device or the like that can operate at high speed can be provided. Alternatively, an imaging system that can operate at high speed can be provided. Alternatively, an imaging device or the like that can acquire an image with excellent image quality can be provided. Alternatively, an imaging system or the like that can acquire an image with excellent image quality can be provided. Alternatively, an imaging system or the like in which the quality of the captured image is good can be provided. Alternatively, an imaging device or the like with low power consumption can be provided. Alternatively, an imaging system or the like with low power consumption can be provided. Alternatively, a highly reliable imaging device or the like can be provided. Alternatively, a highly reliable imaging system or the like can be provided. Alternatively, a novel imaging device or the like can be provided. Alternatively, a novel imaging system or the like can be provided. Alternatively, a novel semiconductor device or the like can be provided.

FIG. 6 illustrates a configuration example of an imaging device and an external device. FIG. 9 illustrates an imaging element. FIG. 9 illustrates an imaging element. FIG. 9 illustrates an imaging element. FIG. 9 illustrates an imaging element. FIG. 9 illustrates an imaging element. FIG. 9 illustrates an imaging element. 4A and 4B illustrate operation of an optical element and an imaging element. 7 is a flowchart for explaining an example of an imaging operation. The figure explaining the structural example of a neural network. 4A and 4B illustrate operation of an optical element and an imaging element. 7 is a flowchart for explaining an example of an imaging operation. The figure explaining the structural example of a neural network. 8A and 8B illustrate a structure example of a semiconductor device. FIG. 10 illustrates a configuration example of a memory cell. FIG. 6 illustrates a configuration example of an offset circuit. 6 is a timing chart illustrating an operation example of a semiconductor device. 3A and 3B illustrate a configuration example of a pixel circuit and a timing chart illustrating an imaging operation. FIG. 6 illustrates a configuration example of a pixel circuit. 3A and 3B illustrate a configuration example of a pixel circuit and a timing chart illustrating an imaging operation. FIG. 6 illustrates a configuration example of a pixel circuit. 2A and 2B illustrate a configuration example of an image sensor and a block diagram illustrating a circuit configuration example of the image sensor. Sectional drawing which shows the structure of an image pick-up element. Sectional drawing which shows the structure of an image pick-up element. Sectional drawing which shows the structure of an image pick-up element. The perspective view of the package which accommodated the image pick-up element. FIG. 9 illustrates a configuration example of an electronic device.

Embodiments will be described in detail with reference to the drawings. However, the present invention is not limited to the following description, and it is easily understood by those skilled in the art that modes and details can be variously changed without departing from the spirit and scope of the present invention. Therefore, the present invention should not be construed as being limited to the description of the embodiments below. Note that in the structures of the invention described below, the same portions or portions having similar functions are denoted by the same reference numerals in different drawings, and description thereof is not repeated.

In addition, the position, size, range, and the like of each component illustrated in the drawings and the like may not represent the actual position, size, range, or the like in order to facilitate understanding of the invention. Therefore, the disclosed invention is not necessarily limited to the position, size, range, or the like disclosed in the drawings and the like. For example, in an actual manufacturing process, a layer or a resist mask may be unintentionally lost due to a process such as etching, but may be omitted to facilitate understanding of the invention.

In particular, in a top view (also referred to as a “plan view”), a perspective view, and the like, some components may be omitted in order to facilitate understanding of the invention. Moreover, description of some hidden lines may be omitted.

In the present specification and the like, ordinal numbers such as “first” and “second” are used to avoid confusion between components, and do not indicate any order or order such as process order or stacking order. In addition, even in terms that do not have an ordinal number in this specification and the like, an ordinal number may be added in the claims to avoid confusion between the constituent elements. In addition, the ordinal numbers given in this specification and the like may differ from the ordinal numbers given in the claims. Even in the present specification and the like, terms with ordinal numbers are sometimes omitted in the claims.

In this specification and the like, the terms “electrode” and “wiring” do not functionally limit these components. For example, an “electrode” may be used as part of a “wiring” and vice versa. Furthermore, the terms “electrode” and “wiring” include a case where a plurality of “electrodes” and “wirings” are provided integrally.

Note that the terms “film” and “layer” can be interchanged with each other depending on the case or circumstances. For example, the term “conductive layer” may be changed to the term “conductive film”. Alternatively, for example, the term “insulating film” may be changed to the term “insulating layer” in some cases.

In this specification and the like, a transistor refers to a gate (a gate terminal or a gate electrode), a source (a source terminal, a source region, or a source electrode), and a drain (a drain terminal, a drain region, or a drain electrode) unless otherwise specified. An element having at least three terminals including, or an element having at least four terminals including a back gate (a back gate terminal or a back gate electrode). A channel formation region is provided between the source and the drain, and a current can flow between the source and the drain through the channel formation region. Note that in this specification and the like, a channel formation region refers to a region through which a current mainly flows.

The transistors described in this specification and the like are enhancement-type (normally-off) field-effect transistors unless otherwise specified. The transistors described in this specification and the like are n-channel transistors unless otherwise specified. Therefore, the threshold voltage (also referred to as “Vth”) is greater than 0 V unless otherwise specified.

Note that in this specification and the like, unless otherwise specified, Vth of a transistor having a back gate refers to Vth when the potential of the back gate is the same as that of the source or the gate.

In this specification and the like, unless otherwise specified, off-state current refers to drain current (also referred to as “Id”) when a transistor is off (also referred to as a non-conduction state or a cutoff state). . In the n-channel transistor, the potential difference between the gate and the source (also referred to as “gate voltage” or “Vg”) with respect to the source is greater than the threshold voltage unless otherwise specified. In a p-channel transistor, Vg is higher than a threshold voltage. For example, the off-state current of an n-channel transistor sometimes refers to a drain current when Vg is lower than Vth.

In the description of the off-state current, the drain may be read as the source. That is, the off-state current sometimes refers to a current that flows through the source when the transistor is off.

In this specification and the like, the term “leakage current” may be used in the same meaning as off-state current. In this specification and the like, off-state current may refer to current that flows between a source and a drain when a transistor is off, for example.

In this specification and the like, the potential VDD refers to a power supply potential that is higher than the potential VSS. Further, the potential VSS indicates a power supply potential that is lower than the potential VDD. Alternatively, the ground potential can be used as VDD or VSS. For example, when VDD is a ground potential, VSS is a potential lower than the ground potential, and when VSS is a ground potential, VDD is a potential higher than the ground potential.

In general, the “voltage” often indicates a potential difference between a certain potential and a reference potential (for example, a ground potential (GND) or a source potential). Further, the “potential” is relative, and the potential applied to the wiring or the like may change depending on the reference potential. Therefore, there are cases where “voltage” and “potential” can be paraphrased. Note that in this specification and the like, VSS is a reference potential unless otherwise specified.

In the present specification and the like, the terms “upper” and “lower” do not limit that the positional relationship between the components is directly above or directly below and is in direct contact. For example, in the expression “electrode B on the insulating layer A”, the electrode B does not need to be provided directly on the insulating layer A, and another configuration is provided between the insulating layer A and the electrode B. Do not exclude things that contain elements.

Further, in this specification and the like, “parallel” means a state in which two straight lines are arranged at an angle of −10 ° to 10 °, unless otherwise specified. Therefore, the case of −5 ° to 5 ° is also included. In addition, “substantially parallel” means a state in which two straight lines are arranged at an angle of −30 ° to 30 °, unless otherwise specified. “Vertical” and “orthogonal” refer to a state in which two straight lines are arranged at an angle of 80 ° or more and 100 ° or less, unless otherwise specified. Therefore, the case of 85 ° to 95 ° is also included. In addition, “substantially vertical” refers to a state in which two straight lines are arranged at an angle of 60 ° or more and 120 ° or less, unless otherwise specified.

In addition, in this specification, etc., the terms “same”, “same”, “equal”, “uniform” (including these synonyms), etc. with respect to the count value and the measured value, unless otherwise specified. And an error of plus or minus 20%.

In this specification and the like, an artificial neural network (ANN, hereinafter referred to as a neural network) refers to all models that model biological neural networks. In general, a neural network has a configuration in which units simulating neurons are connected to each other via units simulating synapses.

The strength of synaptic connections (connection between neurons) (also called weighting factor) can be changed by giving existing information to the neural network. In this way, the process of giving existing information to the neural network and determining the coupling strength is sometimes called “learning”.

Also, new information can be output based on the connection strength by giving some information to the neural network that has undergone "learning" (the connection strength is determined). As described above, in a neural network, a process of outputting new information based on given information and connection strength may be referred to as “inference” or “cognition”.

Examples of the neural network model include a hop field type and a hierarchical type. In particular, a neural network having a multilayer structure is referred to as “deep neural network” (DNN), and machine learning by the deep neural network is referred to as “deep learning”. The DNN includes a fully connected neural network (FC-NN: Full Connected-Neural Network), a convolutional neural network (CNN), a recurrent neural network (RNN), and the like.

(Embodiment 1)
In this embodiment, an imaging device 100 and the like of one embodiment of the present invention will be described with reference to drawings.

<Configuration example of imaging device>
FIG. 1A is a block diagram illustrating a configuration example of the imaging device 100 and the external device 200.

In the drawings attached to the present specification, the components are classified by function and the block diagram is shown as an independent block. However, it is difficult to completely separate actual components by function. A component may be related to a plurality of functions, or one function may be related to a plurality of components.

In addition, the configurations of the imaging device 100 and the external device 200 illustrated in FIG. 1A are examples, and need not include all the components. The imaging device 100 only needs to include necessary components among the components illustrated in FIG. Moreover, you may have structural elements other than the structural element shown to FIG. 1 (A). Similarly, the external device 200 only needs to include necessary components from among the components illustrated in FIG. Moreover, you may have structural elements other than the structural element shown to FIG. 1 (A).

[Imaging device 100]
The imaging apparatus 100 includes an imaging unit 101, a control unit 130, a calculation unit 140, an image processing unit 150, a storage unit 160, an auxiliary storage unit 170, an external input / output unit 180, and a communication unit 190. Note that the imaging apparatus 100 may include an optical member such as a lens, a mirror, or a prism.

The imaging unit 101 includes an optical element 110 and an imaging element 120. FIG. 1B is a perspective view illustrating the configuration of the imaging unit 101. The optical element 110 includes a liquid crystal element 111 and a polarizing filter 112. The image sensor 120 has a plurality of pixels 121 on the light receiving surface.

The optical element 110 is provided so that the polarization filter 112 faces the light receiving surface of the image sensor 120. A part of the light 501 incident on the optical element 110 passes through the liquid crystal element 111 and the polarizing filter 112 and reaches the light receiving surface of the imaging element 120, and is converted into an electric signal by the pixel 121.

The liquid crystal element 111 has a function of rotating the polarization axis of the light 501 in accordance with the voltage applied to the liquid crystal element 111. Generally, materials that operate in various operation modes such as a TN (Twisted Nematic) mode, an IPS (In-Plane-Switching) mode, and a VA (Vertical Alignment) mode are known as liquid crystal materials used for a liquid crystal element. In this embodiment, a liquid crystal element that operates in a TN (Twisted Nematic) mode is used as the liquid crystal element 111.

The polarizing filter 112 has a function of transmitting light having a specific polarization angle (direction of the polarization axis). The optical element 110 has a function of adjusting the transmittance of light having an arbitrary polarization angle by controlling the voltage applied to the liquid crystal element 111.

The control unit 130 (Controller) has a function of controlling operations related to imaging. The control unit 130 controls operations of the optical element 110 and the image sensor 120. The calculation unit 140 has a function of performing calculations related to the operation of the entire imaging apparatus, and for example, a central processing unit (CPU) can be used.

Further, the control unit 130 or the calculation unit 140 may be realized by a PLD (Programmable Logic Device) such as an FPGA (Field Programmable Gate Array) or an FPAA (Field Programmable Analog Array).

The image processing unit 150 has a function of performing data processing relating to an image, and for example, an image processing device (GPU: Graphics Processing Unit) or the like can be used. Further, the image processing unit 150 includes a neural network 151 for generating image data. The neural network 151 may be configured by software. The image processing unit 150 has a function of comparing or calculating video information acquired by the image sensor 120.

The storage unit 160 preferably has a function of storing programs and setting items relating to the operation of the imaging apparatus 100, and at least a part thereof is preferably a rewritable memory. For example, the storage unit 160 may include a volatile memory such as a RAM (Random Access Memory) and a nonvolatile memory such as a ROM (Read Only Memory).

As the RAM provided in the storage unit 160, for example, a DRAM (Dynamic Random Access Memory) is used, and a memory space is virtually allocated and used as a work space of the storage unit 160. The operating system, application program, program module, program data, etc. stored in the auxiliary storage unit 170 are loaded into the RAM for execution. These data, programs, and program modules loaded in the RAM are directly accessed and operated by the arithmetic unit 140.

On the other hand, the ROM can store BIOS (Basic Input / Output System), firmware and the like that do not require rewriting. As the ROM, a mask ROM, an OTPROM (One Time Programmable Read Only Memory), an EPROM (Erasable Programmable Read Only Memory), or the like can be used. Examples of EPROM include UV-EPROM (Ultra-Violet Erasable Programmable Read Only Memory) and EEPROM (Electrically Erasable Programmable Read Only Memory) capable of erasing stored data by ultraviolet irradiation.

The auxiliary storage unit 170 is a storage device for storing data such as captured images. In addition, teacher data used in the neural network 151 is stored. In addition, as described above, the auxiliary storage unit 170 can store an operating system, application programs, program modules, program data, and the like.

As the auxiliary storage unit 170, for example, a non-volatile storage element such as a flash memory, an MRAM (Magnetorative Random Access Memory), a PRAM (Phase change RAM), a ReRAM (Resistive RAM), or a FeRAM (Ferroelectric RAM) is applied. A device or a storage device to which a volatile storage element such as a DRAM (Dynamic RAM) or SRAM (Static RAM) is applied may be used. For example, a recording medium drive such as a hard disk drive (HDD) or a solid state drive (SSD) may be used.

In addition, for example, a storage device such as an HDD or an SSD that can be attached / detached via the external input / output unit 180, or a media drive of a recording medium such as a flash memory, a Blu-ray disc, or a DVD can be used as the auxiliary storage unit 170. Note that the auxiliary storage unit 170 may not be built in the imaging apparatus 100, and a storage device placed outside the imaging apparatus 100 may be used as the auxiliary storage unit 170. In that case, it may be configured to be connected via the external input / output unit 180 or to exchange data by wireless communication by the communication unit 190.

The external input / output unit 180 has a function of controlling signal input / output between the external device 200 and the imaging apparatus 100. As an external port of the external input / output unit 180, an HDMI (registered trademark) terminal, a USB terminal, a LAN (Local Area Network) connection terminal, or the like may be used. The external input / output unit 180 may have a transmission / reception function for optical communication using infrared rays, visible light, ultraviolet rays, or the like.

The communication unit 190 can perform communication via an antenna. For example, a control signal for connecting the imaging apparatus 100 to the computer network is controlled in accordance with a command from the arithmetic unit 140, and the signal is transmitted to the computer network. As a result, the Internet, intranet, extranet, PAN (Personal Area Network), LAN (Local Area Network), CAN (Camper Area Network, and MAN (MetroApolNetwork), which are the foundations of the World Wide Web (WWW). Communication can be performed by connecting the imaging apparatus 100 to a computer network such as a wide area network (GAE) or a GAN (global area network). When a plurality of methods are used as the communication method, a plurality of antennas may be provided depending on the communication method.

The communication unit 190 may be provided with, for example, a high frequency circuit (RF circuit) to transmit and receive RF signals. The high-frequency circuit is a circuit for mutually converting an electromagnetic signal and an electric signal in a frequency band determined by the legislation of each country and performing communication with other communication devices wirelessly using the electromagnetic signal. Several tens of kHz to several tens of GHz is generally used as a practical frequency band. The high-frequency circuit connected to the antenna includes a high-frequency circuit unit corresponding to a plurality of frequency bands, and the high-frequency circuit unit includes an amplifier (amplifier), a mixer, a filter, a DSP (Digital Signal Processor), an RF transceiver, and the like. It can be configured. When performing wireless communication, as communication protocols or communication technologies, LTE (Long Term Evolution), GSM (Global System for Mobile Communications: registered trademark), EDGE (Enhanced Data Rates for GSM Evolution), CDMA2000 (CDMA 2000), CDMA2000 (CD2000). Use communication standards such as WCDMA (Wideband Code Division Multiple Access: registered trademark), or specifications standardized by IEEE such as Wi-Fi (registered trademark), Bluetooth (registered trademark), ZigBee (registered trademark), etc. Can do.

[External device 200]
The external device 200 includes a control unit 230, a calculation unit 240, an image processing unit 250, a storage unit 270, an auxiliary storage unit 280, an external input / output unit 290, and the like. The image processing unit 250 has a configuration equivalent to that of the neural network 151. A neural network 251 is included. Note that the neural network 151 and the neural network 251 may be configured by software.

[Image sensor 120]
FIG. 2A shows the front surface (light receiving surface) of the image sensor 120. On the light receiving surface of the image sensor 120, a plurality of pixels are arranged in a matrix of m rows and n columns (m and n are both integers of 2 or more). In this specification and the like, the pixel 121 in the first row and first column is referred to as a pixel 121 [1, 1], and the pixel 121 in the m row and n column is referred to as a pixel 121 [m, n]. A pixel 121 in the i-th row and j-th column (i is an integer of 1 to m. J is an integer of 1 to n) is denoted as a pixel 121 [i, j]. In addition, when a description common to all the pixels 121 is performed, the pixel 121 may be simply indicated.

A plurality of pixels 122 functioning as subpixels may be provided in the pixel 121. By providing each of the plurality of pixels 122 with a filter (color filter) that transmits light in different wavelength ranges, information for realizing color image display can be acquired.

FIG. 2B is a plan view illustrating an example of the pixel 121 for obtaining a color image. FIG. 2A illustrates a pixel 122 (hereinafter also referred to as “pixel 122R”) provided with a color filter that transmits light in the red (R) wavelength region, and light in the green (G) wavelength region. A pixel 122 provided with a color filter (hereinafter also referred to as “pixel 122G”) and a pixel 122 provided with a color filter that transmits light in the blue (B) wavelength range (hereinafter also referred to as “pixel 122B”). Have. The pixel 122R, the pixel 122G, and the pixel 122B are combined to function as one pixel 121.

Note that the color filters used for the pixels 122 are not limited to red (R), green (G), and blue (B), and as shown in FIG. 2C, cyan (C), yellow (Y), and yellow (Y), respectively. A color filter that transmits magenta (M) light may be used. A full-color image can be acquired by providing a pixel 122 that detects light of three different wavelength ranges in one pixel 121.

FIG. 2D shows a color filter that transmits yellow (Y) light in addition to the pixel 122 provided with a color filter that transmits red (R), green (G), and blue (B) light, respectively. The pixel 121 having the pixel 122 provided with is illustrated. FIG. 2E shows a color filter that transmits blue (B) light in addition to the pixel 122 provided with a color filter that transmits cyan (C), yellow (Y), and magenta (M) light, respectively. The pixel 121 having the pixel 122 provided with is illustrated. By providing the pixel 122 that detects light of four different wavelength ranges in one pixel 121, the color reproducibility of the acquired image can be further enhanced.

For example, the pixel number ratio (or the light receiving area ratio) of the pixels 122R, 122G, and 122B is not necessarily 1: 1: 1. As shown in FIG. 2F, a Bayer arrangement may be employed in which the pixel number ratio (light receiving area ratio) is red: green: blue = 1: 2: 1. The pixel number ratio (light receiving area ratio) may be red: green: blue = 1: 6: 1.

Note that one pixel 122 may be provided in the pixel 121, but two or more are preferable. For example, by providing two or more pixels 122 that detect light in the same wavelength region, redundancy can be increased and the reliability of the imaging apparatus 100 can be increased.

[Multiple pixel groups]
The plurality of pixels 121 included in the imaging element 120 function by being divided into a plurality of pixel groups. 3 to 7 are enlarged views of a part of the image sensor 120. 3 to 6 show an example in which a plurality of pixels 121 function by being divided into two pixel groups, a first pixel group and a second pixel group. FIG. 7 illustrates an example in which a plurality of pixels 121 included in the image sensor 120 function by being divided into three pixel groups, a first pixel group, a second pixel group, and a third pixel group. In FIG. 3 to FIG. 7, “A” is attached to the pixel 121 belonging to the first pixel group, and “B” is attached to the pixel 121 belonging to the second pixel group. In FIG. 7, “C” is given to the pixel 121 belonging to the third pixel group.

Note that the plurality of pixels 121 included in the imaging element 120 may function by being divided into four or more pixel groups. The pixels 121 included in each pixel group are arranged uniformly or substantially uniformly on the learning surface of the image sensor 120.

FIG. 3A shows a state in which the pixels 121 belonging to the first pixel group and the pixels 121 belonging to the second pixel group are arranged in stripes every other column. In FIG. 3A, the pixels 121 belonging to the first pixel group are arranged in odd columns and the pixels 121 belonging to the second pixel group are arranged in even columns, but the pixels 121 belonging to the first pixel group are arranged in even columns. It does not matter. FIG. 3B shows a state where the pixels 121 belonging to the first pixel group and the pixels 121 belonging to the second pixel group are arranged in stripes every two columns. Although not shown, the pixels 121 belonging to the first pixel group and the pixels 121 belonging to the second pixel group may be arranged every three or more columns.

FIG. 4A shows a state where the pixels 121 belonging to the first pixel group and the pixels 121 belonging to the second pixel group are arranged in stripes every other row. In FIG. 3A, the pixels 121 belonging to the first pixel group are arranged in odd rows and the pixels 121 belonging to the second pixel group are arranged in even rows, but the pixels 121 belonging to the first pixel group are arranged in even rows. It does not matter. FIG. 3B shows a state where the pixels 121 belonging to the first pixel group and the pixels 121 belonging to the second pixel group are arranged in stripes every two rows. Although not shown, the pixels 121 belonging to the first pixel group and the pixels 121 belonging to the second pixel group may be arranged every three or more rows.

FIG. 5A shows a state in which the pixels 121 belonging to the first pixel group and the pixels 121 belonging to the second pixel group are arranged in a checkered pattern every other pixel. FIG. 5B shows a state where the pixels 121 belonging to the first pixel group and the pixels 121 belonging to the second pixel group are arranged in a checkered pattern every two pixels. In this case, four pixels 121 belonging to the same pixel group are adjacent to each other, and the four pixels 121 can function as one subgroup. Although not shown, the pixel 121 belonging to the first pixel group and the pixel 121 belonging to the second pixel group may be arranged every three or more pixels.

Further, as shown in FIG. 6A or FIG. 6B, three pixels 121 belonging to the same pixel group may be arranged in an L shape. The three pixels 121 can function as one subgroup.

In FIG. 7A, the pixel 121 belonging to the first pixel group, the pixel 121 belonging to the second pixel group, and the pixel 121 belonging to the third pixel group are sequentially arranged for each pixel in the row direction and the column direction, respectively. It shows how it is being done. Accordingly, in FIG. 7A, the pixels 121 belonging to the same pixel group are not adjacent to each other in the row direction and the column direction. Further, as shown in FIG. 7B, one subgroup may be formed by four pixels 121 belonging to the same pixel group, and each subgroup may be sequentially arranged in the row direction and the column direction. Similarly to FIG. 7A, in FIG. 7B, subgroups belonging to the same pixel group are not adjacent to each other in the row direction and the column direction.

<Example of imaging operation of imaging device>
An example of an imaging operation of the imaging apparatus 100 will be described with reference to the drawings.

[Imaging Operation Example 1]
A case will be described in which the imaging element 120 includes a first pixel group and a second pixel group, and each pixel group is arranged in a checkered pattern as shown in FIG. The imaging device 100 of one embodiment of the present invention can efficiently reduce reflection caused by reflected light.

8A and 8B are diagrams illustrating the operation of the optical element 110 and the image sensor 120. FIG. In FIG. 8, “A” is assigned to the pixel 121 belonging to the first pixel group, and “B” is assigned to the pixel 121 belonging to the second pixel group. In FIGS. 8A and 8B, an image incident on the image sensor 120 via the optical element 110 is shown as light 501. FIG. 9 is a flowchart for explaining an example of the imaging operation of the imaging apparatus 100.

First, the reset operation of the image sensor 120 is performed (step S600).

Next, a predetermined voltage is applied to the liquid crystal element 111 to determine the angle of the polarization axis of light that can be transmitted through the optical element 110 (step S605). The state of the optical element 110 determined at this time is referred to as a first state. The angle of the polarization axis determined at this time is referred to as a first polarization angle.

Next, an image incident through the optical element 110 is picked up using the first pixel group (step S610) (see FIG. 8A). Imaging information acquired by the first pixel group is held in the first pixel group as first imaging information. In step S610, the second pixel group is not imaged. That is, in step S610, the second pixel group holds the state after step S600.

Next, a voltage different from the first state is applied to the liquid crystal element 111 to determine the polarization axis of light that can be transmitted through the optical element 110 (step S615). The state of the optical element 110 determined at this time is referred to as a second state. The angle of the polarization axis determined at this time is referred to as a second polarization angle. The second polarization angle is a polarization axis rotated by an angle θ from the first polarization angle. The angle θ may be larger than 0 ° and smaller than 180 °. That is, the relative angle between the first polarization angle and the second polarization angle may be less than 180 °.

In FIG. 8B, the angle θ is 90 °. Therefore, when the first polarization angle shown in FIG. 8A is 0 °, the second polarization angle shown in FIG. 8B is 90 °.

Next, an image incident through the optical element 110 is picked up using the second pixel group (step S620) (see FIG. 8B). The imaging information acquired by the second pixel group is held in the second pixel group as second imaging information. In step S620, imaging is not performed with the first pixel group. That is, in step S620, the first pixel group holds the first imaging information.

Next, the first imaging information held in the first pixel group and the second imaging information held in the second pixel group are read (step S625). The read imaging information is stored in the storage unit 160 (step S630).

Next, the image processing unit 150 compares the first imaging information and the second imaging information, and selects the darkest imaging information (imaging information with the lowest average luminance) (step S635).

The light 501 includes an image that is originally intended for photographing and reflected light that causes unnecessary reflection. When light is reflected at the interface between the atmosphere and glass, the polarization component (P wave) parallel to the reflection surface is weakened and the polarization component (S wave) perpendicular to the reflection surface is strengthened. As described above, the optical element 110 has a function of adjusting the transmittance of light having an arbitrary polarization angle by controlling the voltage applied to the liquid crystal element 111. In particular, the reflected light has a strong specific polarization component. The transmittance of the reflected light is easily controlled by the optical element 110.

Therefore, unnecessary light components included in the light 501 can be reduced by allowing the light 501 to pass through the optical element 110. That is, it can be said that the same subject is photographed while changing the polarization axis of the liquid crystal element 111 included in the optical element 110, and the darkest image (image with the lowest average luminance) is the image with the least amount of reflected light. In addition, the imaging device of one embodiment of the present invention can reduce reflected light incident on polarization axes at various angles by controlling a voltage applied to the liquid crystal element 111.

In the case of continuously capturing images without dividing the image sensor into a plurality of pixel groups, it is necessary to perform a read operation and a reset operation for each image capture. That is, after imaging in the first state (step S610), it is necessary to read the imaging information and reset the imaging element. In this case, since the interval from the imaging in the first state to the imaging in the second state becomes long, the situation of the subject and reflected light changes, and the detection accuracy of an image with little reflected light decreases.

In the imaging device 100 of one embodiment of the present invention, the imaging element can be divided into a plurality of pixel groups, and imaging in the first state and imaging in the second state can be performed with different pixel groups. Further, the imaging information can be read out collectively after the completion of all imaging. In the imaging device 100 of one embodiment of the present invention, it is not necessary to read out imaging information and reset the imaging element for each imaging. That is, the imaging interval from imaging in the first state to imaging in the second state can be shortened. Therefore, an image with less reflected light can be detected more accurately.

When the resolution (the number of pixels) of the image sensor 120 is sufficiently higher than the required resolution, there is no substantial problem even if the image sensor is divided into a plurality of pixel groups. However, if the image sensor is divided into a plurality of pixel groups and used, the resolution of one captured image is lowered and the image quality is lowered. Moreover, when there are few pixel groups to divide | segment, the removal of reflected light may be inadequate. In such a case, the captured image can be complemented using the neural network 151 included in the image processing unit 150 to improve the image quality of the captured image.

First, it is determined whether or not the supplement processing is performed based on the user's determination or presetting (step S640). If not, the imaging information selected in step S635 is stored in the auxiliary storage unit 170 (step S650).

When performing the complementing process, the neural network 151 performs the complementing process (step S645). The neural network 151 has a function of compensating for a decrease in resolution caused by dividing the image sensor into a plurality of pixel groups and generating new image information having the same resolution as the image sensor. Further, it has a function of removing reflected light that could not be removed and generating new imaging information.

For example, when the darkest imaging information selected in step S635 is the first imaging information, the weighting coefficient set in the neural network 151 for performing the complementing process is stored in the second imaging information and the auxiliary storage unit 170. Is determined using the teacher data.

Note that it is also possible to perform complementation processing using a weighting coefficient determined only by teacher data. However, by adding not only the teacher data but also the second imaging information to the determination of the weighting factor, the imaging information can be corrected with higher accuracy. This is particularly effective when there are three or more pixel groups to be used.

Here, a configuration example of the neural network 151 will be described (see FIG. 10). The neural network 151 includes an input layer IL, an intermediate layer HL1 (hidden layer), an intermediate layer HL2 (hidden layer), and an output layer OL. In the neural network 151, a hierarchical neural network is constituted by the input layer IL, the intermediate layer HL1, the intermediate layer HL2, and the output layer OL. The intermediate layer HL1 and the intermediate layer HL2 have an arbitrary number of nodes. The intermediate layer is not limited to two layers. The intermediate layer may be one layer or three or more layers.

The first imaging information 301 and the second imaging information 301 are input to the input layer IL, and weighted information is input to the intermediate layer HL1. The information input to the intermediate layer HL1 is weighted and input to the intermediate layer HL2. Further, the information input to the intermediate layer HL2 is weighted and input to the output layer OL. The third imaging information 303 is output from the output layer OL.

The neural network 151 has a configuration in which the number of neurons increases as the hierarchy progresses. That is, the number of neurons included in the intermediate layer HL1 is larger than the number of neurons included in the input layer IL, and the number of neurons included in the intermediate layer HL2 is larger than the number of neurons included in the intermediate layer HL1. Further, the number of neurons included in the output layer OL is larger than the number of neurons included in the intermediate layer HL2. In FIG. 10, the number of neurons is indicated by the number of arrows connecting the respective layers. By generating the neural network 151 so that the number of neurons increases each time the hierarchy advances, the third imaging information 303 with increased resolution is generated based on the first imaging information 301 and the second imaging information 301. Can do. Further, based on the first imaging information 301 and the second imaging information 301, the third imaging information 303 with an increased number of gradations can be generated.

Hierarchical neural networks can be fully connected between layers, or partially connected between layers. Moreover, it can be set as the structure which used the convolution layer and the pooling layer between each layer, ie, CNN.

The weighting factor used by the neural network 151 may be a weighting factor determined by the external device 200. For example, the imaging apparatus 100 and the external device 200 are connected via the external input / output unit 180 and the external input / output unit 290, and the weighting factor determined by the neural network 251 of the external device 200 is stored in the neural network 151. Therefore, the neural network 151 can perform the same operation as the learned neural network 251.

For example, before the image pickup apparatus 100 is shipped from the factory, the weighting coefficient determined after learning with the external device 200 is stored in the image pickup apparatus 100, so that the learning work by the user can be made unnecessary.

Further, learning by the external device 200 may be continuously performed, and the updated weighting coefficient may be stored in the imaging apparatus 100. In addition, a plurality of external devices 200 may be used to generate a weighting factor for update. The transfer of the weighting coefficient can also be performed via a recording medium such as an SD card, various communication means, or the like. Also, a new weighting factor may be determined using the weighting factor of the imaging apparatus 100 and the weighting factor updated by the external device 200. In addition, a new weighting factor may be determined using a weighting factor included in another imaging apparatus 100. By using a weighting coefficient obtained by learning with another imaging apparatus 100 or external device 200, it is possible to perform complementary processing with higher accuracy.

The third imaging information generated by the neural network 151 is stored in the auxiliary storage unit 170 (step S650).

Alternatively, the first imaging information and the second imaging information may be used to determine the first polarization angle and the second polarization angle again by the calculation unit 140, and the imaging operation may be performed again. By repeating the correction of the polarization angle and the imaging operation, an image with better image quality can be acquired.

[Imaging Operation Example 2]
Next, the case where the image sensor 120 has a first pixel group, a second pixel group, and a third pixel group, and each pixel group is arranged as shown in FIG. 7A will be described. By increasing the number of pixel groups and increasing the state of the liquid crystal element 111 corresponding to each pixel group, it is possible to more efficiently reduce the reflection due to the reflected light.

FIGS. 11A, 11 </ b> B, and 11 </ b> C are diagrams for explaining operations of the optical element 110 and the imaging element 120. FIG. 12 is a flowchart for explaining an example of the imaging operation of the imaging apparatus 100. In FIG. 11, “A” is attached to the pixel 121 belonging to the first pixel group, “B” is attached to the pixel 121 belonging to the second pixel group, and “C” is attached to the pixel 121 belonging to the third pixel group. doing. In order to reduce the repetition of the description, differences from the operation example 1 will be mainly described.

Steps S600 to S620 are performed in the same manner as in Operation Example 1 (see FIGS. 11A and 11B and FIG. 12). Here, the second polarization angle is set to 60 °, for example.

Next, a voltage different from the first state and the second state is applied to the liquid crystal element 111 to determine the polarization axis of light that can be transmitted through the optical element 110 (step S621). The state of the optical element 110 determined at this time is referred to as a third state. The angle of the polarization axis determined at this time is referred to as a third polarization angle. The third polarization angle is a polarization axis rotated by an angle θ from the first polarization angle. The angle θ of the third polarization angle may be larger than 0 ° and smaller than 180 °. That is, the relative angle between the first polarization angle and the third polarization angle may be less than 180 °. Further, the angle θ of the third polarization angle is larger than the angle θ of the second polarization angle. Here, the third polarization angle is set to 120 °, for example.

Next, an image incident through the optical element 110 is picked up using the third pixel group (step S622) (see FIG. 11C). Imaging information acquired by the third pixel group is held in the third pixel group as third imaging information. In step S620, imaging is not performed on the first pixel group and the second pixel group. That is, in step S620, the first pixel group holds the first imaging information, and the second pixel group holds the second imaging information.

Similar to the first operation example, the imaging operation in the first state to the imaging operation in the third state is executed without performing the reading operation and the resetting operation on the way. By not performing the reset operation in the middle, the interval from imaging in the first state to imaging in the third state can be shortened.

Next, the first imaging information held in the first pixel group, the second imaging information held in the second pixel group, and the third imaging information held in the third pixel group are read (Step S1). S625). The read imaging information is stored in the storage unit 160 (step S630).

Next, the image processing unit 150 compares the first to third imaging information, and selects the darkest imaging information (imaging information with the lowest average luminance) (step S635).

Next, it is determined whether or not to perform the supplement processing by the user's determination or presetting (step S640). If not, the imaging information selected in step S635 is stored in the auxiliary storage unit 170 (step S650). .

When performing the complementing process, the neural network 151 performs the complementing process (step S645).

For example, when the darkest imaging information selected in step S635 is the first imaging information, the weighting factors set in the neural network 151 for performing the complement processing are the second imaging information, the third imaging information, and the auxiliary information. It is determined using the teacher data stored in the storage unit 170.

When the number of pixel groups used for detection of reflected light is increased, the resolution (number of pixels) per pixel group decreases. As described above, it is possible to perform the complementing process using the weighting factor determined only by the teacher data. However, not only the teacher data but also the second imaging information and the third imaging information are added to the determination of the weighting factor. Thus, the imaging information can be corrected with higher accuracy. That is, by using the imaging information other than the first imaging information for the complementing process by the neural network 151, it is possible to improve the accuracy of the complementing process and generate an image with good image quality with high reproducibility.

In the case of the operation example 2, the first imaging information, the second imaging information, and the third imaging information are input to the input layer IL of the neural network 151, and output as the fourth imaging information from the output layer OL through the intermediate layer. .

The fourth imaging information generated by the neural network 151 is stored in the auxiliary storage unit 170 (step S650).

In addition, using the first imaging information, the second imaging information, and the third imaging information, the calculation unit 140 determines the first polarization angle, the second polarization angle, and the third polarization angle again, and performs the imaging operation again. May be. By repeating the correction of the polarization angle and the imaging operation, an image with better image quality can be acquired.

This embodiment can be implemented in appropriate combination with the structures described in the other embodiments.

(Embodiment 2)
In this embodiment, a configuration example of a semiconductor device that can be used for the neural network described in the above embodiment will be described.

As shown in FIG. 13A, the neural network NN can be configured by an input layer IL, an output layer OL, and an intermediate layer (hidden layer) HL. Each of the input layer IL, the output layer OL, and the intermediate layer HL has one or a plurality of neurons (units). The intermediate layer HL may be one layer or two or more layers. A neural network having two or more intermediate layers HL can also be called DNN (deep neural network), and learning using a deep neural network can also be called deep learning.

Input data is input to each neuron of the input layer IL, an output signal of a neuron of the previous layer or the rear layer is input to each neuron of the intermediate layer HL, and an output of the neuron of the previous layer is input to each neuron of the output layer OL. A signal is input. Each neuron may be connected to all neurons in the preceding and following layers (total connection), or may be connected to some neurons.

FIG. 13B shows an example of computation by neurons. Here, a neuron N and two neurons in the previous layer that output signals to the neuron N are shown. Neurons N includes an output x ₁ of the neurons in the previous layer, the output x ₂ of neurons prior layer is inputted. Then, the neurons N, the output _{x 1} and the sum _x 1 _w 1 ₊ x 2 _{w 2} weight _{w 1} of the multiplication result _(x 1 _{w 1)} and the output _{x 2} and the weight _{w 2} of the multiplication result _(x 2 _{w 2)} After being calculated, the bias b is added as necessary to obtain the value a = x ₁ w ₁ + x ₂ w ₂ + b. The value a is converted by the activation function h, and the output signal y = h (a) is output from the neuron N.

As described above, the operation by the neuron includes an operation of adding the product of the output of the neuron in the previous layer and the weight, that is, a product-sum operation (the above x ₁ w ₁ + x ₂ w ₂ ). This product-sum operation may be performed on software using a program, or may be performed by hardware. When the product-sum operation is performed by hardware, a product-sum operation circuit can be used. As this product-sum operation circuit, a digital circuit or an analog circuit may be used. When an analog circuit is used for the product-sum operation circuit, the processing speed can be improved and the power consumption can be reduced by reducing the circuit scale of the product-sum operation circuit or reducing the number of accesses to the memory.

The product-sum operation circuit may be formed using a transistor (also referred to as a “Si transistor”) containing silicon (such as single crystal silicon) in a channel formation region, or an oxide semiconductor that is a kind of metal oxide in the channel formation region. A transistor including a transistor (also referred to as an “OS transistor”) may be used. In particular, an OS transistor has a very small off-state current, and thus is suitable as a transistor constituting an analog memory of a product-sum operation circuit. Note that the product-sum operation circuit may be configured using both the Si transistor and the OS transistor. Hereinafter, a configuration example of a semiconductor device having the function of the product-sum operation circuit will be described.

<Configuration example of semiconductor device>
FIG. 14 shows a configuration example of a semiconductor device MAC having a function of performing a neural network operation. The semiconductor device MAC has a function of performing a product-sum operation on the first data corresponding to the connection strength (weight) between the neurons and the second data corresponding to the input data. Note that the first data and the second data can be analog data or multivalued data (discrete data), respectively. Further, the semiconductor device MAC has a function of converting data obtained by the product-sum operation using an activation function.

The semiconductor device MAC includes a cell array CA, a current source circuit CS, a current mirror circuit CM, a circuit WDD, a circuit WLD, a circuit CLD, an offset circuit OFST, and an activation function circuit ACTV.

The cell array CA includes a plurality of memory cells MC and a plurality of memory cells MCref. In FIG. 14, the cell array CA has m rows and n columns (m and n are integers of 1 or more) of memory cells MC (MC [1,1] to [m, n]) and m memory cells MCref (MCref The example of a structure which has [1] thru | or [m]) is shown. Memory cell MC has a function of storing first data. The memory cell MCref has a function of storing reference data used for product-sum operation. The reference data can be analog data or multi-value data.

The memory cell MC [i, j] (i is an integer of 1 to m, j is an integer of 1 to n) includes the wiring WL [i], the wiring RW [i], the wiring WD [j], and the wiring BL [J]. The memory cell MCref [i] is connected to the wiring WL [i], the wiring RW [i], the wiring WDref, and the wiring BLref. Here, a current flowing between the memory cell MC [i, j] and the wiring BL [ _j] is expressed as I _{MC [i, j],} and a current flowing between the memory cell MCref [i] and the wiring BLref is expressed as I _{MCref [ i]} .

A specific configuration example of the memory cell MC and the memory cell MCref is shown in FIG. FIG. 15 shows memory cells MC [1,1], [2,1] and memory cells MCref [1], [2] as representative examples, but the same applies to other memory cells MC and memory cells MCref. Can be used. Each of the memory cell MC and the memory cell MCref includes transistors Tr11 and Tr12 and a capacitor C11. Here, the case where the transistors Tr11 and Tr12 are n-channel transistors will be described.

In the memory cell MC, the gate of the transistor Tr11 is connected to the wiring WL, one of the source and the drain is connected to the gate of the transistor Tr12 and the first electrode of the capacitor C11, and the other of the source or the drain is connected to the wiring WD. Has been. One of a source and a drain of the transistor Tr12 is connected to the wiring BL, and the other of the source and the drain is connected to the wiring VR. The second electrode of the capacitor C11 is connected to the wiring RW. The wiring VR is a wiring having a function of supplying a predetermined potential. Here, as an example, a case where a low power supply potential (such as a ground potential) is supplied from the wiring VR will be described.

A node connected to one of the source and the drain of the transistor Tr11, the gate of the transistor Tr12, and the first electrode of the capacitor C11 is a node NM. The nodes NM of the memory cells MC [1,1] and [2,1] are denoted as nodes NM [1,1] and [2,1], respectively.

The memory cell MCref has the same configuration as the memory cell MC. However, the memory cell MCref is connected to the wiring WDref instead of the wiring WD, and is connected to the wiring BLref instead of the wiring BL. In the memory cells MCref [1] and [2], a node connected to one of the source and the drain of the transistor Tr11, the gate of the transistor Tr12, and the first electrode of the capacitor C11 is a node NMref [1]. , [2].

The node NM and the node NMref function as a memory cell MC and a holding node for the memory cell MCref, respectively. The node NM holds first data, and the node NMref holds reference data. Further, currents IMC _[1,1] and _{IMC [2,1]} flow from the wiring BL [1] to the transistors Tr12 of the memory cells MC [1,1] and [2,1], respectively. Further, currents I _{MCref [1]} and I _{MCref [2]} flow from the wiring BLref to the transistors Tr12 of the memory cells MCref [1] and [2], respectively.

Since the transistor Tr11 has a function of holding the potential of the node NM or the node NMref, the off-state current of the transistor Tr11 is preferably small. Therefore, an OS transistor with an extremely low off-state current is preferably used as the transistor Tr11. As a result, fluctuations in the potential of the node NM or the node NMref can be suppressed, and the calculation accuracy can be improved. In addition, the frequency of the operation of refreshing the potential of the node NM or the node NMref can be suppressed, and power consumption can be reduced.

The transistor Tr12 is not particularly limited, and for example, a Si transistor or an OS transistor can be used. When an OS transistor is used as the transistor Tr12, the transistor Tr12 can be manufactured using the same manufacturing apparatus as the transistor Tr11, and manufacturing cost can be reduced. Note that the transistor Tr12 may be an n-channel type or a p-channel type.

The current source circuit CS is connected to the wirings BL [1] to [n] and the wiring BLref. The current source circuit CS has a function of supplying current to the wirings BL [1] to [n] and the wiring BLref. Note that the current value supplied to the wirings BL [1] to [n] may be different from the current value supplied to the wiring BLref. Here, the current supplied from the current source circuit CS to the wirings BL [1] to [n] is expressed as I _C , and the current supplied from the current source circuit CS to the wiring BLref is expressed as I _Cref .

The current mirror circuit CM includes wirings IL [1] to [n] and a wiring ILref. The wirings IL [1] to [n] are connected to the wirings BL [1] to [n], respectively, and the wiring ILref is connected to the wiring BLref. Here, connection points between the wirings IL [1] to [n] and the wirings BL [1] to [n] are denoted as nodes NP [1] to [n]. Further, a connection point between the wiring ILref and the wiring BLref is referred to as a node NPref.

The current mirror circuit CM has a function of flowing a current I _CM corresponding to the potential of the node NPref to the wiring ILref and a function of flowing the current I _CM to the wirings IL [1] to [n]. Figure 14 is discharged current _{I CM} from the wiring BLref to the wiring ILref, wiring BL [1] to the wiring from the [n] IL [1] to [n] to the current _{I CM} is an example to be discharged . Further, currents flowing from the current mirror circuit CM to the cell array CA via the wirings BL [1] to [n] are denoted as I _B [1] to [n]. A current flowing from the current mirror circuit CM to the cell array CA via the wiring _BLref is denoted as I _Bref .

The circuit WDD is connected to the wirings WD [1] to [n] and the wiring WDref. The circuit WDD has a function of supplying a potential corresponding to the first data stored in the memory cell MC to the wirings WD [1] to [n]. Further, the circuit WDD has a function of supplying a potential corresponding to reference data stored in the memory cell MCref to the wiring WDref. The circuit WLD is connected to the wirings WL [1] to [m]. The circuit WLD has a function of supplying a signal for selecting the memory cell MC or the memory cell MCref to which data is written to the wirings WL [1] to [m]. The circuit CLD is connected to the wirings RW [1] to [m]. The circuit CLD has a function of supplying a potential corresponding to the second data to the wirings RW [1] to [m].

The offset circuit OFST is connected to the wirings BL [1] to [n] and the wirings OL [1] to [n]. The offset circuit OFST has a function of detecting a current amount flowing from the wirings BL [1] to [n] to the offset circuit OFST and / or a change amount of a current flowing from the wirings BL [1] to [n] to the offset circuit OFST. Have The offset circuit OFST has a function of outputting the detection result to the wirings OL [1] to [n]. Note that the offset circuit OFST may output a current corresponding to the detection result to the wiring OL, or may convert a current corresponding to the detection result into a voltage and output the voltage to the wiring OL. The current flowing between the cell array CA and the offset circuit OFST is expressed as I _α [1] to [n].

A configuration example of the offset circuit OFST is shown in FIG. The offset circuit OFST illustrated in FIG. 16 includes circuits OC [1] to [n]. Each of the circuits OC [1] to [n] includes a transistor Tr21, a transistor Tr22, a transistor Tr23, a capacitor C21, and a resistor R1. The connection relationship of each element is as shown in FIG. Note that a node connected to the first electrode of the capacitor C21 and the first terminal of the resistor element R1 is referred to as a node Na. A node connected to the second electrode of the capacitor C21, one of the source or the drain of the transistor Tr21, and the gate of the transistor Tr22 is referred to as a node Nb.

The wiring VrefL has a function of supplying the potential Vref, the wiring VaL has a function of supplying the potential Va, and the wiring VbL has a function of supplying the potential Vb. The wiring VDDL has a function of supplying the potential VDD, and the wiring VSSL has a function of supplying the potential VSS. Here, the case where the potential VDD is a high power supply potential and the potential VSS is a low power supply potential will be described. The wiring RST has a function of supplying a potential for controlling the conduction state of the transistor Tr21. The transistor Tr22, the transistor Tr23, the wiring VDDL, the wiring VSSL, and the wiring VbL constitute a source follower circuit.

Next, an operation example of the circuits OC [1] to [n] will be described. In addition, although the operation example of circuit OC [1] is demonstrated here as a typical example, circuit OC [2] thru | or [n] can be operated similarly. First, when a first current flows through the wiring BL [1], the potential of the node Na becomes a potential corresponding to the first current and the resistance value of the resistance element R1. At this time, the transistor Tr21 is on, and the potential Va is supplied to the node Nb. Thereafter, the transistor Tr21 is turned off.

Next, when a second current flows through the wiring BL [1], the potential of the node Na changes to a potential corresponding to the second current and the resistance value of the resistance element R1. At this time, since the transistor Tr21 is in an off state and the node Nb is in a floating state, the potential of the node Nb changes due to capacitive coupling as the potential of the node Na changes. Here, if the change in the potential of the node Na is ΔV _Na and the capacitive coupling coefficient is 1, the potential of the node Nb is Va + ΔV _Na . When the threshold voltage of the transistor Tr22 and _{V th,} the potential Va + ΔV _Na -V _th is output from the wiring OL [1]. Here, by setting Va _{= V th,} it is possible to output the potential [Delta] V _Na from the wiring OL [1].

The potential ΔV _Na is determined according to the amount of change from the first current to the second current, the resistance element R1, and the potential Vref. Here, since the resistance element R1 and the potential Vref are known, the amount of change in the current flowing from the potential ΔV _Na to the wiring BL can be obtained.

A signal corresponding to the amount of current detected by the offset circuit OFST and / or the amount of change in current as described above is input to the activation function circuit ACTV via the wirings OL [1] to [n].

The activation function circuit ACTV is connected to the wirings OL [1] to [n] and the wirings NIL [1] to [n]. The activation function circuit ACTV has a function of performing an operation for converting the signal input from the offset circuit OFST according to a predefined activation function. As the activation function, for example, a sigmoid function, a tanh function, a softmax function, a ReLU function, a threshold function, or the like can be used. The signal converted by the activation function circuit ACTV is output as output data to the wirings NIL [1] to [n].

<Operation example of semiconductor device>
Using the semiconductor device MAC, the product-sum operation of the first data and the second data can be performed. Hereinafter, an operation example of the semiconductor device MAC when performing the product-sum operation will be described.

FIG. 17 shows a timing chart of an operation example of the semiconductor device MAC. In FIG. 17, the wiring WL [1], the wiring WL [2], the wiring WD [1], the wiring WDref, the node NM [1,1], the node NM [2,1], and the node NMref [1] in FIG. , The transition of the potential of the node NMref [2], the wiring RW [1], and the wiring RW [2], and the transition of the values of the current I _B [1] −I _α [1] and the current I _Bref . . The current I _B [1] −I _α [1] corresponds to the sum of currents flowing from the wiring BL [1] to the memory cells MC [1,1] and [2,1].

Note that the operation will be described here by focusing on the memory cells MC [1,1] and [2,1] and the memory cells MCref [1] and [2] shown in FIG. 15 as representative examples. The MC and the memory cell MCref can be operated similarly.

[Storage of first data]
First, at time T01-T02, the potential of the wiring WL [1] becomes high level, the potential of the wiring WD [1] becomes V _PR −V _{W [1,1]} higher than the ground potential (GND), and the wiring potential of WDref becomes the _{V PR} greater potential than the ground potential. Further, the potentials of the wiring RW [1] and the wiring RW [2] are the reference potential (REFP). Note that the potential V _{W [1, 1]} is a potential corresponding to the first data stored in the memory cell MC [1, 1]. The potential _VPR is a potential corresponding to the reference data. Accordingly, the transistor Tr11 included in the memory cell MC [1,1] and the memory cell MCref [1] is turned on, and the potential of the node NM [1,1] is V _PR −V _{W [1,1]} and the node NMref. The potential of [1] becomes _VPR .

At this time, a current I _{MC [1,1], 0} flowing from the wiring BL [1] to the transistor Tr12 of the memory cell MC [1,1] can be expressed by the following equation. Here, k is a constant determined by the channel length, channel width, mobility, capacitance of the gate insulating film, and the like of the transistor Tr12. V _th is the threshold voltage of the transistor Tr12.

_{I MC [1,1], 0 =} k (V PR -V W [1,1] -V th) 2 (E1)

Further, the current I _{MCref [1], 0} flowing from the wiring BLref to the transistor Tr12 of the memory cell MCref [1] can be expressed by the following equation.

I _{MCref [1], 0} = k (V _PR −V _th ) ² (E2)

Next, at time T02 to T03, the potential of the wiring WL [1] is set to a low level. Accordingly, the transistor Tr11 included in the memory cell MC [1,1] and the memory cell MCref [1] is turned off, and the potentials of the node NM [1,1] and the node NMref [1] are held.

Note that as described above, an OS transistor is preferably used as the transistor Tr11. Accordingly, leakage current of the transistor Tr11 can be suppressed, and the potentials of the node NM [1,1] and the node NMref [1] can be accurately held.

Next, at time T03-T04, the potential of the wiring WL [2] is at a high level, the potential of the wiring WD [1] is V _PR −V _{W [2,1]} higher than the ground potential, and the wiring WDref The potential becomes a potential _VPR larger than the ground potential. Note that the potential V _{W [2, 1]} is a potential corresponding to the first data stored in the memory cell MC [2, 1]. As a result, the transistor Tr11 included in the memory cell MC [2,1] and the memory cell MCref [2] is turned on, and the potential of the node NM [1,1] is V _PR −V _{W [2,1]} and the node NMref. The potential of [1] becomes _VPR .

At this time, the current I _{MC [2,1], 0} flowing from the wiring BL [1] to the transistor Tr12 of the memory cell MC [2,1] can be expressed by the following equation.

_{I MC [2,1], 0 =} k (V PR -V W [2,1] -V th) 2 (E3)

Further, the current I _{MCref [2], 0} flowing from the wiring BLref to the transistor Tr12 of the memory cell MCref [2] can be expressed by the following equation.

I _{MCref [2], 0} = k (V _PR −V _th ) ² (E4)

Next, at time T04 to T05, the potential of the wiring WL [2] is at a low level. Accordingly, the transistor Tr11 included in the memory cell MC [2,1] and the memory cell MCref [2] is turned off, and the potentials of the node NM [2,1] and the node NMref [2] are held.

With the above operation, the first data is stored in the memory cells MC [1,1] and [2,1], and the reference data is stored in the memory cells MCref [1] and [2].

Here, currents flowing through the wiring BL [1] and the wiring BLref from time T04 to T05 are considered. A current is supplied from the current source circuit CS to the wiring BLref. Further, the current flowing through the wiring BLref is discharged to the current mirror circuit CM and the memory cells MCref [1] and [2]. When the current supplied from the current source circuit CS to the wiring BLref is I _Cref and the current discharged from the wiring BLref to the current mirror circuit CM is I _{CM, 0} , the following equation is established.

I _Cref −I _{CM, 0} = I _{MCref [1], 0} + I _{MCref [2], 0} (E5)

A current from the current source circuit CS is supplied to the wiring BL [1]. Further, the current flowing through the wiring BL [1] is discharged to the current mirror circuit CM and the memory cells MC [1,1], [2,1]. In addition, a current flows from the wiring BL [1] to the offset circuit OFST. When the current supplied from the current source circuit CS to the wiring BL [1] is I _{C, 0} and the current flowing from the wiring BL [1] to the offset circuit OFST is I _{α, 0} , the following equation is established.

_{I C -I CM, 0 = I} MC [1,1], 0 + I MC [2,1], 0 + I α, 0 (E6)

[Product-sum operation of first data and second data]
Next, at time T05 to T06, the potential of the wiring RW [1] is V _{X [1]} larger than the reference potential. At this time, the potential V _{X [1]} is supplied to the respective capacitive elements C11 of the memory cell MC [1,1] and the memory cell MCref [1], and the potential of the gate of the transistor Tr12 is increased by capacitive coupling. Note that the potential V _{x [1]} is a potential corresponding to the second data supplied to the memory cell MC [1, 1] and the memory cell MCref [1].

The amount of change in the potential of the gate of the transistor Tr12 is a value obtained by multiplying the amount of change in the potential of the wiring RW by the capacitive coupling coefficient determined by the configuration of the memory cell. The capacitive coupling coefficient is calculated by the capacitance of the capacitive element C11, the gate capacitance of the transistor Tr12, the parasitic capacitance, and the like. Hereinafter, for the sake of convenience, description will be made assuming that the amount of change in potential of the wiring RW and the amount of change in potential of the gate of the transistor Tr12 are the same, that is, the capacitive coupling coefficient is 1. Actually, the potential V _x may be determined in consideration of the capacitive coupling coefficient.

When the potential V _{X [1]} is supplied to the capacitor C11 of the memory cell MC [1] and the memory cell MCref [1], the potentials of the node NN [1] and the node NMref [1] are V _{X [1],} respectively _. To rise.

Here, from time T05 to T06, the current I _{MC [1,1], 1} flowing from the wiring BL [1] to the transistor Tr12 of the memory cell MC [1,1] can be expressed by the following equation.

_{I MC [1,1], 1 =} k (V PR -V W [1,1] + V X [1] -V th) 2 (E7)

That is, by supplying the potential V _{X [1]} to the wiring RW [1], the current flowing from the wiring BL [1] to the transistor Tr12 of the memory cell MC [1,1] is ΔI _{MC [1,1]} = I _{MC [1,1], 1} −I _{MC [1,1], 0 is} increased.

Further, current I _{MCref [1], 1} flowing from the wiring BLref to the transistor Tr12 of the memory cell MCref [1] from time T05 to T06 can be expressed by the following equation.

I _{MCref [1], 1} = k (V _PR + V _{X [1]} −V _th ) ² (E8)

That is, by supplying the potential V _{X [1]} to the wiring RW [1], the current flowing from the wiring BLref to the transistor Tr12 of the memory cell MCref [1] is ΔI _{MCref [1]} = I _{MCref [1], 1} -I _{MCref [1],} incremented by ₀ .

Consider the current flowing through the wiring BL [1] and the wiring BLref. A current I _Cref is supplied from the current source circuit CS to the wiring BLref. Further, the current flowing through the wiring BLref is discharged to the current mirror circuit CM and the memory cells MCref [1] and [2]. When the current discharged from the wiring BLref to the current mirror circuit CM is I _{CM, 1} , the following equation is established.

I _Cref −I _{CM, 1} = I _{MCref [1], 1} + I _{MCref [2], 0} (E9)

The wiring BL [1], the current _{I C} is supplied from the current source circuit CS. Further, the current flowing through the wiring BL [1] is discharged to the current mirror circuit CM and the memory cells MC [1,1], [2,1]. Further, a current also flows from the wiring BL [1] to the offset circuit OFST. When the current flowing from the wiring BL [1] to the offset circuit OFST is I _{α, 1} , the following equation is established.

_{I C -I CM, 1 = I} MC [1,1], 1 + I MC [2,1], 1 + I α, 1 (E10)

From the equations (E1) to (E10) _, the difference between the current I _{α, 0} and the current I _{α, 1} (differential current ΔI _α ) can be expressed by the following equation.

ΔI _α = I _{α, 0} −I _{α, 1} = 2 kV _{W [1,1]} V _{X [1]} (E11)

Thus, the differential current ΔI _α has a value corresponding to the product of the potentials V _{W [1, 1]} and V _{X [1]} .

After that, at time T06 to T07, the potential of the wiring RW [1] becomes the ground potential, and the potentials of the node NM [1,1] and the node NMref [1] are the same as those at time T04-T05.

Next, at time T07 to T08, the potential of the wiring RW [1] is V _{X [1]} larger than the reference potential, and the potential of the wiring RW [2] is V _{X [2]} larger than the reference potential. Supplied. As a result, the potential V _{X [1]} is supplied to the respective capacitive elements C11 of the memory cell MC [1,1] and the memory cell MCref [1], and the node NM [1,1] and the node NMref [ 1] is increased by V _{X [1]} . In addition, the potential V _{X [2]} is supplied to the respective capacitor C11 of the memory cell MC [2,1] and the memory cell MCref [2], and the node NM [2,1] and the node NMref [2 _] are connected by capacitive coupling. ] Increases by V _{X [2]} .

Here, at time T07-T08, the current I _{MC [2,1], 1} flowing from the wiring BL [1] to the transistor Tr12 of the memory cell MC [2,1] can be expressed by the following equation.

_{I MC [2,1], 1 =} k (V PR -V W [2,1] + V X [2] -V th) 2 (E12)

That is, by supplying the potential V _{X [2]} to the wiring RW [2], the current flowing from the wiring BL [1] to the transistor Tr12 of the memory cell MC [2, 1] is ΔI _{MC [2,1]} = I _{MC [2,1], 1} −I _{MC [2,1], 0} increases.

Further, current I _{MCref [2], 1} flowing from the wiring BLref to the transistor Tr12 of the memory cell MCref [2] from time T05 to T06 can be expressed by the following equation.

I _{MCref [2], 1} = k (V _PR + V _{X [2]} −V _th ) ² (E13)

That is, by supplying the potential V _{X [2]} to the wiring RW [2], the current flowing from the wiring BLref to the transistor Tr12 of the memory cell MCref [2] is ΔI _{MCref [2]} = I _{MCref [2], 1} -I _{MCref [2],} incremented by ₀ .

Consider the current flowing through the wiring BL [1] and the wiring BLref. A current I _Cref is supplied from the current source circuit CS to the wiring BLref. Further, the current flowing through the wiring BLref is discharged to the current mirror circuit CM and the memory cells MCref [1] and [2]. When the current discharged from the wiring BLref to the current mirror circuit CM is I _{CM, 2} , the following equation is established.

I _Cref −I _{CM, 2} = I _{MCref [1], 1} + I _{MCref [2], 1} (E14)

The wiring BL [1], the current _{I C} is supplied from the current source circuit CS. Further, the current flowing through the wiring BL [1] is discharged to the current mirror circuit CM and the memory cells MC [1,1], [2,1]. Further, a current also flows from the wiring BL [1] to the offset circuit OFST. Assuming that the current flowing from the wiring BL [1] to the offset circuit OFST is I _{α, 2} , the following equation is established.

_{I C -I CM, 2 = I} MC [1,1], 1 + I MC [2,1], 1 + I α, 2 (E15)

Then, from the equations (E1) to (E8) and the equations (E12) to (E15) _, the difference between the current I _{α, 0} and the current I _{α, 2} (differential current ΔI _α ) is expressed by the following equation. be able to.

ΔI _α = I _{α, 0} −I _{α, 2} = 2k (VW _{[1,1] VX [1]} + _{VW [2,1]} VX _[2] ) (E16)

As described above, the differential current ΔI _α is obtained by adding the product of the potential V _{W [1, 1]} and the potential V _{X [1] and} the product of the potential V _{W [2, 1]} and the potential V _{X [2].} The value depends on the combined result.

After that, at times T08 to T09, the potentials of the wirings RW [1] and [2] become the ground potential, and the potentials of the nodes NM [1,1] and [2,1] and the nodes NMref [1] and [2] are It becomes the same as time T04-T05.

As shown in the equations (E9) and (E16), the difference current ΔI _α input to the offset circuit OFST includes the potential V _X corresponding to the first data (weight) and the second data (input data). ) corresponding to a value corresponding to the combined result plus the product of the potential V _W. That is, by measuring the differential current ΔI _α with the offset circuit OFST, it is possible to obtain a product-sum operation result of the first data and the second data.

In the above description, the memory cells MC [1,1] and [2,1] and the memory cells MCref [1] and [2] are particularly focused. However, the number of the memory cells MC and the memory cells MCref should be arbitrarily set. Can do. The differential current ΔIα when the number of rows m of the memory cell MC and the memory cell MCref is an arbitrary number can be expressed by the following equation.

ΔI _α = 2kΣ _i V _{W [i, 1]} V _{X [i]} (E17)

Further, by increasing the number of columns n of the memory cells MC and the memory cells MCref, the number of product-sum operations executed in parallel can be increased.

As described above, by using the semiconductor device MAC, the product-sum operation of the first data and the second data can be performed. Note that by using the structure shown in FIG. 15 as the memory cell MC and the memory cell MCref, a product-sum operation circuit can be formed with a small number of transistors. Therefore, the circuit scale of the semiconductor device MAC can be reduced.

When the semiconductor device MAC is used for computation in the neural network, the number of rows m of the memory cells MC corresponds to the number of input data supplied to one neuron, and the number of columns n of the memory cells MC corresponds to the number of neurons. Can do. For example, consider a case where a product-sum operation using the semiconductor device MAC is performed in the intermediate layer HL shown in FIG. At this time, the number m of rows of the memory cells MC is set to the number of input data supplied from the input layer IL (the number of neurons of the input layer IL), and the number of columns n of the memory cells MC is the number of neurons of the intermediate layer HL. Can be set to any number.

Note that the structure of the neural network to which the semiconductor device MAC is applied is not particularly limited. For example, the semiconductor device MAC can also be used for a convolutional neural network (CNN), a recursive neural network (RNN), an auto encoder, a Boltzmann machine (including a limited Boltzmann machine), and the like.

As described above, by using the semiconductor device MAC, the product-sum operation of the neural network can be performed. Furthermore, by using the memory cell MC and the memory cell MCref shown in FIG. 15 for the cell array CA, an integrated circuit IC capable of improving calculation accuracy, reducing power consumption, or reducing the circuit scale is provided. it can.

This embodiment can be implemented in appropriate combination with the structures described in the other embodiments.

(Embodiment 3)
In this embodiment, an example of an imaging device to which one embodiment of the present invention can be applied is described with reference to drawings.

<Configuration example of pixel circuit>
[Configuration example 1]
FIG. 18A is a diagram illustrating a pixel circuit applicable to the pixel 121. The pixel circuit includes a photoelectric conversion element 50 and transistors 51 to 56.

In the pixel circuit illustrated in FIG. 18A, one electrode (anode) of the photoelectric conversion element 50 is electrically connected to one of a source and a drain of the transistor 51. The other electrode (cathode) of the photoelectric conversion element 50 is electrically connected to one of a source and a drain of the transistor 55. The other of the source and the drain of the transistor 55 is electrically connected to one of the source and the drain of the transistor 56. A gate of the transistor 55 is electrically connected to the wiring 72. The other of the source and the drain of the transistor 56 is electrically connected to the wiring 77. A gate of the transistor 56 is electrically connected to the wiring 74.

One of the source and the drain of the transistor 52 is electrically connected to one of the source and the drain of the transistor 51. The other of the source and the drain of the transistor 52 is electrically connected to the wiring 73. A gate of the transistor 52 is electrically connected to the wiring 76.

The other of the source and the drain of the transistor 51 is electrically connected to the gate of the transistor 53. One of the source and the drain of the transistor 53 is electrically connected to one of the source and the drain of the transistor 54. The other of the source and the drain of the transistor 53 is electrically connected to the wiring 79. The other of the source and the drain of the transistor 54 is electrically connected to the wiring 71. A gate of the transistor 54 is electrically connected to the wiring 78. Note that a capacitor which is electrically connected to the gate of the transistor 53 may be provided.

A node where one electrode (anode) of the photoelectric conversion element 50, one of the source or drain of the transistor 51, and one of the source or drain of the transistor 52 is electrically connected functions as a charge storage portion (NR). To do. A node where the other of the source and the drain of the transistor 51 and the gate of the transistor 53 are electrically connected functions as a charge detection portion (ND).

Here, the wiring 71 can function as an output line for outputting a signal from the pixel. The wiring 73, the wiring 77, and the wiring 79 can function as power supply lines. For example, the wiring 73 is supplied with a low power supply potential (eg, GND), and the wiring 79 is supplied with a high power supply potential (eg, VDD).

For the photoelectric conversion element 50, it is preferable to use a photoelectric conversion element that produces an avalanche multiplication effect in order to increase the light detection sensitivity at low illuminance. In order to produce the avalanche multiplication effect, a relatively high potential HVDD is required. Therefore, the wiring 77 is supplied with a relatively high potential HVDD. Note that a low power supply potential is also supplied to the wiring 77. Therefore, the potential HVDD or the low power supply potential is supplied to the other electrode (cathode) of the photoelectric conversion element 50 through the transistor 55 and the transistor 56. More specifically, in the case where the transistors 55 and 56 are n-channel transistors, the other electrode (cathode) of the photoelectric conversion element 50 has a low power supply potential or a threshold of the transistor 55 higher than the potential HVDD. A potential lower by the value voltage and the threshold voltage of the transistor 56 is supplied. The photoelectric conversion element 50 can also be used by applying a potential that does not produce an avalanche multiplication effect.

The wiring 72, the wiring 74, the wiring 75, the wiring 76, and the wiring 78 can function as signal lines for controlling on / off of each transistor.

The transistor 51 can have a function of transferring the potential of the charge storage unit (NR) that changes in accordance with the output of the photoelectric conversion element 50 to the charge detection unit (ND). The transistor 52 has a function of initializing the potentials of the charge accumulation unit (NR) and the charge detection unit (ND). The transistor 53 has a function of outputting a signal corresponding to the potential of the charge detection portion (ND). The transistor 54 has a function of selecting a pixel from which a signal is read. The transistor 55 has a function of being turned on or off when the row to which the pixel 121 belongs is selected. The transistor 56 has a function of being turned on or off when the column to which the pixel 121 belongs is selected.

When a high voltage is applied to the photoelectric conversion element 50, it is necessary to use a high breakdown voltage transistor that can withstand the high voltage as a transistor connected to the photoelectric conversion element 50. For example, an OS transistor or the like can be used as the high voltage transistor. Specifically, OS transistors are preferably used as the transistor 51, the transistor 52, the transistor 55, and the transistor 56.

The transistor 51, the transistor 52, the transistor 55, and the transistor 56 are desired to have excellent switching characteristics. However, since the transistor 53 is desired to have excellent amplification characteristics, it is preferable that the transistor has a high on-state current. . Therefore, a transistor using silicon as an active layer or an active region (also referred to as “Si transistor”) is preferably used as the transistor 53 and the transistor 54.

With the above-described structures of the transistors 51 to 56, an imaging device with high light detection sensitivity at low illuminance and capable of outputting a signal with little noise can be manufactured. In addition, since the light detection sensitivity is high, the light capture time can be shortened and imaging can be performed at high speed.

Note that the transistor is not limited to the above structure, and an OS transistor may be applied to the transistor 53 and the transistor 54. Alternatively, Si transistors may be applied to the transistors 51, 52, 55, and 56. A p-channel transistor may be used as at least one of the transistors 51 to 56.

For example, as illustrated in FIG. 19A, the transistor 55 and the transistor 56 may be p-channel transistors. In any case, the imaging operation of the pixel circuit is possible.

[Operation example 1]
Next, operation of the pixel circuit illustrated in FIG. 18A will be described with reference to a timing chart in FIG. Note that in this embodiment, the transistors 51 to 56 are n-channel transistors. In an example operation described below, the wiring 76 connected to the gate of the transistor 52 is supplied with VDD as “H” and GND as “L”. The wiring 75 connected to the gate of the transistor 51 and the wiring 78 connected to the gate of the transistor 54 are supplied with VDD as “H” and GND as “L”. Further, the potential of VDD is supplied to the wiring 79 connected to the source of the transistor 53. The wiring 72 connected to the gate of the transistor 55 and the wiring 74 connected to the gate of the transistor 56 are supplied with HVDD as “H” and GND as “L”. Note that a potential other than the above may be supplied to each wiring.

[Reset operation]
At time T1, GND is supplied to the wiring 77, the wiring 72, the wiring 74, the wiring 75, and the wiring 76 are set to “H”, and the potentials of the charge storage portion (NR) and the charge detection portion (ND) are reset to the reset potential (GND). ).

At time T2, the wiring 72, the wiring 74, the wiring 75, and the wiring 76 are set to “L”. Then, the potentials of the charge accumulation unit (NR) and the charge detection unit (ND) are held at the reset potential (GND).

At time T3, HVDD is supplied to the wiring 77. The operations from time T1 to time T3 can be performed simultaneously on all the pixels included in the image sensor.

[Accumulation operation]
By setting the wiring 72 and the wiring 74 to “H” at time T4, the potential of the charge storage portion (NR) changes (accumulation operation). The potential of the charge storage unit (NR) changes from GND to HVDD at the maximum according to the intensity of light incident on the photoelectric conversion element 50.

[Transfer operation]
At time T5, the wiring 75 is set to “H”, and the charge in the charge storage portion (NR) is transferred to the charge detection portion (ND).

[Holding operation]
At time T6, the wiring 72, the wiring 74, and the wiring 75 are set to “L”. At this point, the potential of the charge detection unit (ND) is determined. A potential corresponding to the intensity of light incident on the photoelectric conversion element 50 is held in the charge detection unit (ND).

Note that when the wiring 72, the wiring 74, and the wiring 76 are set to “L”, the photoelectric conversion element 50 and the charge storage portion (NR) are in a floating state. When the photoelectric conversion element 50 is in a floating state, holes and electrons generated by light irradiation immediately recombine and disappear. For this reason, even if light is applied to the photoelectric conversion element 50 during the holding operation, the potential of the charge storage portion (NR) does not substantially change.

Therefore, as illustrated in FIG. 19B, the transistor 51 and the wiring 75 are not provided, and the charge accumulation portion (NR) and the charge detection portion (ND) can be used together. In this case, the transfer operation described above can be omitted.

[Read operation]
In a period from time T7 to time T8, the wiring 78 is set to “H”, and a signal corresponding to the potential of the charge detection portion (ND) is output to the wiring 71. That is, an output signal corresponding to the intensity of light incident on the photoelectric conversion element 50 in the accumulation operation can be obtained.

[Configuration example 2]
FIG. 20A is a diagram illustrating another configuration example of a pixel circuit applicable to the pixel 121. The pixel circuit includes a photoelectric conversion element 50 and transistors 51 to 55. Note that, in order to reduce the repetition of the description, differences from the configuration example 1 described mainly with reference to FIG.

In the pixel circuit illustrated in FIG. 20A, one electrode (anode) of the photoelectric conversion element 50 is connected to one of the source and the drain of the transistor 51, one of the source and the drain of the transistor 52, and the source or the drain of the transistor 55. Electrically connected to one side. The other electrode (cathode) of the photoelectric conversion element 50 is electrically connected to the wiring 77. The other of the source and the drain of the transistor 52 and the other of the source and the drain of the transistor 55 are electrically connected to the wiring 73. A gate of the transistor 52 is electrically connected to the wiring 74. A gate of the transistor 55 is electrically connected to the wiring 72. One node (anode) of the photoelectric conversion element 50, one of the source and drain of the transistor 51, one of the source and drain of the transistor 52, and one of the source and drain of the transistor 55 are electrically connected to each other. Part (NR).

The other structures are the same as those of the pixel circuit shown in FIG. In the pixel circuit illustrated in FIG. 20A, the transistor 52 and the transistor 55 have a function of initializing potentials of the charge accumulation portion (NR) and the charge detection portion (ND). The transistor 55 has a function of being turned off when a row to which the pixel 121 belongs is selected and turned on when not selected. The transistor 52 has a function of being turned off when a column to which the pixel 121 belongs is selected and turned on when not selected.

[Operation example 2]
Next, operation of the pixel circuit illustrated in FIG. 20A will be described with reference to a timing chart in FIG. Note that, in order to reduce the repetition of the description, differences from the operation example 1 described mainly with reference to FIG. 18B will be described.

In an example operation described below, the wiring 72 and the wiring 74 are supplied with the potential of VDD as “H” and the potential of GND as “L”. It is assumed that HVDD is supplied to the wiring 77.

[Reset operation]
At time T11, the wiring 72, the wiring 74, and the wiring 75 are set to “H”, and the potentials of the charge accumulation portion (NR) and the charge detection portion (ND) are set to the reset potential (GND).

At time T12, the wiring 75 is set to “L”. Then, the potential of the charge detection unit (ND) is held at the reset potential (GND).

[Accumulation operation]
By setting the wiring 72 and the wiring 74 to “L” at time T13, the potential of the charge storage portion (NR) changes (accumulation operation). The potential of the charge storage unit (NR) changes from GND to HVDD at the maximum according to the intensity of light incident on the photoelectric conversion element 50.

[Transfer operation]
At time T14, the wiring 75 is set to “H”, and the charge in the charge storage portion (NR) is transferred to the charge detection portion (ND).

[Holding operation]
The wiring 75 is set to “L” at time T15. At this point, the potential of the charge detection unit (ND) is determined. A potential corresponding to the intensity of light incident on the photoelectric conversion element 50 is held in the charge detection unit (ND).

[Read operation]
In a period from time T16 to time T17, the wiring 78 is set to “H”, and a signal corresponding to the potential of the charge detection portion (ND) is output to the wiring 71. That is, an output signal corresponding to the intensity of light incident on the photoelectric conversion element 50 in the accumulation operation can be obtained.

As shown in FIG. 21, the transistor 51 and the wiring 75 are not provided, and the charge storage unit (NR) and the charge detection unit (ND) can also be used. In this case, the transfer operation described above can be omitted.

When the pixel circuit shown in FIG. 21 is used for an imaging device having the first pixel group and the second pixel group, the first pixel is set by first turning off the transistor 52 and the transistor 55 of the pixels included in the first pixel group. A group accumulation operation may be performed, and then the accumulation operation may be performed with the transistors 52 and 55 of the pixels included in the first pixel group and the second pixel group turned off. The transfer operation and holding operation described above can be omitted.

In this case, the mixed imaging information obtained by adding the first imaging information and the second imaging information is accumulated in the pixels of the first pixel group. The first imaging information can be obtained by subtracting the second imaging information from the mixed imaging information.

<Pixel configuration example>
FIG. 22A illustrates a configuration example of an imaging element having the pixel circuit described above. The pixel can include a layer 61, a layer 62, and a layer 63, each of which has a region that overlaps with each other.

The layer 61 has the configuration of the photoelectric conversion element 50. The photoelectric conversion element 50 includes an electrode 65 corresponding to a pixel electrode, a photoelectric conversion unit 66, and an electrode 67 corresponding to a common electrode.

The electrode 65 is preferably a low-resistance metal layer. For example, aluminum, titanium, tungsten, tantalum, silver, or a stacked layer thereof can be used.

For the electrode 67, a conductive layer having a high light-transmitting property with respect to visible light is preferably used. For example, indium oxide, tin oxide, zinc oxide, indium-tin oxide, gallium-zinc oxide, indium-gallium-zinc oxide, graphene, or the like can be used. Note that the electrode 67 may be omitted.

For the photoelectric conversion unit 66, for example, a pn junction photodiode using a selenium-based material as a photoelectric conversion layer can be used. It is preferable to use a selenium-based material that is a p-type semiconductor for the layer 66a and a gallium oxide that is an n-type semiconductor for the layer 66b.

A photoelectric conversion element using a selenium-based material has a high external quantum efficiency with respect to visible light. In the photoelectric conversion element, by using the avalanche multiplication effect, a highly sensitive sensor with a large amplification of electrons with respect to the amount of incident light can be obtained. In addition, since the selenium-based material has a high light absorption coefficient, it has production advantages such that the photoelectric conversion layer can be formed as a thin film. A thin film of a selenium-based material can be formed using a vacuum evaporation method, a sputtering method, or the like.

Examples of the selenium-based material include crystalline selenium such as single crystal selenium and polycrystalline selenium, amorphous selenium, copper, indium, selenium compound (CIS), or copper, indium, gallium, selenium compound (CIGS), etc. Can be used.

The n-type semiconductor is preferably formed using a material having a wide band gap and a light-transmitting property with respect to visible light. For example, zinc oxide, gallium oxide, indium oxide, tin oxide, or an oxide in which they are mixed can be used. These materials also have a function as a hole injection blocking layer, and can reduce the dark current.

Note that the layer 61 is not limited to the above structure, and a pn junction photodiode using one of a p-type silicon semiconductor and an n-type silicon semiconductor as the layer 66a and the other of the p-type silicon semiconductor and the n-type silicon semiconductor as the layer 66b. It may be. Alternatively, a pin junction photodiode in which an i-type silicon semiconductor layer is provided between the layer 66a and the layer 66b may be used.

The pn junction photodiode or the pin junction photodiode can be formed using single crystal silicon. At this time, it is preferable that the layer 61 and the layer 62 are electrically joined using a bonding process. Further, the pin junction photodiode can be formed using a thin film such as amorphous silicon, microcrystalline silicon, or polycrystalline silicon.

The layer 62 can be a layer including an OS transistor (transistors 51 to 56), for example. In the circuit configuration of the pixel shown in FIG. 18A, the potential of the charge detection portion (ND) is small when the intensity of light incident on the photoelectric conversion element 50 is small. Since the OS transistor has an extremely low off-state current, a current corresponding to the gate potential can be accurately output even when the gate potential is extremely small. Therefore, the range of illuminance that can be detected, that is, the dynamic range can be expanded.

In addition, due to the low off-state current characteristics of the transistor 51, the transistor 52, the transistor 55, and the transistor 56, the period in which charge can be held in the charge detection portion (ND) and the charge accumulation portion (NR) can be extremely long. Therefore, it is possible to apply a global shutter system in which charge accumulation is performed simultaneously in all pixels without complicating the circuit configuration and operation method.

The layer 63 can be a supporting substrate or a layer having a Si transistor (transistor 53, transistor 54). The Si transistor can have a structure having an active region on a single crystal silicon substrate and a crystalline silicon active layer on an insulating surface. Note that in the case where a single crystal silicon substrate is used for the layer 63, a pn junction photodiode or a pin junction photodiode may be formed over the single crystal silicon substrate. In this case, the layer 61 can be omitted.

FIG. 22B is a block diagram illustrating a circuit configuration example of an image sensor 89 corresponding to the image sensor 120. The imaging element has a pixel array 81 having pixels 80 arranged in a matrix, a circuit 82 (row driver) having a function of selecting a row of the pixel array 81, and a function of selecting a column of the pixel array 81. A circuit 86 (column driver); a circuit 83 (CDS circuit) for performing correlated double sampling processing on the output signal of the pixel 80; and a function of converting analog data output from the circuit 83 into digital data. A circuit 84 (A / D conversion circuit or the like), and a circuit 85 having a function of selecting and reading data converted by the circuit 84. Note that the circuit 83 may be omitted.

For example, the elements of the pixel array 81 excluding the photoelectric conversion element can be provided in the layer 62 illustrated in FIG. Elements of the circuits 82 to 85 can be provided in the layer 63. These circuits can be constituted by CMOS circuits using silicon transistors.

With such a structure, a transistor suitable for each circuit can be used, and the area of the imaging element can be reduced.

23A, 23B, and 23C are diagrams illustrating a specific configuration of the imaging element illustrated in FIG. For easy understanding, FIGS. 23A, 23B, and 23C show some of the pixels included in the image sensor. FIG. 23A is a cross-sectional view illustrating the channel length direction of the transistors 51, 53, 54, and 55. FIG. 23B is a cross-sectional view taken along dashed-dotted line A1-A2 and illustrates a cross section of the transistor 52 in the channel width direction. FIG. 23C is a cross-sectional view along dashed-dotted line B1-B2 and shows a cross section of the transistor 53 in the channel width direction.

The imaging element can be a stack of layers 61 to 63. The layer 61 can include a partition 92 in addition to the photoelectric conversion element 50 having a selenium layer. The partition wall 92 is provided so as to cover the step of the electrode 65. The selenium layer used for the photoelectric conversion element 50 has a high resistance and can be configured not to be separated between pixels.

The layer 62 is provided with transistors 51, 52, 55, and 56 which are OS transistors. The transistors 51 and 52 both have a structure having a back gate 91. Note that at least one of the transistors 51, 52, 55, and 56 may have a back gate. As shown in FIG. 23B, the back gate 91 may be electrically connected to a front gate of a transistor provided to face the back gate 91. Alternatively, the back gate 91 may be configured to be able to supply a fixed potential different from that of the front gate.

In FIG. 23A, a self-aligned top gate transistor is illustrated as the OS transistor, but a non-self-aligned transistor may be used as shown in FIG.

In the layer 63, a transistor 53 and a transistor 54 which are Si transistors are provided. In FIG. 23A, the Si transistor exemplifies a structure having a fin-type semiconductor layer provided on the silicon substrate 400. However, as shown in FIG. 24B, a planar substrate having an active region in the silicon substrate 201. It may be a mold. Alternatively, as illustrated in FIG. 18C, a transistor including a silicon thin film semiconductor layer 210 may be used. The semiconductor layer 210 can be, for example, single crystal silicon (SOI (Silicon on Insulator)) formed on the insulating layer 220 on the silicon substrate 202. Alternatively, it may be polycrystalline silicon formed on an insulating surface such as a glass substrate. In addition, the layer 63 can be provided with a circuit for driving a pixel.

An insulating layer 93 having a function of preventing hydrogen diffusion is provided between the region where the OS transistor is formed and the region where the Si transistor is formed. Hydrogen in the insulating layer provided in the vicinity of the active regions of the transistors 53 and 54 terminates dangling bonds of silicon. On the other hand, hydrogen in the insulating layer provided in the vicinity of the oxide semiconductor layer which is an active layer of the transistors 51 and 52 is one of the factors that generate carriers in the oxide semiconductor layer.

The reliability of the transistors 53 and 54 can be improved by confining hydrogen in one layer by the insulating layer 93. In addition, since the diffusion of hydrogen from one layer to the other layer is suppressed, the reliability of the transistors 51 and 52 can be improved.

As the insulating layer 93, for example, aluminum oxide, aluminum oxynitride, gallium oxide, gallium oxynitride, yttrium oxide, yttrium oxynitride, hafnium oxide, hafnium oxynitride, yttria-stabilized zirconia (YSZ), or the like can be used.

FIG. 25A is a cross-sectional view illustrating an example in which a color filter or the like is added to the imaging element of one embodiment of the present invention. In the cross-sectional view, a part of a region having a pixel circuit for three pixels is shown. An insulating layer 300 is formed on the layer 61 where the photoelectric conversion element 50 is formed. The insulating layer 300 can be formed using a silicon oxide film having high light-transmitting property with respect to visible light. A silicon nitride film may be stacked as a passivation film. Further, a dielectric film such as hafnium oxide may be laminated as the antireflection film.

A light shielding layer 310 may be formed on the insulating layer 300. The light shielding layer 310 has a function of preventing color mixing of light passing through the upper color filter. For the light-blocking layer 310, a metal layer such as aluminum or tungsten can be used. Further, the metal layer and a dielectric film having a function as an antireflection film may be stacked.

An organic resin layer 320 can be provided as a planarization film over the insulating layer 300 and the light shielding layer 310. A color filter 330 (color filter 330a, color filter 330b, color filter 330c) is formed for each pixel. For example, colors such as R (red), G (green), B (blue), Y (yellow), C (cyan), and M (magenta) are assigned to the color filters 330a, 330b, and 330c. Thus, a color image can be obtained.

An insulating layer 360 having a light-transmitting property with respect to visible light or the like can be provided over the color filter 330.

In addition, as shown in FIG. 25B, an optical conversion layer 350 may be used instead of the color filter 330. With such a configuration, an imaging element that can obtain images in various wavelength regions can be obtained.

For example, if a filter that blocks light having a wavelength shorter than or equal to that of visible light is used for the optical conversion layer 350, an infrared imaging element can be obtained. If a filter that blocks light having a wavelength shorter than or equal to the near infrared wavelength is used for the optical conversion layer 350, a far infrared imaging device can be obtained. If a filter that blocks light having a wavelength longer than or equal to that of visible light is used for the optical conversion layer 350, an ultraviolet imaging device can be obtained.

In addition, when a scintillator is used for the optical conversion layer 350, an imaging element that can be used for an X-ray imaging element or the like and obtain an image that visualizes the intensity of radiation can be obtained. When radiation such as X-rays transmitted through the subject is incident on the scintillator, it is converted into light (fluorescence) such as visible light or ultraviolet light by a photoluminescence phenomenon. Then, image data is acquired by detecting the light with the photoelectric conversion element 50. Moreover, you may use the image pick-up element of the said structure for a radiation detector etc.

A scintillator contains a substance that emits visible light or ultraviolet light by absorbing energy when irradiated with radiation such as X-rays or gamma rays. For example, Gd ₂ O ₂ S: Tb, Gd ₂ O ₂ S: Pr, Gd ₂ O ₂ S: Eu, BaFCl: Eu, NaI, CsI, CaF ₂ , BaF ₂ , CeF ₃ , LiF, LiI, ZnO, etc. What was disperse | distributed to resin or ceramics can be used.

Note that the photoelectric conversion element 50 using a selenium-based material can directly convert radiation such as X-rays into electric charges, and thus can be configured to eliminate a scintillator.

In addition, as illustrated in FIG. 25C, a microlens array 340 may be provided over the color filter 330a, the color filter 330b, and the color filter 330c. Light passing through the individual lenses included in the microlens array 340 passes through the color filter directly below and is irradiated onto the photoelectric conversion element 50. Alternatively, the microlens array 340 may be provided over the optical conversion layer 350 illustrated in FIG.

Hereinafter, an example of a package and a camera module containing an image sensor chip will be described. For the image sensor chip, the configuration of the imaging element can be used.

FIG. 26A1 is an external perspective view of the upper surface side of the package containing the image sensor chip. The package includes a package substrate 410 for fixing the image sensor chip 450, a cover glass 420, and an adhesive 430 for bonding the two.

FIG. 26A2 is an external perspective view of the lower surface side of the package. The bottom surface of the package has a BGA (Ball Grid Array) configuration with solder balls as bumps 440. In addition, not only BGA but LGA (Land grid array), PGA (Pin Grid Array), etc. may be sufficient.

FIG. 26 (A3) is a perspective view of the package shown with the cover glass 420 and part of the adhesive 430 omitted. An electrode pad 460 is formed on the package substrate 410, and the electrode pad 460 and the bump 440 are electrically connected through a through hole. The electrode pad 460 is electrically connected to the image sensor chip 450 by a wire 470.

FIG. 26B1 is an external perspective view of the upper surface side of the camera module in which the image sensor chip is housed in a lens-integrated package. The camera module includes a package substrate 411 that fixes the image sensor chip 451, a lens cover 421, a lens 435, and the like. Further, an IC chip 490 having functions such as an image sensor driving circuit and a signal conversion circuit is also provided between the package substrate 411 and the image sensor chip 451, and has a configuration as a SiP (System in package). Yes.

FIG. 26B2 is an external perspective view of the lower surface side of the camera module. The package substrate 411 has a QFN (Quad Flat No-Lead Package) configuration in which mounting lands 441 are provided on a lower surface and side surfaces. Note that this configuration is an example, and a QFP (Quad Flat Package), the above-described BGA, or the like may be used.

FIG. 26 (B3) is a perspective view of the module shown with a part of the lens cover 421 and the lens 435 omitted. The land 441 is electrically connected to the electrode pad 461, and the electrode pad 461 is electrically connected to the image sensor chip 451 or the IC chip 490 by wires 471.

By mounting the image sensor chip in a package having the above-described form, mounting on a printed board or the like is facilitated, and the image sensor chip can be incorporated into various semiconductor devices and electronic devices.

This embodiment can be implemented in appropriate combination with the structures described in the other embodiments.

(Embodiment 4)
Electronic devices that can use the imaging device and / or imaging system according to one embodiment of the present invention include a display device, a personal computer, an image storage device or an image playback device including a recording medium, a mobile phone, and a portable type. Game consoles, portable data terminals, electronic book terminals, video cameras, digital still cameras and other cameras, goggle-type displays (head-mounted displays), navigation systems, sound playback devices (car audio, digital audio players, etc.), copiers, facsimiles , Printers, multifunction printers, automatic teller machines (ATMs), vending machines, and the like. Specific examples of these electronic devices are shown in FIGS.

FIG. 27A illustrates a monitoring camera, which includes a housing 951, a lens 952, a support portion 953, and the like. The imaging device of one embodiment of the present invention can be provided as one of the components for acquiring an image in the monitoring camera. The surveillance camera is an idiomatic name and does not limit the application. For example, a device having a function as a surveillance camera is also called a camera or a video camera.

FIG. 27B illustrates a video camera, which includes a first housing 971, a second housing 972, a display portion 973, operation keys 974, a lens 975, a connection portion 976, and the like. The operation key 974 and the lens 975 are provided in the first housing 971, and the display portion 973 is provided in the second housing 972. The imaging device of one embodiment of the present invention can be provided as one of the components for acquiring an image in the video camera.

FIG. 27C illustrates a digital camera, which includes a housing 961, a shutter button 962, a microphone 963, a light-emitting portion 967, a lens 965, and the like. The imaging device of one embodiment of the present invention can be provided as one of the components for acquiring an image in the digital camera.

FIG. 27D illustrates a wristwatch-type information terminal, which includes a housing 931, a display portion 932, a wristband 933, operation buttons 935, a crown 936, a camera 939, and the like. The display unit 932 may be a touch panel. The imaging device of one embodiment of the present invention can be provided as one of the components for acquiring an image in the information terminal.

FIG. 27E illustrates an example of a mobile phone, which includes a housing 981, a display portion 982, operation buttons 983, an external connection port 984, a speaker 985, a microphone 986, a camera 987, and the like. The mobile phone includes a touch sensor in the display portion 982. All operations such as making a call or inputting characters can be performed by touching the display portion 982 with a finger, a stylus, or the like. The imaging device of one embodiment of the present invention can be provided as one of the components for acquiring an image in the mobile phone.

FIG. 27F illustrates a portable data terminal including a housing 911, a display portion 912, a camera 919, and the like. Information can be input and output by a touch panel function of the display portion 912. The imaging device of one embodiment of the present invention can be provided as one of the components for acquiring an image in the portable data terminal.

This embodiment can be implemented in appropriate combination with the structures described in the other embodiments.

DESCRIPTION OF SYMBOLS 100 Image pick-up apparatus 101 Image pick-up part 110 Optical element 111 Liquid crystal element 112 Polarization filter 120 Image pick-up element 121 Pixel 122 Pixel 130 Control part 140 Operation part 150 Image processing part 151 Neural network 160 Storage part 170 Auxiliary storage part 180 External input / output part 190 Communication part 200 External Device 201 Silicon Substrate 202 Silicon Substrate 210 Semiconductor Layer 220 Insulating Layer 230 Control Unit 240 Operation Unit 250 Image Processing Unit 251 Neural Network 270 Storage Unit 280 Auxiliary Storage Unit 290 External Input / Output Unit

Claims (6)

An optical element, an image sensor, and an arithmetic unit;
The optical element is
Operable in both a first state that transmits light of the first polarization axis or a second state that transmits light of the second polarization axis;
The image sensor is
Having a first pixel group and a second pixel group;
The computing unit is
An imaging apparatus having a function of executing learning or inference by a neural network. In claim 1,
The liquid crystal element is an imaging device that operates in a TN mode. In claim 1 or claim 2,
An imaging apparatus in which a relative angle between the first polarization axis and the second polarization axis is less than 180 °. In any one of Claims 1 thru | or 3,
The image pickup apparatus, wherein the neural network is a deep neural network, a convolutional neural network, a recurrent neural network, a self-encoder, a deep Boltzmann machine, or a deep belief network. In any one of Claims 1 thru | or 4,
The imaging device in which the first pixel group and the second pixel group are arranged in a checkered pattern. An imaging system using the imaging apparatus according to any one of claims 1 to 5,
Obtaining first imaging information with the first pixel group when the optical element is in a first state;
Obtaining second imaging information in the second pixel group when the optical element is in a second state;
Using the first imaging information and the second imaging information,
Generating third imaging information in the arithmetic unit;
An imaging system.

Images