WO2021075321A1

WO2021075321A1 - Image capture apparatus, electronic device and image capture method

Info

Publication number: WO2021075321A1
Application number: PCT/JP2020/037944
Authority: WO
Inventors: 裕幸小沢; 斉甲斐; 山田　聡
Original assignee: ソニー株式会社
Priority date: 2019-10-18
Filing date: 2020-10-07
Publication date: 2021-04-22
Also published as: JP2021068950A

Abstract

[Problem] To achieve an optimum signal processing independently of external instructions. [Solution] An image capture apparatus according to the present invention comprises: a pixel array unit having a plurality of pixels for performing photoelectric conversions; a converter for converting analog image signals outputted by the pixel array unit to digital pixel data; a signal processing unit for performing signal processing of the digital pixel data; and a brightness information detector for detecting, on the basis of the digital pixel data, brightness information incident upon the pixel array unit, wherein the signal processing unit performs at least a part of the signal processing on the basis of the brightness information detected by the brightness information detector.

Description

Imaging equipment, electronic devices and imaging methods

The present disclosure relates to an imaging device, an electronic device, and an imaging method.

In recent years, it has been required to perform various signal processing at high speed on image data captured by an image sensor. In addition, due to advances in semiconductor process technology, semiconductor devices such as image sensor chips, memory chips, and signal processing chips that are packaged by connecting multiple chips with bumps, dies on which image sensors are placed, and memories and signals A semiconductor device in which a die on which a processing circuit or the like is arranged is laminated and packaged has been proposed.

International release WO2018 / 051809A1

When a semiconductor device containing an image sensor and a signal processing circuit (hereinafter referred to as an image pickup device) is mounted on an electronic device such as a smartphone, the signal processing circuit in the image pickup device is the application processor mounted on the electronic device. In many cases, various signal processing is performed according to the instructions.

Since the application processor often has hardware performance capable of performing advanced signal processing at a higher speed than the signal processing circuit in the imaging device, the signal processing circuit in the imaging device follows the instructions of the application processor. It is common to perform signal processing. For example, the brightness adjustment and white balance adjustment of image data can also be performed by the signal processing circuit in the image pickup device, but since the adjustment can be performed with higher accuracy inside the application processor, the adjustment is performed inside the image pickup device. The signal processing circuit of is often handed over to the application processor after performing signal processing on the premise that brightness adjustment and white balance adjustment are performed inside the application processor.

However, when the signal processing circuit in the image pickup device performs signal processing depending on the application processor, the following problems occur.

1. The application processor may shift to sleep mode in order to reduce power consumption, but when the application processor goes into sleep mode, brightness adjustment and white balance adjustment are performed on the image data captured by the image sensor. It cannot be done properly.

2. When performing recognition processing or the like in the imaging device using the data processed by the signal processing circuit in the image pickup device, if the signal processing circuit performs signal processing in response to an instruction from the application processor, it is not always optimal for recognition processing or the like. The signal processing result may not be obtained, and the reliability of the recognition processing or the like may be lowered.

Therefore, the present disclosure provides an imaging device, an electronic device, and an imaging method capable of performing optimum signal processing without depending on an external instruction such as an application processor.

In order to solve the above problems, according to the present disclosure, a pixel array unit having a plurality of pixels that perform photoelectric conversion and a pixel array unit
A converter that converts an analog pixel signal output from the pixel array unit into digital pixel data,
A signal processing unit that performs signal processing on the digital pixel data, and
A brightness information detector that detects brightness information incident on the pixel array unit based on the digital pixel data is provided.
The signal processing unit provides an imaging device that performs at least a part of the signal processing based on the brightness information detected by the brightness information detector.

An information processing unit that performs at least one of a predetermined recognition process and a detection process based on the data signal-processed by the signal processing unit may be provided.

The signal processing unit performs the signal processing according to an instruction from the outside, and if there is no instruction from the outside, performs the signal processing according to the brightness information detected by the brightness information detector.
The information processing unit performs at least one of the recognition process and the detection process based on the data obtained by the signal processing unit performing the signal processing according to the brightness information detected by the brightness information detector. You may.

The signal processing unit
A first signal processing unit that performs a first signal processing on the digital pixel data,
It has a second signal processing unit that performs a second signal processing on the digital pixel data or data that has performed at least a part of the first signal processing.
The second signal processing unit performs at least a part of the second signal processing based on the brightness information detected by the brightness information detector.
An information processing unit that performs at least one of a predetermined recognition process and a detection process based on the output data of the second signal processing unit may be further provided.

The first signal processing and the second signal processing include common signal processing.
The second signal processing unit may perform at least a part of the common signal processing based on the brightness information detected by the brightness information detector.

The common signal processing includes brightness adjustment processing of an image captured by the pixel array unit.
The second signal processing unit may perform the brightness adjustment process based on the brightness information detected by the brightness information detector.

The common signal processing includes a white balance adjustment process of an image captured by the pixel array unit.
The second signal processing unit may perform the white balance adjustment process based on the brightness information detected by the brightness information detector.

The recognition process includes a process of giving input data to a calculation model generated by machine learning and performing a calculation.
The input data may be output data of the signal processing unit.

The second signal processing unit or the information processing unit may perform motion detection processing based on the brightness information detected by the brightness information detector.

Based on the brightness information detected by the brightness information detector, the signal processing unit may include an arithmetic unit that calculates a set value for performing the signal processing.

The first substrate having the pixel array portion and
The converter, the signal processing unit, and the second substrate having the brightness information detector, which are laminated on the first substrate, may be provided.

The first substrate and the second substrate may be bonded by any of a CoC (Chip on Chip) method, a CoW (Chip on Wafer) method, or a WoW (Wafer on Wafer) method.

A gain adjusting unit that adjusts the gain of the analog pixel signal based on the brightness information detected by the brightness information detector may be provided.

The gain adjusting unit adjusts the gain according to an instruction from the outside, and if there is no instruction from the outside, adjusts the gain based on the brightness information detected by the brightness information detector. May be good.

In another aspect of the present disclosure, an imaging device that outputs captured image data and
A processor that performs predetermined signal processing on the image data is provided.
The image pickup device
A pixel array unit having a plurality of pixels that perform photoelectric conversion,
A converter that converts an analog pixel signal output from the pixel array unit into digital pixel data,
A signal processing unit that performs signal processing on the digital pixel data, and
A brightness information detector that detects brightness information incident on the pixel array unit based on the digital pixel data is provided.
The signal processing unit performs at least a part of the signal processing based on the brightness information detected by the brightness information detector.
An electronic device is provided in which the signal-processed image data is supplied to the processor by the signal processing unit.

The processor sends an instruction to the first signal processing unit to perform processing based on the output data of the first signal processing unit, and the first operation mode has lower power consumption than the first operation mode. It has a second operation mode in which one does not send an instruction to the first signal processing unit and does not receive the output data of the first signal processing unit.
The second signal processing unit or the information processing unit performs motion detection processing based on the brightness information detected by the brightness information detector regardless of the operation mode of the processor.
When the processor is in the second operation mode, the second signal processing unit or the information processing unit performs the motion detection process based on the brightness information detected by the brightness information detector, and the motion detection process is performed. When motion is detected by the motion detection process, the processor may be returned from the second operation mode to the first operation mode.

In another aspect of the present disclosure, there is a step of performing photoelectric conversion in the pixel array unit and outputting an analog pixel signal.
The step of converting the analog pixel signal into digital pixel data,
A step of detecting brightness information incident on the pixel array unit based on the digital pixel data, and
Provided is an imaging method including a step of performing signal processing on the digital pixel data and performing at least a part of the signal processing based on the brightness information.

The block diagram which shows the schematic structure of the electronic device provided with the image pickup apparatus according to 1st Embodiment. The block diagram which shows the internal structure of the signal processing unit in an image pickup apparatus, and the signal processing unit in an application processor 3. It is a figure which shows the example which the signal processing part in an image pickup apparatus further has a gain adjustment part for AWB in addition to the internal structure of FIG. The figure which shows an example of the chip structure of the image pickup apparatus of FIG. The figure which shows the layout example of the 1st substrate 31 in which a pixel array part 13 is arranged. The figure which shows the layout example of the 2nd board in which a control part, ADC, a signal processing part, a memory, a DSP, and a selector are arranged. The flowchart which shows the processing procedure performed by the image pickup apparatus by 1st Embodiment. The block diagram which shows the schematic structure of the electronic device provided with the image pickup apparatus by 2nd Embodiment. The block diagram which shows the internal structure of the 1st signal processing part and the 2nd signal processing part in an image pickup apparatus, and the internal structure of a signal processing part in an application processor. It is a figure which shows the example which the 2nd signal processing part in an image pickup apparatus further has a gain adjustment part for AWB in addition to the internal structure of FIG. The control unit controls the timing at which the OPD generates the brightness information detection signal, controls the reading of the brightness information detection signal generated by the OPD, and the individual in the second signal processing unit based on the read brightness information detection signal. The block diagram of the image pickup apparatus 1 in the case of performing the process of calculating the set value set in the processing part of 1 and the control of transmitting the calculated set value to each processing part in a 2nd signal processing part. The block diagram which shows the schematic configuration example of the vehicle control system which is an example of the mobile body control system to which the technique which concerns on this disclosure can be applied. The figure which shows the example of the installation position of the image pickup part. The figure which shows an example of the schematic structure of the endoscopic surgery system to which the technique (the present technique) which concerns on this disclosure can be applied. The block diagram which shows an example of the functional structure of the camera head and CCU shown in FIG. The figure which shows an example of the schematic structure of the diagnosis support system to which the technique which concerns on this disclosure is applied.

Hereinafter, embodiments of the imaging device and the electronic device will be described with reference to the drawings. In the following, the main components of the image pickup device and the electronic device will be mainly described, but the image pickup device and the electronic device may have components and functions not shown or described. The following description does not exclude components or functions not shown or described.

(First Embodiment)
FIG. 1 is a block diagram showing a schematic configuration of an electronic device 2 provided with an image pickup apparatus 1 according to the first embodiment. The electronic device 2 of FIG. 1 includes an image pickup device 1 and an application processor 3. The electronic device 2 in FIG. 1 is a smartphone, a mobile phone, a tablet, a PC, a digital camera, or the like having an imaging function, and the specific device thereof does not matter.

The image pickup device 1 can be realized by one semiconductor device, and this semiconductor device may be called an image sensor or a solid-state image pickup device. The imaging device 1 includes an imaging unit 4, a control unit 5, a converter (hereinafter referred to as ADC: Analog to Digital Converter) 6, a signal processing unit 7, a memory 8, a DSP (Digital Signal Processor) 9, and a selector. 10 and a brightness information detector (hereinafter, OPD: Optical Detector) 11 are provided.

The imaging unit 4 has an optical system 12 and a pixel array unit 13. The optical system 12 includes, for example, a zoom lens, a single focus lens, an aperture, and the like. The optical system 12 guides the incident light to the pixel array unit 13. The pixel array unit 13 has a plurality of pixels arranged in the two-dimensional direction. Each pixel is composed of a plurality of unit pixels for a plurality of colors such as RGB. Each unit pixel has a light receiving element such as a photodiode. The light receiving element photoelectrically converts the incident light and outputs an analog pixel signal. The light incident on the image pickup unit 4 passes through the optical system 12 and is imaged on a light receiving surface in which a plurality of light receiving elements are arranged. Each light receiving element accumulates electric charges according to the intensity of the incident light and outputs an analog pixel signal according to the amount of accumulated electric charges.

The control unit 5 controls each unit in the image pickup apparatus 1 according to an instruction from the application processor 3 or the like. The control unit 5 may be integrated with the DSP 9 described later.

The ADC 6 converts the analog pixel signal output from the imaging unit 4 into digital pixel data. Since the ADC 6 performs A / D conversion, the signal processing unit 7, the DSP 9, the memory 8 and the selector 10 on the subsequent stage side of the ADC 6 handle digital pixel data. A voltage generation circuit may be provided inside the ADC 6 or separately from the ADC 6 to generate a drive voltage for driving the image pickup unit 4 from a power supply voltage or the like supplied to the image pickup apparatus 1.

The signal processing unit 7 performs various signal processing on the digital pixel data. The signal processing unit 7 may perform signal processing on the digital pixel data output from the ADC 6 or may perform signal processing on the digital pixel data output from the ADC 6 and stored in the memory 8. .. The signal processing unit 7 performs signal processing according to an instruction from the outside, and if there is no instruction from the outside, performs signal processing according to the brightness information detected by the OPD 11. The specific contents of signal processing performed by the signal processing unit 7 will be described later. The data processed by the signal processing unit 7 is once stored in the memory 8 and then sent to the DSP 9. Alternatively, the data processed by the signal processing unit 7 may be sent to the DSP 9 without being stored in the memory 8.

The DSP 9 has a function of an information processing unit that performs at least one of a predetermined recognition process and a detection process based on the data signal-processed by the signal processing unit 7. The DSP 9 performs at least one of the recognition process and the detection process based on the data obtained by the signal processing unit 7 performing the signal processing according to the brightness information detected by the OPD 11. The DSP 9 performs arithmetic processing using a machine-learned calculation model by, for example, executing a program stored in the memory 8. Various information about the learned calculation model is stored in the memory 8 in advance, and the DSP 9 reads necessary information about the calculation model from the memory 8 and inputs the output data of the signal processing unit 7 into the calculation model. , Perform arithmetic processing. The specific form of the machine-learned calculation model does not matter, but for example, it is a calculation model by a deep neural network (hereinafter, DNN). This calculation model is designed based on the parameters generated by inputting the output data of the ADC 6 or the output data of the signal processing unit 7 as an input and inputting the training data associated with the label for this input into the trained calculation model. be able to. Further, the DNN may be composed of a multi-layer neural network. The DSP 9 can perform a predetermined recognition process, for example, by an arithmetic process using the DNN. Here, the recognition process is a process of automatically recognizing whether or not the image data, which is the output data of the signal processing unit 7, includes characteristic image information. More specifically, the recognition process is a process in which input data is given to a calculation model generated by machine learning and calculated, and the input data is output data of the signal processing unit 7.

The DSP 9 performs a product-sum calculation of the dictionary coefficient stored in the memory 8 and the image data in the process of executing the calculation process based on the learned calculation model stored in the memory 8. The calculation result by the DSP 9 is stored in the memory 8 and input to the selector 10. The result of the arithmetic processing using the calculation model by DSP9 may be image data or various information (metadata) obtained from the image data. The DSP 9 or the control unit 5 described above may have a function of a memory controller that controls writing and reading to the memory 8, or a memory controller may be provided separately from the DSP 9 and the control unit 5. Further, the DSP 9 may perform detection processing such as motion detection processing and face detection processing. The detection process may be performed by the signal processing unit 7 instead of the DSP 9. Alternatively, the signal processing unit 7 and the DSP 9 may cooperate to perform the detection process.

The memory 8 stores digital pixel data output from the ADC 6, a program executed by the DSP 9, various information related to the learned calculation model used by the DSP 9 for arithmetic processing, and the like. Further, the memory 8 may store the data of the calculation processing result of the DSP 9. The memory 8 is a readable and writable RAM (RandomAccessMemory). By exchanging the information about the calculation model in the memory 8, the DSP 9 can execute various machine learning calculation models, and can perform recognition processing and detection processing having high versatility and a wide range of application. When the DSP 9 performs arithmetic processing based on a calculation model for a specific purpose, the memory 8 may be a ROM (Read Only Memory).

The selector 10 selects and outputs the output data of the signal processing unit 7 or the output data of the DSP 9 based on the selection control signal from the control unit 5. The output data of the selector 10 is input to, for example, the application processor 3.

The OPD 11 detects the brightness information incident on the pixel array unit 13 based on the digital pixel data output from the ADC 6. More specifically, the OPD 11 detects the average brightness of the light incident on the pixel array unit 13 or the brightness based on the integrated value of the incident light based on the digital pixel data. The brightness detected by the OPD 11 is the brightness information around the image pickup device 1. In the present specification, the signal output by the OPD 11 is referred to as a brightness information detection signal. The OPD 11 may generate a brightness information detection signal by calculating the average value of digital pixel data corresponding to the analog pixel signal photoelectrically converted by all the light receiving elements in the pixel array unit 13, or may generate the brightness information detection signal in the pixel array unit 13. The brightness information detection signal may be generated by calculating the average value of the digital pixel data corresponding to the analog pixel signal photoelectrically converted by some of the light receiving elements of the above. Further, a dedicated light receiving element for OPD 11 may be provided in the pixel array unit 13.

The application processor 3 is a semiconductor device separate from the image pickup device 1, and is mounted on the same or different base substrate as the image pickup device 1. The application processor 3 has a CPU (Central Processing Unit) inside the application processor 3 and executes programs such as an operating system and various application software. The application processor 3 includes a signal processing unit 14 as described later, and performs various signal processing. The signal processing unit 14 in the application processor 3 can perform advanced signal processing at a higher speed than the signal processing unit 7 in the image pickup apparatus 1. Further, the application processor 3 includes an OPD 15. The OPD 15 generates a brightness information detection signal based on one frame of digital pixel data captured by the pixel array unit 13 and output from the ADC 6. Since the application processor 3 has a processing performance capable of performing complicated processing in a shorter time than that of the image pickup device 1, there is a high possibility that a brightness information detection signal having higher reliability than that of the OPD 11 in the image pickup device 1 can be generated. ..

In addition, the application processor 3 may be equipped with a function of performing image processing, signal processing, etc., such as a GPU (Graphics Processing Unit) or a baseband processor. The application processor 3 executes various processes on the image data and the calculation result from the image pickup apparatus 1 as necessary, controls the display of the image on the display unit of the electronic device 2, and determines the processing result. Data is transmitted to an external cloud server 21 via a predetermined network 20.

Note that the predetermined network 20 can be applied with various communication networks such as the Internet, a wired LAN (Local Area Network), a wireless LAN, a mobile communication network, and a proximity wireless communication such as Bluetooth (registered trademark). Further, the destination of image data and calculation results is not limited to the cloud server 21, but is various information processing devices having a communication function such as a stand-alone server, a file server, and a communication terminal such as a mobile phone. May be good.

FIG. 1 shows an example in which the application processor 3 sends an instruction to the signal processing unit 7 in the image pickup apparatus 1, but an ISP (Image Signal Processor) that operates under the control of the application processor 3 is provided. May be good. In this case, the ISP may send an instruction to the signal processing unit 7. In the following, an example in which the application processor 3 sends an instruction to the signal processing unit 7 will be described, but in reality, a processor other than the application processor 3 such as an ISP may send an instruction to the signal processing unit 7. It shall be interpreted.

(Internal configuration of signal processing unit 7)
FIG. 2 is a block diagram showing an internal configuration of a signal processing unit 7 in the image pickup apparatus 1 and a signal processing unit 14 in the application processor 3 according to the first embodiment.

The specific content of the signal processing performed by the signal processing unit 7 in the image pickup apparatus 1 is arbitrary. The signal processing unit 7 in the image pickup apparatus 1 of FIG. 2 includes a processing A unit 7a, a gain adjusting unit 7b for automatic exposure (AE: Automatic Exposure), a processing B unit 7c, and a processing C unit 7d. The AE gain adjusting unit 7b performs a process of raising the pixel value of the digital pixel data to make it brighter.

The specific processing contents of the processing A part 7a, the processing B part 7c, and the processing C part 7d are arbitrary, for example, lens shading processing, clamping processing, horizontal cropping processing, defect correction processing, and the like. The lens shading process is a process of increasing the pixel value of the peripheral portion because the peripheral portion becomes dark when the subject light passes through the lens. The clamping process is a process that defines the level of black. The horizontal cropping process is a process of adjusting the image size by cutting out a part of the width of the image. The part to be cut out is the part that does not constitute a valid image. The defect correction process is a process of interpolating pixel defects using peripheral pixel data. Defect correction processing includes static defect correction processing and dynamic defect correction processing. First, static defect correction processing may be performed, and then dynamic defect correction processing may be performed as needed. The static defect correction process is a process of fixedly interpolating known defect pixels using peripheral pixel data. Since the defective pixels are not fixed and the power consumption is large, the dynamic defect correction process may be performed as needed.

As described above, there are various types of signal processing that can be candidates for the processing A unit 7a, the processing B unit 7c, and the processing C unit 7d, and which signal processing is performed in which order is arbitrary. Further, the number of specific signal processes performed by the signal processing unit 7 is also arbitrary, and signal processing may be performed in a number larger or smaller than the four shown in FIG.

The signal processing unit 7 in the image pickup apparatus 1 can perform various signal processing based on the brightness information detection signal output from the OPD 11. The signal processing unit 7 basically performs signal processing according to the instruction of the application processor 3, but when the application processor 3 is stopped, the application processor 3 needs image data from the imaging device 1. Therefore, the signal processing unit 7 in the image pickup apparatus 1 performs signal processing based on the brightness information detection signal output from the OPD 11 in order to generate optimum data for performing recognition processing and detection processing. .. For example, when the ambient brightness of the image pickup apparatus 1 is dark, the gain of the gain adjusting unit 7b for AE is adjusted to be further increased. Further, when the image pickup apparatus 1 is illuminated with a specific light source color, the white balance is biased, so the white balance is adjusted by the gain adjusting unit 7e for AWB.

As described above, the DSP 9 performs recognition processing and detection processing based on the image data captured by the pixel array unit 13, but in order to perform highly reliable recognition processing and detection processing, the image data input to the DSP 9 is performed. Must be data suitable for performing recognition processing and detection processing. The image data input to the DSP 9 is the data after the signal processing unit 7 performs signal processing on the digital pixel data. The signal processing unit 7 performs various signal processing on the digital pixel data based on the brightness information detection signal output from the ODP 11, and obtains image data suitable for the DSP 9 to perform the recognition processing and the detection processing. Generate and send to DSP9. The DSP 9 can improve the recognition accuracy and the detection accuracy by performing the recognition process and the detection process based on the image data output from the signal processing unit 7.

The signal processing unit 14 in the application processor 3 includes a processing X unit 14a, an AE gain adjusting unit 14b, a white balance (AWB: Auto White Balance) gain adjusting unit 14c, a processing Y unit 14d, and a processing Z unit. It has 14e and. The AE gain adjusting unit 14b performs the same processing as the AE gain adjusting unit 7b in the image pickup apparatus 1. The AWB gain adjusting unit 14c properly adjusts the white balance of the image data. The specific contents of the signal processing performed by the processing X unit 14a, the processing Y unit 14d, and the processing Z unit 14e are arbitrary. For example, the processing X unit 14a, the processing Y unit 14d, and the processing Z unit 14e can perform the above-mentioned lens shading processing, clamping processing, horizontal crop processing, defect correction unit processing, and the like.

The signal processing unit 14 in the application processor 3 performs each signal processing based on the brightness information detection signal from the OPD 15. Since the signal processing unit 14 in the application processor 3 can perform more complicated and advanced signal processing than the signal processing unit 7 in the image pickup apparatus 1, the signal processing performed by the signal processing unit 14 in the application processor 3 can be performed. , The reliability is higher than the signal processing performed by the signal processing in the image pickup apparatus 1.

The application processor 3 has a normal operation mode (first operation mode) and a sleep mode (second operation mode). In the normal operation mode, the application processor 3 instructs the signal processing unit 7 in the image pickup apparatus 1 about the content of the signal processing to be performed by the signal processing unit 7. The signal processing unit 7 in the image pickup apparatus 1 performs signal processing according to an instruction from the application processor 3. As described above, since the signal processing unit 14 in the application processor 3 can perform more advanced and complicated signal processing than the signal processing unit 7 in the image pickup device 1 in a short time, the application processor 3 is in the image pickup device 1. It is possible to instruct to send digital pixel data in which some signal processing that can be performed by the signal processing unit 7 is omitted. For example, the application processor 3 instructs the signal processing unit 7 in the image pickup apparatus 1 to send digital pixel data in which the processing of the AE gain adjusting unit 7b and the AWB gain adjusting unit is omitted. According to this instruction, the signal processing unit 7 in the image pickup apparatus 1 omits the processing of the AE gain adjusting unit 7b and the AWB gain adjusting unit, and applies the digital pixel data obtained by performing at least a part of the remaining signal processing. Send to processor 3.

On the other hand, when the application processor 3 shifts to the sleep mode, the application processor 3 does not give an instruction to the signal processing unit 7 in the image pickup apparatus 1 and does not receive data from the signal processing unit 7. In this case, the signal processing unit 7 in the image pickup apparatus 1 adjusts the gains of the AE gain adjusting unit 7b and the AWB gain adjusting unit based on the brightness information detection signal from the OPD 11. For example, the signal processing unit 7 adjusts the gain for AE so that the DSP 9 can input the optimum data to the DNN for performing the recognition process using the DNN in the image pickup apparatus 1.

(Modification example of internal configuration of signal processing unit 7)
FIG. 3 shows an example in which the signal processing unit 7 in the image pickup apparatus 1 further includes an AWB gain adjusting unit 7e in addition to the internal configuration of FIG. In FIG. 3, the AWB gain adjusting unit 7e is arranged between the AE gain adjusting unit 7b and the processing B unit 7c, but it may be arranged at another location. The processing of the AWB gain adjusting unit 7e may be performed before the processing of the AE gain adjusting unit 7b. The AWB gain adjusting unit 7e appropriately adjusts the white balance of the image data in the same manner as the AWB gain adjusting unit 14c.

When the application processor 3 shifts to the sleep mode, the signal processing unit 7 of FIG. 3 adjusts the gains of the AE gain adjusting unit 7b and the AWB gain adjusting unit 7e based on the brightness information detection signal from the OPD 11. .. At this time, the signal processing unit 7 adjusts the gain for AE and the gain for AWB so that the DSP 9 can input the optimum data for performing the recognition process and the detection process using the DNN to the DNN. Since the signal processing unit 7 of FIG. 3 can adjust the gains of the AE gain adjusting unit 7b and the AWB gain adjusting unit 7e based on the brightness information detection signal from the OPD 11, the image data captured by the pixel array unit 13 can be adjusted. Not only can the brightness be optimized, but the white balance can also be optimized. By "optimizing", it means that the DSP 9 provides the DSP 9 with the most suitable input data for performing the recognition process and the detection process.

Although FIGS. 1 to 3 show an example in which the signal processing unit 7 performs at least a part of signal processing based on the brightness information detection signal from the OPD 11, the signal processing unit 7 or the control unit 5 may perform the opD 11 The gain adjustment of the analog pixel signal before the ADC 6 performs the A / D conversion may be instructed based on the brightness information detection signal from. In this case, this instruction signal is sent to the imaging unit 4 to adjust the gain of an amplifier (not shown). In this way, the gain of the analog pixel signal may be adjusted based on the brightness information detection signal from the OPD 11. This gain adjustment may be performed by the application processor 3 in the sleep mode, and may be performed by an instruction from the application processor 3 in the normal operation mode.

(Chip structure of imaging device 1)
Next, the chip structure of the image pickup apparatus 1 of FIG. 1 will be described. FIG. 4 is a diagram showing an example of the chip structure of the image pickup apparatus 1 of FIG. The image pickup apparatus 1 of FIG. 4 is a laminated body in which a first substrate 31 and a second substrate 32 are laminated. The first substrate 31 and the second substrate 32 are sometimes called dies. In the example of FIG. 4, the first substrate 31 and the second substrate 32 are rectangular, but the specific shapes and sizes of the first substrate 31 and the second substrate 32 are arbitrary. Further, the first substrate 31 and the second substrate 32 may have the same size or may be different sizes from each other.

The pixel array unit 13 shown in FIG. 1 is arranged on the first substrate 31. Further, at least a part of the optical system 12 of the imaging unit 4 may be mounted on the first substrate 31 on-chip.

The control unit 5, ADC 6, signal processing unit 7, memory 8, DSP 9, selector 10, and OPD 11 shown in FIG. 1 are arranged on the second substrate 32. In addition, an input / output interface unit (not shown), a power supply circuit, or the like may be arranged on the second substrate 32.

As a specific form of bonding, even if a so-called CoC (Chip on Chip) method is adopted in which the first substrate 31 and the second substrate 32 are cut out from a wafer, separated into individual pieces, and then laminated on top of each other. Good. Alternatively, one of the first substrate 31 and the second substrate 32 (for example, the first substrate 31) is cut out from the wafer and individualized, and then the individualized first substrate 31 is separated into the second substrate 32 before individualization. The so-called CoW (Chip on Wafer) method may be adopted. Alternatively, a so-called WoW (Wafer on Wafer) method, in which the first substrate 31 and the second substrate 32 are bonded together in a wafer state, may be adopted.

For example, plasma bonding or the like can be used as the bonding method between the first substrate 31 and the second substrate 32. However, various other joining methods may be used.

5A and 5B are diagrams showing an example of the layout of the first substrate 31 and the second substrate 32. FIG. 5A shows a layout example of the first substrate 31 on which the pixel array unit 13 is arranged. In the example of FIG. 5A, the pixel array unit 13 is arranged on one side L101 side of the four sides L101 to L104 of the first substrate 31. In other words, the pixel array unit 13 is arranged so that its central portion O101 is closer to the side L101 than the central portion O100 of the first substrate 31. When the surface of the first substrate 31 on which the pixel array portion 13 is provided is rectangular, the side L101 may be, for example, the shorter side of the first substrate 31. However, the present invention is not limited to this, and the pixel array portion 13 may be arranged on the longer side.

In the region close to the side L101 among the four sides of the pixel array unit 13, in other words, in the region between the side L101 and the pixel array unit 13, each unit pixel 101a in the pixel array unit 13 is placed on the second substrate. As wiring for electrically connecting to the ADC 6 arranged in 32, a TSV array 102 in which a plurality of through wires (Through Silicon Via: hereinafter referred to as TSV) penetrating the first substrate 31 are arranged is provided. By bringing the TSV array 102 close to the side L101 where the pixel array unit 13 is close to each other in this way, it becomes easy to secure an arrangement space for the ADC 6 and the like on the second substrate 32.

The TSV array 102 is a region close to one of the two sides L103 and L104 intersecting the side L101 (however, the side L103 may be used), in other words, the side L104 (or the side L103). It may also be provided in the area between the pixel array unit 13 and the pixel array unit 13.

Of the four sides L101 to L104 of the first substrate 31, each of the sides L102 to L103 in which the pixel array portion 13 is not arranged on one side is provided with a pad array 103 having a plurality of pads arranged in a straight line. ing. The pad array 103 may include, for example, a pad (also referred to as a power supply pin) to which a power supply voltage for an analog circuit such as a pixel array unit 13 or an ADC 6 is applied. Further, the pad array 103 may include a pad (also referred to as a power supply pin) to which a power supply voltage for a digital circuit such as a signal processing unit 7, a DSP 9, a memory 8, a selector 10, and a control unit 5 is applied. Alternatively, the pad array 103 may include pads (also referred to as signal pins) for interfaces such as MIPI (Mobile Industry Processor Interface) and SPI (Serial Peripheral Interface). Alternatively, the pad array 103 may include pads (also referred to as signal pins) for input / output of clocks and data. Each pad is electrically connected to, for example, an external power supply circuit or interface circuit via a wire. It is preferable that the pad array 103 and the TSV array 102 are sufficiently separated from each other so that the influence of the reflection of the signal from the wire connected to each pad in the pad array 103 can be ignored.

FIG. 5B shows a layout example of the second board 32 in which the control unit 5, the ADC 6, the signal processing unit 7, the memory 8, the DSP 9, and the selector 10 are arranged. The ADC 6, the control unit 5, the signal processing unit 7, the DSP 9, and the memory 8 are arranged on the second substrate 32. In the layout example of FIG. 5B, the ADC 6 is divided into two regions, an ADC section 6a and a DAC section (Digital to Analog Converter) 6b. The DAC 6b is a circuit that supplies a reference voltage for AD conversion to the ADC unit 6a, and is included in a part of the ADC 6 in a broad sense. Further, although not shown in FIG. 5B, the selector 10 and the OPD 11 are also arranged on the second substrate 32.

Further, the second substrate 32 is provided with wiring 122 which is electrically connected by contacting each TSV (hereinafter, simply referred to as TSV array 102) in the TSV array 102 penetrating the first substrate 31. There is. Further, the second substrate 32 is provided with a pad array 123 in which a plurality of pads electrically connected to each pad in the pad array 103 of the first substrate 31 are linearly arranged.

For the connection between the TSV array 102 and the wiring 122, for example, two TSVs, that is, the TSV provided on the first substrate 31 and the TSV provided from the first substrate 31 to the second substrate 32 are connected by the outer surface of the chip. The so-called twin TSV system may be adopted. Alternatively, a so-called shared TSV method in which the first substrate 31 to the second substrate 32 are connected by a common TSV may be adopted. However, the present invention is not limited to these, and various methods such as a so-called Cu-Cu bonding method in which exposed copper (Cu) is bonded to the bonding surface of the first substrate 31 and the bonding surface of the second substrate 32, respectively. A connection form can be adopted.

The connection form between each pad in the pad array 103 of the first substrate 31 and each pad in the pad array 123 of the second substrate 32 is, for example, wire bonding. However, the present invention is not limited to this, and a connection form such as a through hole or casting may be used.

In the layout example of the second substrate 32, for example, the vicinity of the wiring 122 connected to the TSV array 102 is on the upstream side, and the ADC unit 6a is sequentially arranged from the upstream along the flow of the signal read from the pixel array unit 13. And the signal processing unit 7 and the DSP 9 are arranged. That is, the ADC unit 6a to which the pixel signal read from the pixel array unit 13 is first input is arranged in the vicinity of the wiring 122 which is the most upstream side, and then the signal processing unit 7 is arranged and the most from the wiring 122. The DSP9 is arranged in a distant area. In this way, by arranging the ADC 6 to the DSP 9 from the upstream side along the signal flow, the wiring connecting each part can be shortened. As a result, it is possible to reduce the signal delay, reduce the signal propagation loss, improve the S / N ratio, reduce the power consumption, and the like.

Further, the control unit 5 is arranged in the vicinity of the wiring 122 on the upstream side, for example. In FIG. 5B, the control unit 5 is arranged between the ADC 6 and the signal processing unit 7. By adopting such a layout, it is possible to reduce the signal delay when the control unit 5 controls the pixel array unit 13, reduce the signal propagation loss, improve the S / N ratio, reduce the power consumption, and the like. Further, the signal pins and power supply pins for the analog circuit are arranged together in the vicinity of the analog circuit (for example, the lower side in FIG. 5B), and the remaining signal pins and power supply pins for the digital circuit are arranged in the vicinity of the digital circuit (for example, the lower side in FIG. 5B). , The upper side in FIG. 5B), or the power supply pin for the analog circuit and the power supply pin for the digital circuit can be arranged sufficiently apart.

Further, in the layout shown in FIG. 5B, the DSP 9 is arranged on the side opposite to the ADC portion 6a, which is the most downstream side. By adopting such a layout, in other words, the DSP 9 is arranged in a region that does not overlap with the pixel array unit 13 in the stacking direction of the first substrate 31 and the second substrate 32 (hereinafter, simply referred to as the vertical direction). be able to.

In this way, by configuring the pixel array unit 13 and the DSP 9 not to overlap in the vertical direction, it is possible to reduce noise generated by the DSP 9 executing signal processing from entering the pixel array unit 13. As a result, even when the DSP 9 is operated as a processing unit that executes an operation based on the trained model, it is possible to reduce noise input to the pixel array unit 13 due to the signal processing of the DSP 9, and thus the quality is improved. It is possible to acquire an image with reduced deterioration.

The memory 8 is arranged in the vicinity of the DSP 9 and the signal processing unit 7. Various information about the learned calculation model is stored in the memory 8, and the DSP 9 reads the information about the calculation model from the memory 8 and performs arithmetic processing using the calculation model, and the result of the arithmetic processing is stored in the memory 8. Remember in. Therefore, by arranging the memory 8 in the vicinity of the DSP 9, the signal propagation time when accessing the memory 8 can be shortened, and the DSP 9 can access the memory 8 at high speed.

The pad array 123 is arranged, for example, at a position on the second substrate 32 corresponding to the pad array 103 of the first substrate 31 in the vertical direction. Here, among the pads included in the pad array 123, the pads located in the vicinity of the ADC unit 6a are used for propagating the power supply voltage and the analog signal for the analog circuit (mainly the ADC unit 6a). On the other hand, the pads located near the control unit 5, the signal processing unit 7, the DSP 9, and the memory 8 are power supply voltages and digital signals for digital circuits (mainly, the control unit 5, signal processing unit 7, DSP 9, and memory 8). Used for propagation of. With such a pad layout, the distance on the wiring connecting each pad and each part can be shortened. As a result, it is possible to reduce the signal delay, reduce the propagation loss of the signal and the power supply voltage, improve the S / N ratio, reduce the power consumption, and the like.

FIG. 6 is a flowchart showing a processing procedure performed by the image pickup apparatus 1 according to the first embodiment. First, the pixel array unit 13 performs photoelectric conversion and outputs an analog pixel signal (step S1). Next, the analog pixel signal is converted into digital pixel data (step S2). Next, the brightness information incident on the pixel array unit 13 is detected based on the digital pixel data (step S3). Next, signal processing is performed on the digital pixel data, and at least a part of the signal processing is performed based on the above-mentioned brightness information (step S4).

As described above, in the first embodiment, the OPD 11 is provided in the image pickup apparatus 1 to generate the brightness information detection signal, and the signal processing unit 7 in the image pickup apparatus 1 performs signal processing based on the brightness information detection signal. Therefore, when the application processor 3 does not give an instruction to the signal processing unit 7, the signal processing unit 7 can perform the optimum signal processing for the image pickup apparatus 1. More specifically, in the case of performing information processing such as recognition processing and detection processing based on the output data of the signal processing unit 7 in the image pickup apparatus 1, in order to perform information processing such as recognition processing and detection processing. The signal processing unit 7 can perform signal processing so as to obtain the optimum input data. As a result, the intelligent imaging device 1 can perform highly reliable information processing such as recognition processing and detection processing inside the imaging device 1, and can also perform information processing such as recognition processing and detection processing as well as simply performing imaging. Is obtained.

(Second Embodiment)
FIG. 7 is a block diagram showing a schematic configuration of the electronic device 2 provided with the image pickup apparatus 1 according to the second embodiment. The image pickup device 1 of FIG. 7 is different from the image pickup device 1 of FIG. 1 in that it includes a first signal processing unit 71 and a second signal processing unit 72 instead of the signal processing unit 7 in the image pickup device 1 of FIG. ing.

The first signal processing unit 71 performs the first signal processing on the digital pixel data. The first signal processing unit 71 performs the first signal processing according to the instruction of the application processor 3, and is common to the signal processing unit 7 in the image pickup apparatus 1 of FIG. 1 in this respect. As will be described later, the specific content of the first signal processing performed by the first signal processing unit 71 may be the same as the signal processing performed by the signal processing unit 7 in the image pickup apparatus 1 of FIG. 1, or at least. Some may be different.

The second signal processing unit 72 performs the second signal processing on the digital pixel data or the data obtained by performing at least a part of the first signal processing. The second signal processing unit 72 performs at least a part of the second signal processing based on the brightness information detected by the OPD 11. As described above, at least a part of the second signal processing is performed with reference to the brightness information detected by the OPD 11. The DSP 9 performs at least one of a predetermined recognition process and a detection process based on the output data of the second signal processing unit 72. The second signal processing unit 72 performs signal processing without depending on the instruction of the application processor 3 at least in the sleep mode. That is, the second signal processing unit 72 can perform signal processing at the original judgment of the image pickup apparatus 1. During normal operation of the application processor 3, the second signal processing unit 72 may perform the second signal processing based on the instruction of the application processor 3, or may perform the second signal processing at its own discretion of the imaging device 1. You may go. FIG. 7 shows an example in which the second signal processing unit 72 is arranged on the rear side of the first signal processing unit 71, but the second signal processing unit 72 is in parallel with the first signal processing unit 71. Signal processing can be performed. More specifically, the digital pixel data from the ADC 6 is input to both the first signal processing unit 71 and the second signal processing unit 72, and the first signal processing unit 71 and the second signal processing unit 72 are in parallel. Signal processing may be performed.

Alternatively, the digital pixel data from the ADC 6 is input to the first signal processing unit 71, the data at the stage where the signal processing is halfway performed by the first signal processing unit 71 is output, and the data is input to the second signal processing unit 72. You may. In this case, the second signal processing unit 72 uses the signal processing result of the first signal processing unit 71 halfway, and performs the subsequent signal processing independently of the first signal processing unit 71.

Alternatively, the digital pixel data from the ADC 6 may be input to the first signal processing unit 71, and the data resulting from the signal processing to the end by the first signal processing unit 71 may be input to the second signal processing unit 72. In this case, the second signal processing unit 72 additionally performs its own signal processing after the signal processing of the first signal processing unit 71 is completed.

The first signal processing and the second signal processing may include common signal processing. The second signal processing unit 72 may perform at least a part of common signal processing based on the brightness information detected by the OPD 11.

The common signal processing may include the brightness adjustment processing of the image captured by the pixel array unit 13. Further, the second signal processing unit 72 may perform the brightness adjustment process based on the brightness information detected by the OPD 11.

The common signal processing may include white balance adjustment processing of the image captured by the pixel array unit 13. The second signal processing unit 72 may perform white balance adjustment processing based on the brightness information detected by the OPD 11.

FIG. 8 is a block diagram showing an internal configuration of the first signal processing unit 71 and the second signal processing unit 72 in the image pickup apparatus 1 according to the second embodiment and an internal configuration of the signal processing unit 14 in the application processor 3. Is.

The internal configuration of the first signal processing unit 71 is the same as that of the signal processing unit 7 in the image pickup apparatus 1 of FIG. 4, for example, processing A unit 7a, AE gain adjusting unit 7b, processing B unit 7c, and processing. It has a C portion 7d. The OPD 11 was connected to the signal processing unit 7 in the image pickup apparatus 1 of FIG. 1, but the OPD 11 was not connected to the first signal processing unit 71, and the OPD 11 was connected to the second signal processing unit 72. There is. The first signal processing unit 71 performs various signal processing according to the instructions of the application processor 3, and sends data indicating the final signal processing result to the application processor 3. As described above, the first signal processing unit 71 performs various signal processing in order to generate data suitable for performing advanced signal processing in the application processor 3.

The second signal processing unit 72 performs signal processing at its own discretion of the image pickup apparatus 1. The image pickup apparatus 1 according to the present embodiment performs information processing such as recognition processing and detection processing using the image data captured by the DSP 9 in the pixel array unit 13. Therefore, the second signal processing unit 72 performs signal processing for generating optimum image data for information processing such as recognition processing and detection processing.

The second signal processing unit 72 includes a processing D unit 7f, an AE gain adjusting unit 7g, an AWB gain adjusting unit 7h, a processing E unit 7i, and a processing F unit 7j. The specific contents of signal processing of the processing D unit 7f, the processing E unit 7i, and the processing F unit 7j are arbitrary. For example, the processing D unit 7f, the processing E unit 7i, and the processing F unit 7j each have a lens shading process (Lenz Shade), a demosaic process, a linear matrix / gamma / hue gain process (LinearMatrix Gamma HueGain), and a dewarp process. (Dewarp), gain / YC matrix / normalization processing (Gain YC Matrix Normalize), face recognition / motion detection processing (Face / Motion Detection), and rotation scaling processing (Rotation Scale) are executed. You may.

The lens shading process is a process that raises the pixel value of the peripheral part. The demosaic process is a process of generating pixel data of three colors (RGB) from a four-color arrangement. The linear matrix / gamma / huge gain process is a process such as image linearization, gamma correction, and color adjustment. The dewarp process is a process of correcting a distorted image with a fisheye lens or a wide-angle lens into a flat image as much as possible. The dewarp process is a process of correcting using an adjustment table provided for each lens. The gain / YC matrix / normalization process is a gain adjustment, color adjustment, and normalization process for inputting to the DNN. The digital pixel data from the ADC 6 has a pixel value of 8 to 12 bits, whereas the input data of the DNN has a pixel value of 0 to 1, so that bit shift and normalization processing are required.

What kind of signal processing the second signal processing unit 72 performs depends on the data input to the second signal processing unit 72. For example, when digital pixel data from the ADC 6 is input to the first signal processing unit 71 and the second signal processing unit 72, the second signal processing unit 72 performs signal processing in parallel with the first signal processing unit 71. It will be. Therefore, the second signal processing unit 72 needs to perform signal processing equivalent to the signal processing performed by the first signal processing unit 71. On the other hand, when the data at the stage where the first signal processing unit 71 has partially processed the signal is input to the second signal processing unit 72, the signal processing already performed by the first signal processing unit 71 is the second signal processing. It is not necessary to do it in the part 72. However, since the first signal processing unit 71 performs signal processing according to the instruction of the application processor 3, even if the signal processing has already been performed by the first signal processing unit 71, the second signal processing unit 72 processes the signal again. May be done. Similarly, when the data for which the signal processing completed by the first signal processing unit 71 is completed is input to the second signal processing unit 72, the second signal processing unit 72 is basically different from the first signal processing unit 71. Although signal processing is performed, in some cases, the signal processing performed by the first signal processing unit 71 may be redone by changing the set value.

The OPD 11 is connected to the second signal processing unit 72. The second signal processing unit 72 performs various signal processing based on the brightness information detection signal output from the OPD 11. For example, when the image data captured by the pixel array unit 13 is dark as a whole, signal processing such as brightness adjustment is performed so that optimum data is input to the DNN that performs recognition processing and detection processing.

The application processor 3 may shift to the sleep mode in order to reduce power consumption. When the application processor 3 shifts to the sleep mode, the application processor 3 cannot give an instruction to the first signal processing unit 71. 1 The signal processing unit 71 may not be able to perform effective signal processing. However, since the second signal processing unit 72 performs signal processing regardless of the instruction of the application processor 3, effective signal processing can be continuously performed even when the application processor 3 shifts to the sleep mode.

Further, when the second signal processing unit 72 or the DSP 9 has a motion detection unit, when the motion is detected by the motion detection unit, the application processor 3 is returned to the normal operation mode via the control unit 5. Can be urged to. That is, the operation mode of the application processor 3 can be switched by the signal processing function in the second signal processing unit 72.

FIG. 9 shows an example in which the second signal processing unit 72 in the image pickup apparatus 1 further includes the gain adjusting unit 7e for AWB in addition to the internal configuration of FIG. The second signal processing unit 72 performs signal processing such as brightness adjustment and white balance adjustment based on the brightness information detection signal output from the OPD 11.

In FIGS. 8 and 9 described above, the second signal processing unit 72 calculates the setting value to be set in each processing unit based on the brightness information detection signal output from the OPD 11, and sets the calculated setting value to the calculated setting value. An example in which each processing unit executes signal processing has been described based on the above, but instead of the second signal processing unit 72 performing the calculation processing of the set value based on the brightness information detection signal and the control of the OPD 11, the control of FIG. 7 is performed. A CPU (Central Processing Unit) (not shown) provided separately or separately may perform the operation. Hereinafter, the control unit 5 or the CPU may be referred to as an arithmetic unit.

FIG. 10 shows control of the timing at which the control unit 5 or an arithmetic unit such as a CPU controls the timing at which the OPD 11 generates a brightness information detection signal, the control of reading the brightness information detection signal generated by the OPD 11, and the read brightness information detection. Processing to calculate the set value to be set in each processing unit in the second signal processing unit 72 based on the signal, and control to transmit the calculated set value to each processing unit in the second signal processing unit 72. It is a block diagram of the image pickup apparatus 1 in the case of performing. In the case of FIG. 10, since the control unit 5 in the image pickup apparatus 1 or an arithmetic unit such as a CPU performs control of the OPD 11 and calculation processing based on the brightness information detection signal output from the OPD 11, the second signal processing unit 72 The processing load can be reduced, and the time for each processing unit in the second signal processing unit 72 to perform signal processing can be shortened.

As described above, in the second embodiment, the second signal processing unit 72 that independently performs signal processing in the image pickup apparatus 1 is provided separately from the first signal processing unit 71 that performs signal processing according to the instruction of the application processor 3. Therefore, the second signal processing unit 72 can perform optimal signal processing for performing information processing such as recognition processing and detection processing without being influenced by the instruction of the application processor 3, and the recognition processing and detection processing can be performed. Is more reliable. Further, the application processor 3 may shift to the sleep mode, but even if the application processor 3 shifts to the sleep mode, the second signal processing unit 72 can continue the signal processing, and therefore the application processor 3 can shift to the sleep mode. Regardless of the operation mode of No. 3, information processing such as highly reliable recognition processing and detection processing can be continuously performed in the image pickup apparatus 1.

Further, when the second signal processing unit 72 performs the motion detection process, the control unit 5 notifies the application processor 3 that the motion has been detected, so that the application processor 3 in the sleep mode is returned to the normal operation mode. Can be done.

(Application to other sensors)
In the first and second embodiments described above, the case where the technique according to the present disclosure is applied to the image pickup apparatus (image sensor) 1 for acquiring a two-dimensional image is illustrated, but the technique according to the present disclosure is illustrated. The application destination is not limited to the imaging device. For example, the technique according to the present disclosure can be applied to various light receiving sensors such as a ToF (Time of Flight) sensor, an infrared (IR) sensor, and a DVS (Dynamic Vision Sensor). That is, by making the chip structure of the light receiving sensor a laminated type, it is possible to reduce noise included in the sensor result, reduce the size of the sensor chip, and the like.

(Example of application to mobile)
The technology according to the present disclosure (the present technology) can be applied to various products. For example, the technology according to the present disclosure is realized as a device mounted on any kind of moving body such as an automobile, an electric vehicle, a hybrid electric vehicle, a motorcycle, a bicycle, a personal mobility, an airplane, a drone, a ship, and a robot. You may.

FIG. 11 is a block diagram showing a schematic configuration example of a vehicle control system 12000, which is an example of a mobile control system to which the technique according to the present disclosure can be applied.

The vehicle control system 12000 includes a plurality of electronic control units connected via the communication network 12001. In the example shown in FIG. 11, the vehicle control system 12000 includes a drive system control unit 12010, a body system control unit 12020, an outside information detection unit 12030, an in-vehicle information detection unit 12040, and an integrated control unit 12050. Further, in the example shown in FIG. 11, as the functional configuration of the integrated control unit 12050, a microcomputer 12051, an audio image output unit 12052, and an in-vehicle network I / F (Interface) 12053 are provided.

The drive system control unit 12010 controls the operation of the device related to the drive system of the vehicle according to various programs. For example, the drive system control unit 12010 provides a driving force generator for generating the driving force of the vehicle such as an internal combustion engine or a driving motor, a driving force transmission mechanism for transmitting the driving force to the wheels, and a steering angle of the vehicle. It functions as a control device such as a steering mechanism for adjusting and a braking device for generating a braking force of a vehicle.

The body system control unit 12020 controls the operation of various devices mounted on the vehicle body according to various programs. For example, the body system control unit 12020 functions as a keyless entry system, a smart key system, a power window device, or a control device for various lamps such as headlamps, back lamps, brake lamps, blinkers or fog lamps. In this case, the body system control unit 12020 may be input with radio waves transmitted from a portable device that substitutes for the key or signals of various switches. The body system control unit 12020 receives the input of these radio waves or signals to control the vehicle door lock device, power window device, lamp, and the like.

The vehicle outside information detection unit 12030 detects information outside the vehicle equipped with the vehicle control system 12000. For example, the image pickup unit 12031 is connected to the vehicle exterior information detection unit 12030. The vehicle outside information detection unit 12030 causes the image pickup unit 12031 to capture an image of the outside of the vehicle and receives the captured image. The vehicle exterior information detection unit 12030 may perform object detection processing or distance detection processing such as a person, a vehicle, an obstacle, a sign, or characters on the road surface based on the received image.

The imaging unit 12031 is an optical sensor that receives light and outputs an electric signal according to the amount of the light received. The image pickup unit 12031 can output an electric signal as an image or can output it as distance measurement information. Further, the light received by the imaging unit 12031 may be visible light or invisible light such as infrared light.

The in-vehicle information detection unit 12040 detects the in-vehicle information. For example, a driver state detection unit 12041 that detects the driver's state is connected to the in-vehicle information detection unit 12040. The driver state detection unit 12041 includes, for example, a camera that images the driver, and the in-vehicle information detection unit 12040 determines the degree of fatigue or concentration of the driver based on the detection information input from the driver state detection unit 12041. It may be calculated, or it may be determined whether the driver is dozing.

The microcomputer 12051 calculates the control target value of the driving force generator, the steering mechanism, or the braking device based on the information inside and outside the vehicle acquired by the vehicle exterior information detection unit 12030 or the vehicle interior information detection unit 12040, and the drive system control unit. A control command can be output to 12010. For example, the microcomputer 12051 realizes ADAS (Advanced Driver Assistance System) functions including vehicle collision avoidance or impact mitigation, follow-up driving based on inter-vehicle distance, vehicle speed maintenance driving, vehicle collision warning, vehicle lane deviation warning, and the like. It is possible to perform cooperative control for the purpose of.

Further, the microcomputer 12051 controls the driving force generator, the steering mechanism, the braking device, and the like based on the information around the vehicle acquired by the vehicle exterior information detection unit 12030 or the vehicle interior information detection unit 12040. It is possible to perform coordinated control for the purpose of automatic driving that runs autonomously without depending on the operation.

Further, the microprocessor 12051 can output a control command to the body system control unit 12020 based on the information outside the vehicle acquired by the vehicle exterior information detection unit 12030. For example, the microcomputer 12051 controls the headlamps according to the position of the preceding vehicle or the oncoming vehicle detected by the external information detection unit 12030, and performs coordinated control for the purpose of anti-glare such as switching the high beam to the low beam. It can be carried out.

The audio image output unit 12052 transmits the output signal of at least one of the audio and the image to the output device capable of visually or audibly notifying the passenger or the outside of the vehicle of the information. In the example of FIG. 11, an audio speaker 12061, a display unit 12062, and an instrument panel 12063 are exemplified as output devices. The display unit 12062 may include, for example, at least one of an onboard display and a heads-up display.

FIG. 12 is a diagram showing an example of the installation position of the imaging unit 12031. In FIG. 12, the imaging unit 12031 includes

imaging units

12101, 12102, 12103, 12104, and 12105.

The

imaging units

12101, 12102, 12103, 12104, 12105 are provided at positions such as, for example, the front nose, side mirrors, rear bumpers, back doors, and the upper part of the windshield in the vehicle interior of the vehicle 12100. The image pickup unit 12101 provided on the front nose and the image pickup section 12105 provided on the upper part of the windshield in the vehicle interior mainly acquire an image in front of the vehicle 12100. The

imaging units

12102 and 12103 provided in the side mirrors mainly acquire images of the side of the vehicle 12100. The imaging unit 12104 provided on the rear bumper or the back door mainly acquires an image of the rear of the vehicle 12100. The imaging unit 12105 provided on the upper part of the windshield in the vehicle interior is mainly used for detecting a preceding vehicle, a pedestrian, an obstacle, a traffic light, a traffic sign, a lane, or the like.

Note that FIG. 12 shows an example of the imaging range of the imaging units 12101 to 12104 with a dashed line. The imaging range 12111 indicates the imaging range of the imaging unit 12101 provided on the front nose, the imaging ranges 12112 and 12113 indicate the imaging ranges of the

imaging units

12102 and 12103 provided on the side mirrors, respectively, and the imaging range 12114 indicates the imaging range of the

imaging units

12102 and 12103. The imaging range of the imaging unit 12104 provided on the rear bumper or the back door is shown. For example, by superimposing the image data captured by the imaging units 12101 to 12104, a bird's-eye view image of the vehicle 12100 as viewed from above can be obtained.

At least one of the imaging units 12101 to 12104 may have a function of acquiring distance information. For example, at least one of the image pickup units 12101 to 12104 may be a stereo camera composed of a plurality of image pickup elements, or may be an image pickup element having pixels for phase difference detection.

For example, the microcomputer 12051 has a distance to each three-dimensional object within the imaging range 12111 to 12114 based on the distance information obtained from the imaging units 12101 to 12104, and a temporal change of this distance (relative velocity with respect to the vehicle 12100). By obtaining, it is possible to extract as the preceding vehicle a three-dimensional object that is the closest three-dimensional object on the traveling path of the vehicle 12100 and that travels in substantially the same direction as the vehicle 12100 at a predetermined speed (for example, 0 km / h or more). it can. Further, the microprocessor 12051 can set an inter-vehicle distance to be secured in front of the preceding vehicle in advance, and can perform automatic braking control (including follow-up stop control), automatic acceleration control (including follow-up start control), and the like. In this way, it is possible to perform cooperative control for the purpose of automatic driving or the like in which the vehicle travels autonomously without depending on the operation of the driver.

For example, the microcomputer 12051 converts three-dimensional object data related to a three-dimensional object into two-wheeled vehicles, ordinary vehicles, large vehicles, pedestrians, utility poles, and other three-dimensional objects based on the distance information obtained from the imaging units 12101 to 12104. It can be classified and extracted and used for automatic avoidance of obstacles. For example, the microprocessor 12051 identifies obstacles around the vehicle 12100 into obstacles that are visible to the driver of the vehicle 12100 and obstacles that are difficult to see. Then, the microcomputer 12051 determines the collision risk indicating the risk of collision with each obstacle, and when the collision risk is equal to or higher than the set value and there is a possibility of collision, the microcomputer 12051 via the audio speaker 12061 or the display unit 12062. By outputting an alarm to the driver and performing forced deceleration and avoidance steering via the drive system control unit 12010, driving support for collision avoidance can be provided.

At least one of the imaging units 12101 to 12104 may be an infrared camera that detects infrared rays. For example, the microcomputer 12051 can recognize a pedestrian by determining whether or not a pedestrian is present in the captured image of the imaging units 12101 to 12104. Such pedestrian recognition includes, for example, a procedure for extracting feature points in an image captured by an imaging unit 12101 to 12104 as an infrared camera, and pattern matching processing for a series of feature points indicating the outline of an object to determine whether or not the pedestrian is a pedestrian. It is done by the procedure to determine. When the microcomputer 12051 determines that a pedestrian is present in the captured images of the imaging units 12101 to 12104 and recognizes the pedestrian, the audio image output unit 12052 outputs a square contour line for emphasizing the recognized pedestrian. The display unit 12062 is controlled so as to superimpose and display. Further, the audio image output unit 12052 may control the display unit 12062 so as to display an icon or the like indicating a pedestrian at a desired position.

The above is an example of a vehicle control system to which the technology according to the present disclosure can be applied. The technique according to the present disclosure can be applied to the imaging unit 12031 and the like among the configurations described above. By applying the technique according to the present disclosure to the image pickup unit 12031 or the like, the image pickup unit 12031 or the like can be miniaturized, so that the interior or exterior of the vehicle 12100 can be easily designed. Further, by applying the technique according to the present disclosure to the imaging unit 12031 or the like, it is possible to acquire a clear image with reduced noise, so that it is possible to provide the driver with a more easily visible photographed image. This makes it possible to reduce driver fatigue.

(Example of application to endoscopic surgery system)
The technology according to the present disclosure (the present technology) can be applied to various products. For example, the techniques according to the present disclosure may be applied to endoscopic surgery systems.

FIG. 13 is a diagram showing an example of a schematic configuration of an endoscopic surgery system to which the technique according to the present disclosure (the present technique) can be applied.

FIG. 13 illustrates how the surgeon (doctor) 11131 is performing surgery on patient 11132 on patient bed 11133 using the endoscopic surgery system 11000. As shown, the endoscopic surgery system 11000 includes an endoscope 11100, other surgical tools 11110 such as an abdominal tube 11111 and an energy treatment tool 11112, and a support arm device 11120 that supports the endoscope 11100. , A cart 11200 equipped with various devices for endoscopic surgery.

The endoscope 11100 is composed of a lens barrel 11101 in which a region having a predetermined length from the tip is inserted into the body cavity of the patient 11132, and a camera head 11102 connected to the base end of the lens barrel 11101. In the illustrated example, the endoscope 11100 configured as a so-called rigid mirror having a rigid barrel 11101 is illustrated, but the endoscope 11100 may be configured as a so-called flexible mirror having a flexible barrel. Good.

An opening in which an objective lens is fitted is provided at the tip of the lens barrel 11101. A light source device 11203 is connected to the endoscope 11100, and the light generated by the light source device 11203 is guided to the tip of the lens barrel by a light guide extending inside the lens barrel 11101 to be an objective. It is irradiated toward the observation target in the body cavity of the patient 11132 through the lens. The endoscope 11100 may be a direct endoscope, a perspective mirror, or a side endoscope.

An optical system and an image sensor are provided inside the camera head 11102, and the reflected light (observation light) from the observation target is focused on the image sensor by the optical system. The observation light is photoelectrically converted by the image pickup device, and an electric signal corresponding to the observation light, that is, an image signal corresponding to the observation image is generated. The image signal is transmitted as RAW data to the camera control unit (CCU: Camera Control Unit) 11201.

The CCU11201 is composed of a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), etc., and comprehensively controls the operations of the endoscope 11100 and the display device 11202. Further, the CCU 11201 receives an image signal from the camera head 11102, and performs various image processing on the image signal for displaying an image based on the image signal, such as development processing (demosaic processing).

The display device 11202 displays an image based on the image signal processed by the CCU 11201 under the control of the CCU 11201.

The light source device 11203 is composed of, for example, a light source such as an LED (light LED radio), and supplies irradiation light to the endoscope 11100 when photographing an operating part or the like.

The input device 11204 is an input interface for the endoscopic surgery system 11000. The user can input various information and input instructions to the endoscopic surgery system 11000 via the input device 11204. For example, the user inputs an instruction to change the imaging conditions (type of irradiation light, magnification, focal length, etc.) by the endoscope 11100.

The treatment tool control device 11205 controls the drive of the energy treatment tool 11112 for ablation of tissue, incision, sealing of blood vessels, and the like. The pneumoperitoneum device 11206 uses a gas in the pneumoperitoneum tube 11111 to inflate the body cavity of the patient 11132 for the purpose of securing the field of view by the endoscope 11100 and securing the work space of the operator. To send. Recorder 11207 is a device capable of recording various information related to surgery. The printer 11208 is a device capable of printing various information related to surgery in various formats such as text, images, and graphs.

The light source device 11203 that supplies the irradiation light to the endoscope 11100 when photographing the surgical site can be composed of, for example, an LED, a laser light source, or a white light source composed of a combination thereof. When a white light source is configured by combining RGB laser light sources, the output intensity and output timing of each color (each wavelength) can be controlled with high accuracy. Therefore, the light source device 11203 adjusts the white balance of the captured image. It can be carried out. Further, in this case, the laser light from each of the RGB laser light sources is irradiated to the observation target in a time-divided manner, and the drive of the image sensor of the camera head 11102 is controlled in synchronization with the irradiation timing to support each of RGB. It is also possible to capture the image in a time-divided manner. According to this method, a color image can be obtained without providing a color filter on the image sensor.

Further, the drive of the light source device 11203 may be controlled so as to change the intensity of the output light at predetermined time intervals. By controlling the drive of the image sensor of the camera head 11102 in synchronization with the timing of changing the light intensity to acquire an image in a time-divided manner and synthesizing the image, so-called high dynamic without blackout and overexposure. Range images can be generated.

Further, the light source device 11203 may be configured to be able to supply light in a predetermined wavelength band corresponding to special light observation. In special light observation, for example, by utilizing the wavelength dependence of light absorption in body tissue to irradiate light in a narrow band as compared with the irradiation light (that is, white light) in normal observation, the surface layer of the mucous membrane. So-called narrow band imaging, in which a predetermined tissue such as a blood vessel is photographed with high contrast, is performed. Alternatively, in the special light observation, fluorescence observation may be performed in which an image is obtained by fluorescence generated by irradiating with excitation light. In fluorescence observation, the body tissue is irradiated with excitation light to observe the fluorescence from the body tissue (autofluorescence observation), or a reagent such as indocyanine green (ICG) is locally injected into the body tissue and the body tissue is injected. It is possible to obtain a fluorescence image by irradiating excitation light corresponding to the fluorescence wavelength of the reagent. The light source device 11203 may be configured to be capable of supplying narrow band light and / or excitation light corresponding to such special light observation.

FIG. 14 is a block diagram showing an example of the functional configuration of the camera head 11102 and CCU11201 shown in FIG.

The camera head 11102 includes a lens unit 11401, an imaging unit 11402, a driving unit 11403, a communication unit 11404, and a camera head control unit 11405. CCU11201 has a communication unit 11411, an image processing unit 11412, and a control unit 11413. The camera head 11102 and CCU11201 are communicatively connected to each other by a transmission cable 11400.

The lens unit 11401 is an optical system provided at a connection portion with the lens barrel 11101. The observation light taken in from the tip of the lens barrel 11101 is guided to the camera head 11102 and incident on the lens unit 11401. The lens unit 11401 is configured by combining a plurality of lenses including a zoom lens and a focus lens.

The image sensor constituting the image pickup unit 11402 may be one (so-called single plate type) or a plurality (so-called multi-plate type). When the image pickup unit 11402 is composed of a multi-plate type, for example, each image pickup element may generate an image signal corresponding to each of RGB, and a color image may be obtained by synthesizing them. Alternatively, the image pickup unit 11402 may be configured to have a pair of image pickup elements for acquiring image signals for the right eye and the left eye corresponding to 3D (dimensional) display, respectively. The 3D display enables the operator 11131 to more accurately grasp the depth of the biological tissue in the surgical site. When the image pickup unit 11402 is composed of a multi-plate type, a plurality of lens units 11401 may be provided corresponding to each image pickup element.

Further, the imaging unit 11402 does not necessarily have to be provided on the camera head 11102. For example, the imaging unit 11402 may be provided inside the lens barrel 11101 immediately after the objective lens.

The drive unit 11403 is composed of an actuator, and the zoom lens and focus lens of the lens unit 11401 are moved by a predetermined distance along the optical axis under the control of the camera head control unit 11405. As a result, the magnification and focus of the image captured by the imaging unit 11402 can be adjusted as appropriate.

The communication unit 11404 is composed of a communication device for transmitting and receiving various information to and from the CCU11201. The communication unit 11404 transmits the image signal obtained from the image pickup unit 11402 as RAW data to the CCU 11201 via the transmission cable 11400.

Further, the communication unit 11404 receives a control signal for controlling the drive of the camera head 11102 from the CCU 11201 and supplies the control signal to the camera head control unit 11405. The control signal includes, for example, information to specify the frame rate of the captured image, information to specify the exposure value at the time of imaging, and / or information to specify the magnification and focus of the captured image, and the like. Contains information about the condition.

The imaging conditions such as the frame rate, exposure value, magnification, and focus may be appropriately specified by the user, or may be automatically set by the control unit 11413 of the CCU 11201 based on the acquired image signal. Good. In the latter case, the so-called AE (Auto Exposure) function, AF (Auto Focus) function, and AWB (Auto White Balance) function are mounted on the endoscope 11100.

The camera head control unit 11405 controls the drive of the camera head 11102 based on the control signal from the CCU 11201 received via the communication unit 11404.

The communication unit 11411 is composed of a communication device for transmitting and receiving various information to and from the camera head 11102. The communication unit 11411 receives an image signal transmitted from the camera head 11102 via the transmission cable 11400.

Further, the communication unit 11411 transmits a control signal for controlling the drive of the camera head 11102 to the camera head 11102. Image signals and control signals can be transmitted by telecommunications, optical communication, or the like.

The image processing unit 11412 performs various image processing on the image signal which is the RAW data transmitted from the camera head 11102.

The control unit 11413 performs various controls related to the imaging of the surgical site and the like by the endoscope 11100 and the display of the captured image obtained by the imaging of the surgical site and the like. For example, the control unit 11413 generates a control signal for controlling the drive of the camera head 11102.

Further, the control unit 11413 causes the display device 11202 to display an image captured by the surgical unit or the like based on the image signal processed by the image processing unit 11412. At this time, the control unit 11413 may recognize various objects in the captured image by using various image recognition techniques. For example, the control unit 11413 detects the shape, color, and the like of the edge of an object included in the captured image to remove surgical tools such as forceps, a specific biological part, bleeding, and mist when using the energy treatment tool 11112. Can be recognized. When displaying the captured image on the display device 11202, the control unit 11413 may superimpose and display various surgical support information on the image of the surgical unit by using the recognition result. By superimposing and displaying the surgical support information and presenting it to the surgeon 11131, it is possible to reduce the burden on the surgeon 11131 and to allow the surgeon 11131 to proceed with the surgery reliably.

The transmission cable 11400 that connects the camera head 11102 and CCU11201 is an electric signal cable that supports electric signal communication, an optical fiber that supports optical communication, or a composite cable thereof.

Here, in the illustrated example, the communication was performed by wire using the transmission cable 11400, but the communication between the camera head 11102 and the CCU11201 may be performed wirelessly.

The above is an example of an endoscopic surgery system to which the technology according to the present disclosure can be applied. The technique according to the present disclosure can be applied to, for example, the image pickup unit 11402 of the camera head 11102 among the configurations described above. By applying the technique according to the present disclosure to the camera head 11102, the camera head 11102 and the like can be miniaturized, so that the endoscopic surgery system 11000 can be miniaturized. Further, by applying the technique according to the present disclosure to the camera head 11102 or the like, it is possible to acquire a clear image with reduced noise, so that it is possible to provide an operator with a more easily visible photographed image. As a result, it becomes possible to reduce the fatigue of the operator.

Although the endoscopic surgery system has been described here as an example, the technique according to the present disclosure may be applied to other, for example, a microscopic surgery system.

(Example of application to WSI (Whole Slide Imaging) system)
The technology according to the present disclosure can be applied to various products. For example, even if the technique according to the present disclosure is applied to a pathological diagnosis system or a support system thereof (hereinafter referred to as a diagnosis support system) in which a doctor or the like observes cells or tissues collected from a patient to diagnose a lesion. Good. This diagnostic support system may be a WSI (Whole Slide Imaging) system that diagnoses or supports lesions based on images acquired by using digital pathology technology.

FIG. 15 is a diagram showing an example of a schematic configuration of a diagnostic support system 5500 to which the technique according to the present disclosure is applied. As shown in FIG. 15, the diagnostic support system 5500 includes one or more pathological systems 5510. Further, the medical information system 5530 and the out-licensing device 5540 may be included.

Each of the one or more pathological systems 5510 is a system mainly used by a pathologist, and is introduced into, for example, a laboratory or a hospital. Each pathological system 5510 may be introduced in different hospitals, and may be installed in various networks such as WAN (Wide Area Network) (including the Internet), LAN (Local Area Network), public network, and mobile communication network. It is connected to the medical information system 5530 and the out-licensing device 5540 via the system.

Each pathology system 5510 includes a microscope 5511, a server 5512, a display control device 5513, and a display device 5514.

The microscope 5511 has the function of an optical microscope, images an observation object housed on a glass slide, and acquires a pathological image which is a digital image. The object to be observed is, for example, a tissue or cell collected from a patient, and may be a piece of meat, saliva, blood, or the like of an organ.

The server 5512 stores and stores the pathological image acquired by the microscope 5511 in a storage unit (not shown). When the server 5512 receives a browsing request from the display control device 5513, the server 5512 searches for a pathological image from a storage unit (not shown) and sends the searched pathological image to the display control device 5513.

The display control device 5513 sends a viewing request for the pathological image received from the user to the server 5512. Then, the display control device 5513 displays the pathological image received from the server 5512 on the display device 5514 using a liquid crystal, EL (Electro-Luminescence), CRT (35CathodeRayTube), or the like. The display device 5514 may be compatible with 4K or 8K, and is not limited to one, and may be a plurality of display devices.

Here, when the object to be observed is a solid substance such as a piece of meat of an organ, the object to be observed may be, for example, a stained thin section. The thin section may be prepared, for example, by slicing a block piece cut out from a sample such as an organ. Further, when slicing, the block pieces may be fixed with paraffin or the like.

Various stains such as general stain showing the morphology of the tissue such as HE (Hematoxylin-Eosin) stain and immunostaining showing the immune status of the tissue such as IHC (Immunohistochemistry) stain may be applied to the staining of thin sections. .. At that time, one thin section may be stained with a plurality of different reagents, or two or more thin sections (also referred to as adjacent thin sections) continuously cut out from the same block piece may be different from each other. It may be dyed using.

The microscope 5511 may include a low-resolution imaging unit for imaging at low resolution and a high-resolution imaging unit for imaging at high resolution. The low-resolution imaging unit and the high-resolution imaging unit may have different optical systems or may be the same optical system. In the case of the same optical system, the resolution of the microscope 5511 may be changed according to the image pickup target.

The glass slide containing the observation object is placed on a stage located within the angle of view of the microscope 5511. First, the microscope 5511 acquires an entire image within the angle of view using a low-resolution imaging unit, and identifies a region of an observation object from the acquired overall image. Subsequently, the microscope 5511 divides the region where the observation object exists into a plurality of divided regions of a predetermined size, and sequentially captures each divided region by the high-resolution imaging unit to acquire a high-resolution image of each divided region. To do. In switching the target divided region, the stage may be moved, the imaging optical system may be moved, or both of them may be moved. Further, each divided region may overlap with the adjacent divided region in order to prevent the occurrence of an imaging omission region due to unintended sliding of the glass slide. Further, the whole image may include identification information for associating all 36 body images with the patient. This identification information may be, for example, a character string, a QR code (registered trademark), or the like.

The high resolution image acquired by the microscope 5511 is input to the server 5512. The server 5512 divides each high-resolution image into smaller-sized partial images (hereinafter referred to as tile images). For example, the server 5512 divides one high-resolution image into a total of 100 tile images of 10 × 10 in length and width. At that time, if the adjacent divided regions overlap, the server 5512 may perform stitching processing on the high-resolution images adjacent to each other by using a technique such as template matching. In that case, the server 5512 may generate a tile image by dividing the entire high-resolution image pasted by the stitching process. However, the tile image may be generated from the high resolution image before the stitching process.

Further, the server 5512 can generate a tile image of a smaller size by further dividing the tile image. The generation of such a tile image may be repeated until a tile image having a size set as the minimum unit is generated.

When the tile image of the smallest unit is generated in this way, the server 5512 executes the tile composition process of generating one tile image by synthesizing a predetermined number of adjacent tile images for all the tile images. This tile composition process can be repeated until one tile image is finally generated. By such processing, a tile image group having a pyramid structure in which each layer is composed of one or more tile images is generated. In this pyramid structure, the tile image of one layer and the tile image of a different layer from this layer have the same number of pixels, but their resolutions are different. For example, when a total of four tile images of 2 × 2 are combined to generate one tile image in the upper layer, the resolution of the tile image in the upper layer is 1/2 times the resolution of the tile image in the lower layer used for composition. It has become.

By constructing a tile image group having such a pyramid structure, it is possible to switch the 37 level of detail of the observation object displayed on the display device depending on the hierarchy to which the tile image to be displayed belongs. For example, when the tile image of the lowest layer is used, the narrow area of the observation object is displayed in detail, and the wider area of the observation object is displayed coarser as the tile image of the upper layer is used. it can.

The generated tile image group of the pyramid structure is stored in a storage unit (not shown) together with identification information (referred to as tile identification information) that can uniquely identify each tile image, for example. When the server 5512 receives a request for acquiring a tile image including tile identification information from another device (for example, display control device 5513 or derivation device 5540), the server 5512 transmits the tile image corresponding to the tile identification information to the other device. To do.

Note that the tile image, which is a pathological image, may be generated for each imaging condition such as focal length and staining conditions. When a tile image is generated for each imaging condition, a specific pathological image and another pathological image corresponding to an imaging condition different from the specific imaging condition, which is another pathological image in the same region as the specific pathological image, are displayed. It may be displayed side by side. Specific imaging conditions may be specified by the viewer. Further, when a plurality of imaging conditions are specified for the viewer, pathological images of the same region corresponding to each imaging condition may be displayed side by side.

Further, the server 5512 may store the tile image group having a pyramid structure in a storage device other than the server 5512, for example, a cloud server. Further, a part or all of the tile image generation process as described above may be executed by a cloud server or the like.

The display control device 5513 extracts a desired tile image from the tile image group having a pyramid structure in response to an input operation from the user, and outputs this to the display device 5514. By such a process, the user can obtain the feeling of observing the observation object while changing the observation magnification. That is, the display control device 5513 functions as a virtual microscope. The virtual observation magnification here actually corresponds to the resolution.

Any method may be used as the method for capturing a high-resolution image. The divided area may be imaged while repeatedly stopping and moving the stay 38 to acquire a high-resolution image, or the divided area may be imaged while moving the stage at a predetermined speed to acquire a high-resolution image on the strip. You may. In addition, the process of generating a tile image from a high-resolution image is not an indispensable configuration, and by gradually changing the resolution of the entire high-resolution image bonded by the stitching process, an image whose resolution changes stepwise can be created. It may be generated. Even in this case, it is possible to present the user stepwise from a low-resolution image in a wide area to a high-resolution image in a narrow area.

The medical information system 5530 is a so-called electronic medical record system, and stores information related to diagnosis such as patient identification information, patient disease information, test information and image information used for diagnosis, diagnosis result, and prescription drug. For example, a pathological image obtained by imaging an observation object of a patient can be displayed on the display device 5514 by the display control device 5513 after being temporarily stored via the server 5512. A pathologist using the pathological system 5510 makes a pathological diagnosis based on a pathological image displayed on the display device 5514. The results of the pathological diagnosis made by the pathologist are stored in the medical information system 5530.

The derivation device 5540 can perform analysis on the pathological image. A learning model created by machine learning can be used for this analysis. The derivation device 5540 may derive a classification result of a specific region, an organization identification result, or the like as the analysis result. Further, the derivation device 5540 may derive identification results such as cell information, number, position, and luminance information, and scoring information for them. These information derived by the derivation device 5540 may be displayed on the display device 5514 of the pathology system 5510 as diagnostic support information.

The out-licensing device 5540 may be a server system composed of one or more servers (including a cloud server) and the like. Further, the out-licensing device 5540 may be configured to be incorporated in, for example, a display control device 5513 or a server 5512 in the pathology system 5510. That is, various analyzes on the pathological image may be performed within the pathological system 5510.

The technique according to the present disclosure can be suitably applied to, for example, the microscope 5511 among the configurations described above. Specifically, the technique according to the present disclosure can be applied to the low-resolution imaging unit and / or the high-resolution imaging unit in the microscope 5511. By applying the technique according to the present disclosure to the low-resolution imaging unit and / or the high-resolution imaging unit, the low-resolution imaging unit and / or the high-resolution imaging unit can be miniaturized, and the microscope 5511 can be miniaturized. .. As a result, the microscope 5511 can be easily transported, so that system introduction, system recombination, and the like can be facilitated. Furthermore, by applying the technique according to the present disclosure to the low-resolution imaging unit and / or the high-resolution imaging unit, part or all of the processing from the acquisition of the pathological image to the analysis of the pathological image is performed on the fly in the microscope 5511. Since it is possible, it is possible to output diagnostic support information more quickly and accurately.

The configuration described above can be applied not only to the diagnostic support system but also to general biological microscopes such as confocal microscopes, fluorescence microscopes, and video microscopes. Here, the observation target may be a biological sample such as cultured cells, fertilized eggs, sperms, a biological material such as a cell sheet or a three-dimensional cell tissue, or a living body such as a zebrafish or a mouse. Further, the object to be observed is not limited to the glass slide, and can be observed in a state of being stored in a well plate, a petri dish, or the like.

Further, a moving image may be generated from a still image of an observation object acquired by using a microscope. For example, a moving image may be generated from a still image continuously captured for a predetermined period, or an image sequence may be generated from a still image captured at a predetermined interval. In this way, by generating a moving image from a still image, the movements such as beating and elongation of cancer cells, nerve cells, myocardial tissue, sperm, migration, and the division process of cultured cells and fertilized eggs can be observed. It is possible to analyze the dynamic characteristics of objects using machine learning.

The present technology can have the following configurations.
(1) A pixel array unit having a plurality of pixels that perform photoelectric conversion, and
A converter that converts an analog pixel signal output from the pixel array unit into digital pixel data,
A signal processing unit that performs signal processing on the digital pixel data, and
A brightness information detector that detects brightness information incident on the pixel array unit based on the digital pixel data is provided.
The signal processing unit is an imaging device that performs at least a part of the signal processing based on the brightness information detected by the brightness information detector.
(2) The image pickup apparatus according to (1) above, further comprising an information processing unit that performs at least one of a predetermined recognition process and a detection process based on the data signal processed by the signal processing unit.
(3) The signal processing unit performs the signal processing according to an instruction from the outside, and if there is no instruction from the outside, the signal processing according to the brightness information detected by the brightness information detector. And
The information processing unit performs at least one of the recognition process and the detection process based on the data obtained by the signal processing unit performing the signal processing according to the brightness information detected by the brightness information detector. , The imaging device according to (2) above.
(4) The signal processing unit
A first signal processing unit that performs a first signal processing on the digital pixel data,
It has a second signal processing unit that performs a second signal processing on the digital pixel data or data that has performed at least a part of the first signal processing.
The second signal processing unit performs at least a part of the second signal processing based on the brightness information detected by the brightness information detector.
The image pickup apparatus according to (1), further comprising an information processing unit that performs at least one of a predetermined recognition process and a detection process based on the output data of the second signal processing unit.
(5) The first signal processing and the second signal processing include common signal processing.
The imaging device according to (4), wherein the second signal processing unit performs at least a part of the common signal processing based on the brightness information detected by the brightness information detector.
(6) The common signal processing includes a brightness adjustment processing of an image captured by the pixel array unit.
The imaging device according to (5), wherein the second signal processing unit performs the brightness adjustment process based on the brightness information detected by the brightness information detector.
(7) The common signal processing includes a white balance adjustment process of an image captured by the pixel array unit.
The imaging device according to (5) or (6) above, wherein the second signal processing unit performs the white balance adjustment process based on the brightness information detected by the brightness information detector.
(8) The recognition process includes a process of giving input data to a calculation model generated by machine learning and performing a calculation.
The imaging device according to any one of (2) to (7) above, wherein the input data is output data of the signal processing unit.
(9) The second signal processing unit or the information processing unit performs motion detection processing based on the brightness information detected by the brightness information detector, any one of (4) to (8). The imaging apparatus according to.
(10) Of (1) to (9), the signal processing unit includes an arithmetic unit that calculates a set value for performing the signal processing based on the brightness information detected by the brightness information detector. The imaging device according to any one item.
(11) A first substrate having the pixel array portion and
The imaging image according to any one of (1) to (10), comprising the converter, the signal processing unit, and the second substrate having the brightness information detector, which are laminated on the first substrate. apparatus.
(12) The first substrate and the second substrate are bonded to each other by any of a CoC (Chip on Chip) method, a CoW (Chip on Wafer) method, or a WoW (Wafer on Wafer) method (11). The imaging apparatus according to.
(13) The item according to any one of (1) to (12), further comprising a gain adjusting unit that adjusts the gain of the analog pixel signal based on the brightness information detected by the brightness information detector. Imaging device.
(14) The gain adjusting unit adjusts the gain in response to an external instruction, and if there is no external instruction, the gain adjustment is based on the brightness information detected by the brightness information detector. (13).
(15) An imaging device that outputs captured image data and
A processor that performs predetermined signal processing on the image data is provided.
The image pickup device
A pixel array unit having a plurality of pixels that perform photoelectric conversion,
A converter that converts an analog pixel signal output from the pixel array unit into digital pixel data,
A signal processing unit that performs signal processing on the digital pixel data, and
A brightness information detector that detects brightness information incident on the pixel array unit based on the digital pixel data is provided.
The signal processing unit performs at least a part of the signal processing based on the brightness information detected by the brightness information detector.
An electronic device in which the image data processed by the signal processing unit is supplied to the processor.
(16) The signal processing unit
A first signal processing unit that performs a first signal processing on the digital pixel data,
It has a second signal processing unit that performs a second signal processing on the digital pixel data or data that has performed at least a part of the first signal processing.
The second signal processing unit performs at least a part of the second signal processing based on the brightness information detected by the brightness information detector.
The electronic device according to (15), further comprising an information processing unit that performs at least one of a predetermined recognition process and a detection process based on the output data of the second signal processing unit.
(17) The processor sends an instruction to the first signal processing unit to perform processing based on the output data of the first signal processing unit, and has lower power consumption than the first operation mode. It has a second operation mode in which an instruction is not sent to the first signal processing unit and the output data of the first signal processing unit is not received.
The second signal processing unit or the information processing unit performs motion detection processing based on the brightness information detected by the brightness information detector regardless of the operation mode of the processor.
When the processor is in the second operation mode, the second signal processing unit or the information processing unit performs the motion detection process based on the brightness information detected by the brightness information detector, and the motion detection process is performed. The electronic device according to (16), wherein when motion is detected by the motion detection process, the processor is returned from the second operation mode to the first operation mode.
(18) A step of performing photoelectric conversion in the pixel array section and outputting an analog pixel signal.
The step of converting the analog pixel signal into digital pixel data,
A step of detecting brightness information incident on the pixel array unit based on the digital pixel data, and
An imaging method comprising a step of performing signal processing on the digital pixel data and performing at least a part of the signal processing based on the brightness information.

The aspect of the present disclosure is not limited to the individual embodiments described above, but also includes various modifications that can be conceived by those skilled in the art, and the effects of the present disclosure are not limited to the above-mentioned contents. That is, various additions, changes and partial deletions are possible without departing from the conceptual idea and purpose of the present disclosure derived from the contents defined in the claims and their equivalents.

1 Imaging device, 2 Electronic equipment, 3 Application processor, 4 Imaging unit, 5 Control unit, 6 Converter, 7 Signal processing unit, 7a processing A unit, 7b AE gain adjustment unit, 7c processing B unit, 7d processing C unit , 7e AWB gain adjustment unit, 7f processing D unit, 7g AE gain adjustment unit, 7h AWB gain adjustment unit, 7i processing E unit, 7j processing F unit, 8 memory, 9 DSP, 10 selector, 11 brightness information Detector, 12 optical system, 13 pixel array unit, 14 signal processing unit, 14a processing X unit, 14b AE gain adjustment unit, 14c AWB gain adjustment unit, 14d processing Y unit, 14e processing Z unit, 15 OPD, 20 Network, 21 cloud server, 31 1st board, 32 2nd board, 71 1st signal processing unit, 72 2nd signal processing unit

Claims

A pixel array unit having a plurality of pixels that perform photoelectric conversion,
A converter that converts an analog pixel signal output from the pixel array unit into digital pixel data,
A signal processing unit that performs signal processing on the digital pixel data, and
A brightness information detector that detects brightness information incident on the pixel array unit based on the digital pixel data is provided.
The signal processing unit is an imaging device that performs at least a part of the signal processing based on the brightness information detected by the brightness information detector.
The imaging device according to claim 1, further comprising an information processing unit that performs at least one of a predetermined recognition process and a detection process based on the data processed by the signal processing unit.
The signal processing unit performs the signal processing according to an instruction from the outside, and if there is no instruction from the outside, performs the signal processing according to the brightness information detected by the brightness information detector.
The information processing unit performs at least one of the recognition process and the detection process based on the data obtained by the signal processing unit performing the signal processing according to the brightness information detected by the brightness information detector. , The imaging apparatus according to claim 2.
The signal processing unit
A first signal processing unit that performs a first signal processing on the digital pixel data,
It has a second signal processing unit that performs a second signal processing on the digital pixel data or data that has performed at least a part of the first signal processing.
The second signal processing unit performs at least a part of the second signal processing based on the brightness information detected by the brightness information detector.
The imaging apparatus according to claim 1, further comprising an information processing unit that performs at least one of a predetermined recognition process and a detection process based on the output data of the second signal processing unit.
The first signal processing and the second signal processing include common signal processing.
The imaging device according to claim 4, wherein the second signal processing unit performs at least a part of the common signal processing based on the brightness information detected by the brightness information detector.
The common signal processing includes brightness adjustment processing of an image captured by the pixel array unit.
The imaging device according to claim 5, wherein the second signal processing unit performs the brightness adjustment process based on the brightness information detected by the brightness information detector.
The common signal processing includes a white balance adjustment process of an image captured by the pixel array unit.
The imaging device according to claim 5, wherein the second signal processing unit performs the white balance adjustment process based on the brightness information detected by the brightness information detector.
The recognition process includes a process of giving input data to a calculation model generated by machine learning and performing a calculation.
The imaging device according to claim 2, wherein the input data is output data of the signal processing unit.
The imaging device according to claim 4, wherein the second signal processing unit or the information processing unit performs motion detection processing based on the brightness information detected by the brightness information detector.
The imaging device according to claim 1, further comprising an arithmetic unit in which the signal processing unit calculates a set value for performing the signal processing based on the brightness information detected by the brightness information detector.
The first substrate having the pixel array portion and
The imaging device according to claim 1, further comprising the converter, the signal processing unit, and the second substrate having the brightness information detector, which are laminated on the first substrate.
The eleventh aspect of claim 11, wherein the first substrate and the second substrate are bonded by any of a CoC (Chip on Chip) method, a CoW (Chip on Wafer) method, or a WoW (Wafer on Wafer) method. Imaging device.
The imaging device according to claim 1, further comprising a gain adjusting unit that adjusts the gain of the analog pixel signal based on the brightness information detected by the brightness information detector.
The gain adjusting unit performs the gain adjustment in response to an external instruction, and if there is no external instruction, the gain adjusting unit performs the gain adjustment based on the brightness information detected by the brightness information detector. Item 13. The imaging device according to item 13.
An image pickup device that outputs the captured image data, and
A processor that performs predetermined signal processing on the image data is provided.
The image pickup device
A pixel array unit having a plurality of pixels that perform photoelectric conversion,
A converter that converts an analog pixel signal output from the pixel array unit into digital pixel data,
A signal processing unit that performs signal processing on the digital pixel data, and
A brightness information detector that detects brightness information incident on the pixel array unit based on the digital pixel data is provided.
The signal processing unit performs at least a part of the signal processing based on the brightness information detected by the brightness information detector.
An electronic device in which the image data processed by the signal processing unit is supplied to the processor.
The signal processing unit
A first signal processing unit that performs a first signal processing on the digital pixel data,
It has a second signal processing unit that performs a second signal processing on the digital pixel data or data that has performed at least a part of the first signal processing.
The second signal processing unit performs at least a part of the second signal processing based on the brightness information detected by the brightness information detector.
The electronic device according to claim 15, further comprising an information processing unit that performs at least one of a predetermined recognition process and a detection process based on the output data of the second signal processing unit.
The processor sends an instruction to the first signal processing unit to perform processing based on the output data of the first signal processing unit, and the first operation mode has lower power consumption than the first operation mode. It has a second operation mode in which one does not send an instruction to the first signal processing unit and does not receive the output data of the first signal processing unit.
The second signal processing unit or the information processing unit performs motion detection processing based on the brightness information detected by the brightness information detector regardless of the operation mode of the processor.
When the processor is in the second operation mode, the second signal processing unit or the information processing unit performs the motion detection process based on the brightness information detected by the brightness information detector, and the motion detection process is performed. The electronic device according to claim 16, wherein when motion is detected by the motion detection process, the processor is returned from the second operation mode to the first operation mode.
The step of performing photoelectric conversion in the pixel array section and outputting an analog pixel signal,
The step of converting the analog pixel signal into digital pixel data,
A step of detecting brightness information incident on the pixel array unit based on the digital pixel data, and
An imaging method comprising a step of performing signal processing on the digital pixel data and performing at least a part of the signal processing based on the brightness information.