WO2021070000A1 - 撮像システム及び監視システム - Google Patents
撮像システム及び監視システム Download PDFInfo
- Publication number
- WO2021070000A1 WO2021070000A1 PCT/IB2020/059072 IB2020059072W WO2021070000A1 WO 2021070000 A1 WO2021070000 A1 WO 2021070000A1 IB 2020059072 W IB2020059072 W IB 2020059072W WO 2021070000 A1 WO2021070000 A1 WO 2021070000A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- data
- image
- transistor
- circuit
- wiring
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Ceased
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N1/00—Scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission; Details thereof
- H04N1/46—Colour picture communication systems
- H04N1/465—Conversion of monochrome to colour
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N1/00—Scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission; Details thereof
- H04N1/46—Colour picture communication systems
- H04N1/56—Processing of colour picture signals
- H04N1/60—Colour correction or control
- H04N1/6083—Colour correction or control controlled by factors external to the apparatus
- H04N1/6086—Colour correction or control controlled by factors external to the apparatus by scene illuminant, i.e. conditions at the time of picture capture, e.g. flash, optical filter used, evening, cloud, daylight, artificial lighting, white point measurement, colour temperature
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T1/00—General purpose image data processing
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/10—Terrestrial scenes
- G06V20/17—Terrestrial scenes taken from planes or by drones
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/617—Upgrading or updating of programs or applications for camera control
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/80—Camera processing pipelines; Components thereof
- H04N23/84—Camera processing pipelines; Components thereof for processing colour signals
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
- H04N25/40—Extracting pixel data from image sensors by controlling scanning circuits, e.g. by modifying the number of pixels sampled or to be sampled
- H04N25/46—Extracting pixel data from image sensors by controlling scanning circuits, e.g. by modifying the number of pixels sampled or to be sampled by combining or binning pixels
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N9/00—Details of colour television systems
- H04N9/43—Conversion of monochrome picture signals to colour picture signals for colour picture display
Definitions
- One aspect of the present invention relates to a neural network and an imaging system using the neural network. Further, one aspect of the present invention relates to an electronic device using a neural network. Further, one aspect of the present invention relates to a vehicle using a neural network.
- the present invention relates to an imaging system that obtains a color image from a black-and-white image obtained by a solid-state image sensor by using an image processing technique.
- the present invention relates to a video surveillance system, a security system, or a safety information providing system using the imaging system.
- One aspect of the present invention is not limited to the above technical fields.
- One aspect of the invention disclosed in the present specification and the like relates to a product, a method, or a manufacturing method.
- One aspect of the invention relates to a process, machine, manufacture, or composition of matter. Therefore, more specifically, the technical fields of one aspect of the present invention disclosed in the present specification and the like include semiconductor devices, display devices, light emitting devices, power storage devices, storage devices, electronic devices, lighting devices, input devices, and input / output devices.
- the devices, their driving methods, or their manufacturing methods can be given as an example.
- the semiconductor device refers to all devices that can function by utilizing the semiconductor characteristics
- the electro-optical device, the semiconductor circuit, and the electronic device are all semiconductor devices.
- Image sensors are widely used as components for imaging such as digital cameras or video cameras.
- security equipment such as security cameras, such equipment can accurately image not only in bright places during the day but also at night or in dark places with little lighting and poor light. It needs to be done, and an image sensor with a wide dynamic range is required.
- AI Artificial Intelligence
- a method of learning with a large amount of image data, generating a model, and realizing colorization by inference using the obtained generated model is known.
- Machine learning is a part of AI.
- Patent Document 1 discloses an image pickup apparatus having an oxide semiconductor and using a transistor having an extremely low off-current in a pixel circuit.
- Patent Document 2 discloses a technique for adding a calculation function to an image pickup apparatus.
- Patent Document 3 discloses a technique relating to super-resolution processing.
- Japanese Unexamined Patent Publication No. 2011-119711 Japanese Unexamined Patent Publication No. 2016-123087 Japanese Unexamined Patent Publication No. 2010-262276
- a color filter is used for a color image obtained by an image pickup device such as a conventional image sensor.
- the image sensor is sold with a color filter attached, and the image sensor and a lens are appropriately combined and mounted in an electronic device.
- the color filter reduces the amount of light that reaches the light receiving area simply by being placed on top of the light receiving area of the image sensor. Therefore, it is inevitable that the amount of received light is attenuated by the color filter.
- Various color filters to be combined with the image sensor are being optimized by each company, but they do not pinpoint the wavelength of the light to pass through, but receive broad light in a certain wavelength range.
- One of the problems is to provide an imaging method and an imaging system that obtain an image having high visibility and faithful to an actual color without using a color filter.
- a security camera placed in a light environment with a narrow wavelength range needs to accurately grasp the situation reflected in the captured image as a clue leading to an incident or accident, so the characteristics of the object being projected are displayed in real time. It is necessary to grasp it accurately. Therefore, for example, in the case of a night-vision camera, it is important to focus on the object and capture an image with good visibility of the object even in a dark place.
- the night-vision camera there is also a camera that acquires a color image by color-separating with a special color filter using an infrared light source.
- the reflected light of infrared rays is used, there is a problem that the color is not reflected or expressed in a different color depending on the subject. This problem often occurs when the subject is a material that absorbs infrared rays. For example, human skin is more likely to be imaged whiter than it actually is, and warm colors such as yellow may be imaged blue.
- One of the problems is to provide an imaging method and an imaging system that have high visibility even at night or in a dark place and can obtain an image faithful to the same color as when there is external light.
- the image pickup system of the present invention includes a solid-state image pickup device that does not have a color filter, a storage device, and a learning device. Since no color filter is used, attenuation of light can be avoided, and highly sensitive imaging can be performed even with a small amount of light.
- the obtained black-and-white image data (analog data) is colorized, but is colored using an AI system.
- the AI system that is, a learning device that uses the teacher data stored in the storage device, uses the extracted features to make inferences and adjust the focus, and the color is highly visible even at night or in dark places.
- An image colorized image data (digital data)) can be obtained.
- the learning device includes at least a neural network unit, and can perform not only learning but also inference and output data.
- the learning device may perform inference using a feature amount that has been learned in advance. In that case, by storing the learned features in the storage device and performing the calculation, it is possible to output data at the same level as when the previously learned features are not used.
- the teacher data may be subjected to at least one super-resolution processing.
- the super-resolution processing refers to an image processing that generates a high-resolution image from a low-resolution image.
- An image surveillance system or a security system or a safety information providing system using the above-mentioned imaging system can clearly realize imaging in a relatively dark place.
- a surveillance system equipped with a security camera, which has a solid-state image sensor without a color filter, a learning device, and a storage device while the security camera detects a person.
- a software program is executed in which the solid-state image sensor takes an image and creates colorized image data by inference of the learning device using the teacher data of the storage device.
- the imaging system disclosed in the present specification it is possible to obtain a clear colorized image even when shooting in a dimly lit situation with a small amount of light. Therefore, it is relatively easy to identify a person (such as a face) or identify the characteristics of clothing based on the obtained colorized image.
- this imaging system By applying this imaging system to a security camera, it is possible to estimate a person's face as a color image and display it on a display device.
- FIG. 1 is a block diagram showing one aspect of the present invention.
- FIG. 2 is an example of source code showing one aspect of the present invention.
- FIG. 3 is an example of a source code showing one aspect of the present invention.
- FIG. 4 is an example of a flow chart showing one aspect of the present invention.
- FIG. 5 is a block diagram illustrating an imaging device.
- FIG. 6 is a diagram illustrating the pixel block 200 and the circuit 201.
- FIG. 7 is a diagram illustrating the pixel 100.
- 8A, 8B, and 8C are diagrams illustrating a filter.
- FIG. 9 is an example of a flow chart showing one aspect of the present invention.
- FIG. 10 is an example of source code showing one aspect of the present invention.
- 11A, 11B, and 11C are examples of applied products showing one aspect of the present invention.
- the data acquisition device 10 is a semiconductor chip including a solid-state image sensor 11 and an analog arithmetic circuit 12, and does not have a color filter.
- the data acquisition device 10 has an optical system such as a lens.
- the optical system may have any configuration as long as the imaging characteristics are known, and is not particularly limited.
- the A / D circuit 13 (also referred to as an A / D converter) indicates an analog-to-digital conversion circuit, which converts analog data output from the data acquisition device 10 into digital data. If necessary, an amplifier circuit may be provided between the data acquisition device 10 and the A / D circuit 13 to amplify the analog signal before converting it into digital data.
- the memory unit 14 is a circuit that stores digital data after conversion, and is configured to store data before being input to the neural network unit 16, but is not particularly limited to this configuration. Although it depends on the amount of data output from the data acquisition device or the data processing capacity of the image processing device, if it is small-scale data, the output from the A / D circuit 13 is used as it is in the neural network section without being stored in the memory section 14. It may be configured to input to 16. Further, the output from the A / D circuit 13 may be input to the neural network unit 16 at a remote location by using the Internet communication. For example, the neural network unit 16 may be constructed on a server capable of bidirectional communication.
- the image processing device 20 is a device for estimating the contour or color corresponding to the black-and-white image obtained by the data acquisition device 10.
- the image processing device 20 is divided into a first stage of learning and a second stage of estimation.
- the data acquisition device 10 and the image processing device 20 are configured as separate devices, but the data acquisition device 10 and the image processing device 20 can also be integrally configured. When configured integrally, it is also possible to update the feature amount obtained by the neural network unit in real time.
- the neural network unit 16 is realized by software calculation by a microprocessor.
- a microcontroller is a computer system incorporated into an integrated circuit (IC).
- the neural network unit 16 may be configured by combining a plurality of ICs.
- the learning device includes at least these a plurality of ICs.
- free software can be used if the microcontroller is equipped with Linux (registered trademark), the total cost for configuring the neural network unit 16 can be reduced, which is preferable.
- the present invention is not limited to Linux (registered trademark), and other OS (operating system) may be used.
- the teacher data for learning is stored in the storage unit 18 in advance.
- the teacher data for learning may use a learning set or the like, and the types of images include landscape photographs, portrait photographs, illustrations, and the like.
- the learning device makes inferences using the teacher data for learning.
- the learning device may have any configuration as long as it can perform inference using the teacher data of the storage unit 18 based on black and white analog data and output colorized image data.
- the learning device can output the colorized image data by performing the calculation using the feature amount data of the storage unit 18 based on the black and white analog data. Any configuration is acceptable.
- the feature amount learned in advance is used, the amount of data and the amount of calculation are reduced, so that there is an advantage that the learning device can be configured with a small-scale configuration, for example, one or two ICs.
- Keras is a library that provides convenient functions for deep learning. In particular, it is easy to read and write data to the intermediate layer of the neural network unit and change the weighting coefficient of the neural network unit. Also, in order to handle the Numpy library on Python, Numpy is read. Also, openCV is read for image editing.
- the source code corresponding to the above-mentioned processing is shown in order below the line of In [1] in FIG.
- a color image is used as the data. It is preferable to prepare thousands or tens of thousands of color images. When the number of files of the color image is enormous, a calculation load is applied in the reading process, so for example, the color image may be converted into the h5py format and then read.
- the following convolutional neural networks can be used for colorization. For example, in Python, using keras, it can be described as below the line of In [X] in FIG. 3 (however, X is an arbitrary number and can be changed according to the program).
- a neural network is also called U-net.
- the U-net has a function of complementing data from the convolution layer to the deconvolution layer, which is called a skip connection, can prevent the disappearance of the gradient, and can build a good learning model.
- the grayscaled image of the color image of the teacher data is input to the neural network shown in In [X] in FIG.
- the process of grayscale for example, openCV can be used.
- GaN Generic Adversarial Networks
- the output data of the neural network unit 16 and the time data of the time information acquisition device 17 are combined and stored in the large-scale storage device 15.
- the large-scale storage device 15 accumulates and stores the data obtained from the start of imaging.
- the display unit 19 may be provided with an operation input unit such as a touch panel, and the user selects from the data stored in the large-scale storage device 15 and observes the data as appropriate. be able to. Further, the display unit 19 may be able to access the large-scale storage device 15 by remote control via Internet communication, and the large-scale storage device 15 may be provided with a transmission antenna or a reception antenna.
- the imaging system 21 can be used in a video surveillance system or a security system.
- the display unit of the user's mobile information terminal can be the display unit 19.
- the large-scale storage device 15 By accessing the large-scale storage device 15 from the display unit of the mobile information terminal, it is possible to monitor the user regardless of the location of the user.
- the installation of the imaging system 21 is not limited to the wall of the room, and the entire or part of the configuration of the imaging system 21 is mounted on an unmanned aerial vehicle (also called a drone) equipped with rotary wings to monitor images from the air. You can also do it. In particular, it is possible to take images in an environment with a small amount of light, especially in the evening or at night when the streetlights are not lit.
- an unmanned aerial vehicle also called a drone
- the video surveillance system or the security system has been described, but the present invention is not particularly limited, and semi-automatic by combining with a camera or radar that images the surroundings of the vehicle and an ECU (Electronic Control Unit) that performs image processing and the like. It can also be applied to a vehicle capable of driving or a vehicle capable of fully automatic driving.
- a vehicle using an electric motor has a plurality of ECUs, and the ECUs control the engine and the like.
- the ECU includes a microcomputer.
- the ECU is connected to a CAN (Control Area Network) provided on the electric vehicle.
- CAN is one of the serial communication standards used as an in-vehicle LAN.
- the ECU uses a CPU or GPU.
- a solid-state image sensor without a color filter is used as one of a plurality of cameras (cameras for drive recorders, rear cameras, etc.) mounted on an electric vehicle, and the obtained black-and-white image is inferred by an ECU via a CAN. It may be configured so that a colorized image can be created and displayed on a display device in a vehicle or a display unit of a mobile information terminal.
- FIG. 4 shows an example of a flow for colorizing the black-and-white image obtained by the solid-state image sensor 11 using the block diagram and the program shown in the first embodiment.
- the imaging system 21 shown in the first embodiment is installed in a place to be monitored (room, parking lot, entrance, etc.), activated, and continuous shooting is started.
- Black-and-white image data is acquired using a solid-state image sensor without a color filter (S2).
- An array of a plurality of solid-state image sensors arranged in a matrix direction may be referred to as a pixel array.
- the obtained analog data is filtered using the product-sum calculation circuit (S3).
- Steps S2 and S3 are performed by the image pickup apparatus shown in FIG.
- the image pickup apparatus will be described below.
- FIG. 5 is a block diagram illustrating an imaging device.
- the imaging device includes a pixel array 300, a circuit 201, a circuit 301, a circuit 302, a circuit 303, a circuit 304, a circuit 305, and a circuit 306.
- Each of the circuit 201 and the circuits 301 to 306 is not limited to a single circuit configuration, and may be configured by a combination of a plurality of circuits. Alternatively, any one of the above circuits may be integrated. Further, a circuit other than the above may be connected.
- the pixel array 300 has an imaging function and a calculation function.
- the circuits 201 and 301 have an arithmetic function.
- the circuit 302 has an arithmetic function or a data conversion function.
- the circuits 303, 304 and 306 have a selection function.
- the circuit 303 is electrically connected to the pixel block 200 via the wiring 124.
- the circuit 304 is electrically connected to the pixel block 200 via the wiring 123.
- the circuit 305 has a function of supplying a potential for multiply-accumulate calculation to the pixels.
- a shift register, a decoder, or the like can be used for the circuit having the selection function.
- the circuit 306 is electrically connected to the pixel block 200 via the wiring 113.
- the circuits 301 and 302 may be provided externally.
- the pixel array 300 has a plurality of pixel blocks 200. As shown in FIG. 6, the pixel block 200 has a plurality of pixels 100 arranged in a matrix, and each pixel 100 is electrically connected to the circuit 201 via wiring 112. The circuit 201 can also be provided in the pixel block 200.
- the pixel 100 is electrically connected to the adjacent pixel 100 via a transistor 150 (transistor 150g to 150j).
- transistor 150g to 150j The function of the transistor 150 will be described later.
- the number of pixels of the pixel block 200 is set to 3 ⁇ 3 as an example, but the number of pixels is not limited to this. For example, it can be 2 ⁇ 2, 4 ⁇ 4, or the like. Alternatively, the number of pixels in the horizontal direction and the number of pixels in the vertical direction may be different. Further, some pixels may be shared by adjacent pixel blocks.
- the pixel block 200 and the circuit 201 can be operated as a product-sum calculation circuit.
- the pixel 100 can have a photoelectric conversion device 101, a transistor 102, a transistor 103, a transistor 104, a transistor 105, a transistor 106, and a capacitor 107.
- One electrode of the photoelectric conversion device 101 is electrically connected to one of the source and drain of the transistor 102.
- the other of the source or drain of the transistor 102 is electrically connected to one of the source or drain of the transistor 103, the gate of the transistor 104 and one electrode of the capacitor 107.
- One of the source or drain of the transistor 104 is electrically connected to one of the source or drain of the transistor 105.
- the other electrode of the capacitor 107 is electrically connected to one of the source or drain of the transistor 106.
- the other electrode of the photoelectric conversion device 101 is electrically connected to the wiring 114.
- the other of the source or drain of the transistor 103 is electrically connected to the wiring 115.
- the other of the source or drain of the transistor 105 is electrically connected to the wiring 112.
- the other of the source or drain of the transistor 104 is electrically connected to the GND wiring or the like.
- the other of the source or drain of the transistor 106 is electrically connected to the wiring 111.
- the other electrode of the capacitor 107 is electrically connected to the wiring 117.
- the gate of the transistor 102 is electrically connected to the wiring 121.
- the gate of the transistor 103 is electrically connected to the wiring 122.
- the gate of the transistor 105 is electrically connected to the wiring 123.
- the gate of the transistor 106 is electrically connected to the wiring 124.
- the node FD is an electrical connection point between the other of the source or drain of the transistor 102, one of the source or drain of the transistor 103, one electrode of the capacitor 107, and the gate of the transistor 104. Further, the electrical connection point between the other electrode of the capacitor 107 and one of the source or drain of the transistor 106 is designated as a node FDW.
- the wirings 114 and 115 can have a function as a power supply line.
- the wiring 114 can function as a high-potential power supply line
- the wiring 115 can function as a low-potential power supply line.
- the wirings 121, 122, 123, and 124 can function as signal lines for controlling the continuity of each transistor.
- the wiring 111 can function as a wiring that supplies a potential corresponding to a weighting coefficient to the pixel 100.
- the wiring 112 can function as a wiring that electrically connects the pixel 100 and the circuit 201.
- the wiring 117 can function as a wiring that electrically connects the other electrode of the capacitor 107 of the pixel and the other electrode of the capacitor 107 of another pixel via the transistor 150 (see FIG. 6).
- An amplifier circuit or a gain adjustment circuit may be electrically connected to the wiring 112.
- a photodiode can be used as the photoelectric conversion device 101. Regardless of the type of photodiode, a Si photodiode having silicon in the photoelectric conversion layer, an organic photodiode having an organic photoconductor in the photoelectric conversion layer, or the like can be used. If it is desired to increase the light detection sensitivity at low illuminance, it is preferable to use an avalanche photodiode.
- the transistor 102 can have a function of controlling the potential of the node FD.
- the transistor 103 can have a function of initializing the potential of the node FD.
- the transistor 104 can have a function of controlling the current flowing through the circuit 201 according to the potential of the node FD.
- the transistor 105 can have a function of selecting pixels.
- the transistor 106 can have a function of supplying a potential corresponding to a weighting coefficient to the node FDW.
- a high voltage may be applied, and it is preferable to use a high voltage transistor as the transistor connected to the photoelectric conversion device 101.
- a high voltage transistor for example, a transistor using a metal oxide in the channel forming region (hereinafter, OS transistor) or the like can be used. Specifically, it is preferable to apply an OS transistor to the transistor 102.
- the OS transistor also has a characteristic that the off-current is extremely low.
- the period during which the charge can be held by the node FD and the node FDW can be extremely extended. Therefore, it is possible to apply the global shutter method in which the charge accumulation operation is performed simultaneously in all the pixels without complicating the circuit configuration or the operation method. Further, while holding the image data in the node FD, it is possible to perform a plurality of operations using the image data.
- the transistor 104 has excellent amplification characteristics.
- the transistor 106 it may be preferable to use a transistor having high mobility capable of high-speed operation. Therefore, transistors using silicon in the channel forming region (hereinafter referred to as Si transistors) may be applied to the transistors 104 and 106.
- an OS transistor and a Si transistor may be arbitrarily combined and applied. Moreover, all the transistors may be OS transistors. Alternatively, all the transistors may be Si transistors. Examples of the Si transistor include a transistor having amorphous silicon, a transistor having crystalline silicon (microcrystalline silicon, low temperature polysilicon, single crystal silicon), and the like.
- the potential of the node FD in the pixel 100 is determined by the sum of the reset potential supplied from the wiring 115 and the potential (image data) generated by the photoelectric conversion by the photoelectric conversion device 101.
- the potential corresponding to the weighting coefficient supplied from the wiring 111 is capacitively coupled and determined. Therefore, a current corresponding to the data obtained by adding an arbitrary weighting coefficient to the image data can be passed through the transistor 105.
- the above is an example of the circuit configuration of the pixel 100, and the photoelectric conversion operation can be performed by another circuit configuration.
- each pixel 100 is electrically connected to each other by wiring 112.
- the circuit 201 can perform calculations using the sum of the currents flowing through the transistors 104 of each pixel 100.
- the circuit 201 has a capacitor 202, a transistor 203, a transistor 204, a transistor 205, a transistor 206, and a resistor 207.
- One electrode of the capacitor 202 is electrically connected to one of the source or drain of the transistor 203.
- One of the source and drain of transistor 203 is electrically connected to the gate of transistor 204.
- One of the source or drain of transistor 204 is electrically connected to one of the source or drain of transistor 205.
- One of the source or drain of transistor 205 is electrically connected to one of the source or drain of transistor 206.
- One electrode of the resistor 207 is electrically connected to the other electrode of the capacitor 202.
- the other electrode of the capacitor 202 is electrically connected to the wiring 112.
- the other of the source or drain of transistor 203 is electrically connected to wiring 218.
- the other of the source or drain of transistor 204 is electrically connected to wire 219.
- the other of the source or drain of the transistor 205 is electrically connected to a reference power line such as GND wiring.
- the other of the source or drain of the transistor 206 is electrically connected to the wiring 212.
- the other electrode of resistor 207 is electrically connected to wiring 217.
- Wiring 217, 218, 219 can have a function as a power supply line.
- the wiring 218 can have a function as a wiring for supplying a dedicated potential for reading.
- Wiring 217 and 219 can function as high potential power lines.
- the wirings 213, 215, and 216 can function as signal lines for controlling the continuity of each transistor.
- the wiring 212 is an output line and can be electrically connected to, for example, the circuit 301 shown in FIG.
- the transistor 203 can have a function of resetting the potential of the wiring 211 to the potential of the wiring 218.
- the wiring 211 is a wiring connected to one electrode of the capacitor 202, one of the source or drain of the transistor 203, and the gate of the transistor 204.
- Transistors 204 and 205 can have a function as a source follower circuit.
- the transistor 206 can have a function of controlling reading.
- the circuit 201 has a function as a correlated double sampling circuit (CDS circuit), and can be replaced with a circuit having another configuration having the function.
- CDS circuit correlated double sampling circuit
- an offset component other than the product of the image data (X) and the weighting coefficient (W) is removed, and the target WX is extracted.
- WX can be calculated by using the data with and without imaging for the same pixel and the data when weighting is applied to each of them.
- the total current (I p ) flowing through the pixel 100 when imaging is k ⁇ (X-V th ) 2
- the total current (I p ) flowing through the pixel 100 when weighted is k ⁇ (W + X-V th).
- the total current (I ref ) flowing through the pixel 100 without imaging is k ⁇ (0-V th ) 2
- the total current (I ref ) flowing through the pixel 100 when weighted is k ⁇ (W-).
- V th ) 2 is a constant and Vth is the threshold voltage of the transistor 105.
- the difference (data A) between the data with imaging and the data obtained by weighting the data is calculated.
- k ⁇ ((X-V th ) 2- (W + X-V th ) 2 ) k ⁇ (-W 2 -2W ⁇ X + 2W ⁇ V th ).
- data A and data B can be read out.
- the difference calculation between the data A and the data B can be performed by, for example, the circuit 301.
- the weight supplied to the entire pixel block 200 functions as a filter.
- a filter for example, a convolutional filter of a convolutional neural network (CNN) can be used.
- CNN convolutional neural network
- an image processing filter such as an edge extraction filter can be used.
- the edge extraction filter include the Laplacian filter shown in FIG. 8A, the prewit filter shown in FIG. 8B, and the Sobel filter shown in FIG. 8C.
- the elements of the edge extraction filter can be allocated to each pixel 100 as weights and supplied.
- the data A and the data B it is possible to calculate by using the data with and without imaging and the data when weights are added to each of them.
- the data with and without imaging is data without weighting, and can be rephrased as data with weight 0 added to all pixels 100.
- This operation corresponds to conducting the transistors 150 (transistors 150a to 150f) provided between the pixels 100 (see FIG. 6).
- transistor 150 transistors 150a to 150f
- all the node FDWs of each pixel 100 are short-circuited via the wiring 117.
- the weight ( ⁇ W) When the weight ( ⁇ W) is rewritten by supplying an electric charge from a circuit outside the pixel array 300, it takes time to complete the rewriting due to the capacity of the wiring 111 having a long distance and the like.
- the pixel block 200 is a minute area, the distance of the wiring 117 is short, and the capacity is small. Therefore, in the operation of redistributing the electric charge accumulated in the node FDW in the pixel block 200, the weight ( ⁇ W) can be rewritten at high speed.
- the pixel block 200 shown in FIG. 6 shows a configuration in which transistors 150a to 150f are electrically connected to different gate wires (wiring 113a to 113f). In this configuration, the conduction of the transistors 150a to 150f can be controlled independently, and the operation of acquiring ⁇ W / N can be selectively performed. Further, FIG. 6 shows a configuration in which the transistors 150g to 150j are electrically connected to different gate wires (113g to 113i).
- FIGS. 6 and 8 show an example in which nine transistors 150 (transistors 150a to 150f) are provided between the pixels 100, the number of transistors 150 may be further increased. Further, in the transistors 150g to 150j, some transistors may be omitted so as to eliminate parallel paths.
- the product-sum calculation result data output from the circuit 201 is sequentially input to the circuit 301.
- the circuit 301 may have various calculation functions in addition to the above-mentioned function of calculating the difference between the data A and the data B.
- the circuit 301 can have the same configuration as the circuit 201.
- the function of the circuit 301 may be replaced by software processing.
- the circuit 301 may have a circuit for calculating the activation function.
- a comparator circuit can be used for the circuit.
- the comparator circuit outputs the result of comparing the input data with the set threshold value as binary data. That is, the pixel block 200 and the circuit 301 can act as a part of the neural network.
- the data output from the circuit 301 is sequentially input to the circuit 302.
- the circuit 302 can be configured to include, for example, a latch circuit and a shift register. With this configuration, parallel serial conversion can be performed, and the data input in parallel can be output to the wiring 311 as serial data.
- connection destination of the wiring 311 is not limited. For example, it can be connected to the A / D circuit 13 or the neural network unit 16 shown in FIG. Further, the connection destination of the wiring 311 may be an FPGA (field-programmable gate array).
- FPGA field-programmable gate array
- the A / D circuit 13 converts the filtered analog data into digital data (S4).
- the converted digital data is stored in the memory unit 14 (digital memory unit) (S5).
- the digital data is converted into a signal format (JPEG (registered trademark) or the like) required by the inference program in the subsequent stage (S6).
- JPEG registered trademark
- the converted digital data is convolved using a CPU or the like to infer contours, colors, and the like, and colorize the data (S7).
- a CPU Central Processing Unit
- a PMU Power Management Unit
- the colorized image data is output (S8).
- the colorized image data is saved together with the time data such as the date and time (S9).
- the storage is stored in a large-scale storage device 15, a so-called large-capacity storage device (such as a hard disk) or a database.
- the acquisition of the colorized image data is repeated (during operation). Colorization can be performed in real time by repeating the process.
- each of the above-mentioned steps (S4 to S9) can be realized by one or a plurality of computers.
- latent variables are monitored by cos similarity, and the focus is adjusted by the optical system so that fluctuations are reduced, so that even if the object moves during shooting, It is possible to adjust the object so that it is in focus.
- inference may be performed using the extracted features.
- Focus adjustment may be performed so that the inference result has less variation. For example, when a person is inferred, the focus may be adjusted so that the likelihood is constant or large.
- FIG. 9 shows a flow chart.
- the same reference numerals are used for the same steps as the flow chart shown in FIG. 4 of the second embodiment. Since S1 to S6 in FIG. 4 and S8 to S9 are the same, detailed description thereof will be omitted here.
- step S6 the super-resolution processing is performed a plurality of times on the converted digital data using the first learning model, and the contour is inferred (S7a).
- the digital data after the super-resolution processing is inferred in color and the like using the second learning model and is colored (S7b). Subsequent steps are the same as in the second embodiment.
- the teacher data of the second learning model is subjected to super-resolution processing a plurality of times in advance, an animation image is mixed with a photographic image, or an outline is emphasized by an OPENCV drawcontours function or the like.
- the ratio of mixing the animation image with the photographic image is 1 when the photographic image is 2.
- An animated image is a kind of illustration, but contains a lot of edge components or color components. In the process of colorizing a black-and-white image, edges are extracted as features in the convolutional layer, and the color of each region of the image is inferred based on the features. Therefore, an image containing many edge components is used as teacher data. Is effective in improving the efficiency of machine learning.
- the animation image As the teacher data, the number of teacher data data for obtaining a color image that can reach a certain standard can be reduced, the time required for machine learning can be shortened, and the configuration of the neural network unit can be simplified. ..
- the neural network unit is a part of machine learning. Further, deep learning is a part of the neural network unit.
- the learning of the neural network unit of this embodiment is shown below.
- Keras is a library that provides convenient functions for deep learning. In particular, it is easy to read and write data to the intermediate layer of the neural network and change the weighting coefficient of the neural network. Also, in order to handle the Numpy library on Python, Numpy is read. In addition, scipy is used as image processing.
- the source code corresponding to the above-mentioned processing is shown in order under the line of In [Y1] in FIG. 10 (however, Y1 is an arbitrary number and can be changed according to the program).
- the teacher data is an image with high resolution.
- Supervised data is a set of data or classified data used in supervised learning.
- As the image a color image or a black-and-white image can be used. It is preferable to prepare thousands or tens of thousands of images. When the number of files of the image is enormous, a calculation load is applied in the reading process, so that the image may be converted to the h5py format and then read.
- the super-resolution processing consists of a three-layer convolutional neural network.
- Python using Keras's sexual model, under the line of In [Y2] in FIG. 10 (however, Y2 is an arbitrary number and can be changed according to the program if it is later than Y1). It can be described as.
- [Y2] an example of inputting an image of 33 ⁇ 33 pixels is shown.
- an example of an image of 33 ⁇ 33 pixels is shown, but the image is not particularly limited and may be a 2K size (1920 ⁇ 1080 pixels) image or a 4K size (3840 ⁇ 2160 pixels) image. ..
- the size is the resolution, and the neural network may be designed according to the input image data or the output image data.
- the input image data and the output image data may be different.
- the output image may be in the 2K size.
- 8K size input image data is obtained by a solid-state image sensor, the amount of light that can be obtained by each solid-state image sensor is small, and it is particularly preferable not to use a color filter because a large amount of light can be obtained.
- an image having a reduced resolution of the image of the teacher data is input to the neural network shown in In [Y3].
- the process of lowering the resolution is performed by using, for example, scipy, in the line of In [Y3] in FIG. 10 (however, Y3 is an arbitrary number and can be changed according to the program if it is later than Y2). It can be described as follows. Here, the process of increasing the resolution by 1/3 is shown.
- each of the above-mentioned steps (S4 to S9) can be realized by one or a plurality of computers.
- GaN in which the above neural network is used as a generator may be used.
- the colorized image data obtained in the present embodiment has a smoother outline than the image data in the second embodiment, and optimum colorization is performed.
- a display device a personal computer, an image storage device or an image reproduction device including a recording medium, a mobile phone, a portable device
- Game machines including molds, mobile data terminals, electronic book terminals, video cameras, cameras such as digital still cameras, goggle type displays (head mount displays), navigation systems, sound playback devices (car audio, digital audio players, etc.), copying Examples include machines, facsimiles, printers, multifunction printers, automatic cash deposit / payment machines (ATMs), and vending machines. Specific examples of these electronic devices are shown in FIG.
- FIG. 11A is a surveillance camera, which includes a housing 951, a lens 952, a support portion 953, and the like.
- an imaging system according to one aspect of the present invention can be provided.
- a neural network unit is provided in the housing 951.
- the surveillance camera is an idiomatic name and does not limit its use.
- a device having a function as a surveillance camera is also called a camera or a video camera.
- Surveillance cameras use image sensors that do not use color filters.
- the colorized image data can be created by incorporating the program shown in the second embodiment or the third embodiment as a software program and executing the program in the neural network unit.
- the surveillance camera of the present embodiment when used for at least one of them, a color image can be acquired in a dim environment, which is difficult to acquire with the conventional surveillance camera. Therefore, the conventional surveillance camera The monitoring system can be strengthened by combining with.
- FIG. 11B is also a surveillance camera, which has a support base 954, a camera unit 955, a protective cover 956, and the like.
- the camera unit 955 is provided with a rotation mechanism or the like, and by installing it on the ceiling, it is possible to take an image of the entire surroundings.
- a camera unit 955 can be used as an imaging device included in the monitoring system of one aspect of the present invention.
- the neural network unit of the camera unit 955 estimates based on the data obtained by the camera unit 955, so that the suspicious person can be identified from the information captured by the colorization or the super-resolution.
- FIG. 11C shows an example of an air vehicle.
- the flying object 6500 shown in FIG. 11C has a propeller 6501, a camera 6502, a battery 6503, and the like, and has a function of autonomously flying.
- the image data taken by the camera 6502 is stored in the electronic component 6504.
- the electronic component 6504 can analyze the image data and detect the presence or absence of an obstacle when moving.
- a plurality of types of image pickup devices may be used.
- a camera 6502 can be used as an imaging device included in the monitoring system of one aspect of the present invention.
- the neural network unit estimates based on the data obtained by the camera 6502, so that the suspicious person can be identified from the information captured by colorization or super-resolution.
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- Software Systems (AREA)
- Evolutionary Computation (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Medical Informatics (AREA)
- Databases & Information Systems (AREA)
- Computing Systems (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Artificial Intelligence (AREA)
- Remote Sensing (AREA)
- Image Processing (AREA)
- Studio Devices (AREA)
Priority Applications (3)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| JP2021550721A JPWO2021070000A1 (cg-RX-API-DMAC7.html) | 2019-10-11 | 2020-09-29 | |
| US17/764,622 US11863728B2 (en) | 2019-10-11 | 2020-09-29 | Imaging system and monitoring system with training data that includes a color picture image and a color animation image |
| JP2025027912A JP7862129B2 (ja) | 2019-10-11 | 2025-02-25 | 撮像方法 |
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| JP2019187705 | 2019-10-11 | ||
| JP2019-187705 | 2019-10-11 |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| WO2021070000A1 true WO2021070000A1 (ja) | 2021-04-15 |
Family
ID=75437092
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| PCT/IB2020/059072 Ceased WO2021070000A1 (ja) | 2019-10-11 | 2020-09-29 | 撮像システム及び監視システム |
Country Status (3)
| Country | Link |
|---|---|
| US (1) | US11863728B2 (cg-RX-API-DMAC7.html) |
| JP (2) | JPWO2021070000A1 (cg-RX-API-DMAC7.html) |
| WO (1) | WO2021070000A1 (cg-RX-API-DMAC7.html) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US11924589B2 (en) * | 2019-12-27 | 2024-03-05 | Semiconductor Energy Laboratory Co., Ltd. | Imaging system |
| US12426435B2 (en) | 2020-02-14 | 2025-09-23 | Semiconductor Energy Laboratory Co., Ltd. | Imaging device |
| US12108172B2 (en) * | 2020-02-14 | 2024-10-01 | Sony Group Corporation | Vehicle control system using imaging device capable of object detection |
| US20220301127A1 (en) * | 2021-03-18 | 2022-09-22 | Applied Materials, Inc. | Image processing pipeline for optimizing images in machine learning and other applications |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2007208481A (ja) * | 2006-01-31 | 2007-08-16 | Sogo Keibi Hosho Co Ltd | 画像処理装置、監視センタ、監視システム、画像処理方法、および画像処理プログラム |
| JP2019117559A (ja) * | 2017-12-27 | 2019-07-18 | 日本放送協会 | カラー情報推定モデル生成装置、動画像カラー化装置およびそれらのプログラム |
| JP2019128889A (ja) * | 2018-01-26 | 2019-08-01 | 日本放送協会 | 画像情報変換器およびそのプログラム |
Family Cites Families (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPH07282227A (ja) * | 1994-04-08 | 1995-10-27 | Toshiba Corp | 人物顔領域検出装置 |
| JPH1083487A (ja) * | 1996-09-06 | 1998-03-31 | Hitachi Building Syst Co Ltd | 侵入者検知装置 |
| JP2006178516A (ja) * | 2004-12-20 | 2006-07-06 | Mitsubishi Electric Corp | 監視記録装置 |
| JP5448981B2 (ja) | 2009-04-08 | 2014-03-19 | 株式会社半導体エネルギー研究所 | 液晶表示装置の駆動方法 |
| WO2011055626A1 (en) | 2009-11-06 | 2011-05-12 | Semiconductor Energy Laboratory Co., Ltd. | Semiconductor device |
| US9773832B2 (en) | 2014-12-10 | 2017-09-26 | Semiconductor Energy Laboratory Co., Ltd. | Semiconductor device and electronic device |
| KR102322723B1 (ko) | 2016-08-03 | 2021-11-04 | 가부시키가이샤 한도오따이 에네루기 켄큐쇼 | 촬상 장치, 촬상 모듈, 전자 기기, 및 촬상 시스템 |
| WO2018203169A1 (ja) | 2017-05-02 | 2018-11-08 | 株式会社半導体エネルギー研究所 | 撮像装置、及び電子機器 |
| JP7202297B2 (ja) | 2017-07-14 | 2023-01-11 | 株式会社半導体エネルギー研究所 | 撮像装置および電子機器 |
| KR102688688B1 (ko) * | 2019-10-10 | 2024-07-25 | 엘지전자 주식회사 | 영상을 압축 또는 복원하기 위한 방법 및 장치 |
-
2020
- 2020-09-29 JP JP2021550721A patent/JPWO2021070000A1/ja not_active Withdrawn
- 2020-09-29 US US17/764,622 patent/US11863728B2/en active Active
- 2020-09-29 WO PCT/IB2020/059072 patent/WO2021070000A1/ja not_active Ceased
-
2025
- 2025-02-25 JP JP2025027912A patent/JP7862129B2/ja active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2007208481A (ja) * | 2006-01-31 | 2007-08-16 | Sogo Keibi Hosho Co Ltd | 画像処理装置、監視センタ、監視システム、画像処理方法、および画像処理プログラム |
| JP2019117559A (ja) * | 2017-12-27 | 2019-07-18 | 日本放送協会 | カラー情報推定モデル生成装置、動画像カラー化装置およびそれらのプログラム |
| JP2019128889A (ja) * | 2018-01-26 | 2019-08-01 | 日本放送協会 | 画像情報変換器およびそのプログラム |
Non-Patent Citations (2)
| Title |
|---|
| KOBAYASHI YOHEI, SHIRAYAMA SUSUMU: "A method for transferring color to grayscale images.", JOURNAL OF THE INSTITUTE OF IMAGE INFORMATION AND TELEVISION ENGINEERS, vol. 59, no. 5, 1 May 2005 (2005-05-01), pages 769 - 775, XP055817892 * |
| OYA,TAKAFUMI, MIYATA, TAKAMICHI: "Basic study of automatic colorization of luminance images by learning", PCSJ PICTURE CODING SYMPOSIUM OF JAPAN 2015, 18 November 2015 (2015-11-18), pages 1 - 2 * |
Also Published As
| Publication number | Publication date |
|---|---|
| JP2025084825A (ja) | 2025-06-03 |
| JPWO2021070000A1 (cg-RX-API-DMAC7.html) | 2021-04-15 |
| US11863728B2 (en) | 2024-01-02 |
| US20220417390A1 (en) | 2022-12-29 |
| JP7862129B2 (ja) | 2026-05-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP7862129B2 (ja) | 撮像方法 | |
| US12125144B2 (en) | Image modification techniques | |
| US11070717B2 (en) | Context-aware image filtering | |
| CN108419023B (zh) | 一种生成高动态范围图像的方法以及相关设备 | |
| US12112458B2 (en) | Removal of objects from images | |
| US20250166311A1 (en) | Virtual Walkthrough Experience Generation Based on Neural Radiance Field Model Renderings | |
| CN105957110B (zh) | 用于检测对象的设备和方法 | |
| CN110543850A (zh) | 目标检测方法及装置、神经网络训练方法及装置 | |
| CN104796600B (zh) | 图像合成装置和图像合成方法 | |
| CN102957864A (zh) | 影像设备及控制方法 | |
| JP2010098358A (ja) | 撮像素子および撮像装置 | |
| CN114338958A (zh) | 一种图像处理的方法及相关设备 | |
| KR20240115228A (ko) | 정보 처리 장치, 정보 처리 방법, 프로그램 | |
| US20240054659A1 (en) | Object detection in dynamic lighting conditions | |
| US20080094481A1 (en) | Intelligent Multiple Exposure | |
| US20250287122A1 (en) | Image sensor | |
| CN117994542A (zh) | 异物检测方法、装置以及系统 | |
| US11871107B2 (en) | Automatic camera selection | |
| CN120345007A (zh) | 用于检测图像中被遮挡的对象的多任务对象检测系统 | |
| JP7596305B2 (ja) | 撮像システム | |
| WO2025150483A1 (en) | Information processing apparatus, information processing method, program, and recording medium | |
| WO2022201973A1 (ja) | 情報処理システム及び学習モデルの生成方法 | |
| US20260030881A1 (en) | Information processing device, information processing method, and information processing system | |
| WO2025057684A1 (en) | Image processing system and information processing system | |
| CN119277193B (zh) | 拍摄方法、电子设备及存储介质 |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| 121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 20873893 Country of ref document: EP Kind code of ref document: A1 |
|
| WWE | Wipo information: entry into national phase |
Ref document number: 2021550721 Country of ref document: JP |
|
| NENP | Non-entry into the national phase |
Ref country code: DE |
|
| 122 | Ep: pct application non-entry in european phase |
Ref document number: 20873893 Country of ref document: EP Kind code of ref document: A1 |