WO2024014278A1 - 撮像装置およびデータ出力方法 - Google Patents

撮像装置およびデータ出力方法 Download PDF

Info

Publication number
WO2024014278A1
WO2024014278A1 PCT/JP2023/023702 JP2023023702W WO2024014278A1 WO 2024014278 A1 WO2024014278 A1 WO 2024014278A1 JP 2023023702 W JP2023023702 W JP 2023023702W WO 2024014278 A1 WO2024014278 A1 WO 2024014278A1
Authority
WO
WIPO (PCT)
Prior art keywords
signal processing
imaging device
output
image data
imaging
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2023/023702
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
彬任 桑原
岡部　政信
裕幸 小沢
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Sony Semiconductor Solutions Corp
Original Assignee
Sony Semiconductor Solutions Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Sony Semiconductor Solutions Corp filed Critical Sony Semiconductor Solutions Corp
Priority to JP2024533621A priority Critical patent/JPWO2024014278A1/ja
Priority to US18/880,967 priority patent/US20260017403A1/en
Priority to CN202380051951.9A priority patent/CN119631418A/zh
Publication of WO2024014278A1 publication Critical patent/WO2024014278A1/ja
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
    • G06F21/60—Protecting data
    • G06F21/62—Protecting access to data via a platform, e.g. using keys or access control rules
    • G06F21/6218—Protecting access to data via a platform, e.g. using keys or access control rules to a system of files or objects, e.g. local or distributed file system or database
    • G06F21/6245—Protecting personal data, e.g. for financial or medical purposes
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/52—Surveillance or monitoring of activities, e.g. for recognising suspicious objects
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules

Definitions

  • the present disclosure relates to an imaging device and a data output method, and particularly relates to an imaging device and a data output method that can reduce the risk of privacy violation.
  • Patent Document 1 discloses an image sensor in which an imaging unit that captures an image and a signal processing unit that performs signal processing on image data based on the output of the imaging unit are arranged on a single chip. There is.
  • the image sensor disclosed in Patent Document 1 is configured to be able to selectively output at least one of the signal processing result of signal processing, intermediate data obtained during signal processing, and image data to the outside.
  • the present disclosure has been made in view of this situation, and is intended to reduce the risk of privacy infringement.
  • An imaging device includes an imaging unit that captures an image, in which a plurality of pixels are arranged two-dimensionally, and a signal processing unit that performs signal processing on image data based on the output of the imaging unit. and an output section that is not directly connected to the imaging section and is capable of outputting only the signal processing result of the signal processing to the outside, and the signal processing section deletes the image data after the signal processing. It is an imaging device that
  • An imaging device includes an imaging unit that captures an image, in which a plurality of pixels are arranged two-dimensionally, and a signal processing unit that performs signal processing on image data based on the output of the imaging unit. an output unit capable of outputting the signal processing result of the signal processing to the outside; and a check process for checking that the image data is not included in the output of the signal processing unit before the signal processing.
  • An imaging device includes a check processing section.
  • a data output method includes an imaging unit that captures an image, in which a plurality of pixels are arranged two-dimensionally, and a signal that performs signal processing on image data based on the output of the imaging unit.
  • An imaging device that includes a processing unit and an output unit that is not directly connected to the imaging unit and that can output only the signal processing result of the signal processing to the outside deletes the image data after the signal processing. This is the output method.
  • an imaging unit that captures an image, in which a plurality of pixels are arranged two-dimensionally, and a signal processing unit that performs signal processing on image data based on the output of the imaging unit.
  • the image data is deleted after the signal processing, and an output section that is not directly connected to the imaging section and is capable of outputting only the signal processing result of the signal processing to the outside.
  • an imaging unit that captures an image, in which a plurality of pixels are arranged two-dimensionally, and a signal processing unit that performs signal processing on image data based on the output of the imaging unit.
  • a check process of checking that the image data is not included in the output of the signal processing section before the signal processing. is executed.
  • FIG. 1 is a block diagram showing a configuration example of a conventional imaging device.
  • FIG. 1 is a block diagram illustrating a configuration example of an imaging device to which the technology according to the present disclosure is applied.
  • FIG. 1 is a perspective view showing an outline of an example of the external configuration of an imaging device.
  • 1 is a block diagram showing a configuration example of an imaging device according to a first embodiment;
  • FIG. 3 is a diagram illustrating an example of restricting access to work memory. It is a flowchart explaining operation of DSP.
  • FIG. 2 is a block diagram showing a first configuration example of an imaging device according to a second embodiment.
  • 3 is a flowchart illustrating the operation of a network test block.
  • FIG. 3 is a diagram illustrating checking the network structure.
  • FIG. 3 is a flowchart illustrating the operation of the entire imaging device.
  • FIG. 2 is a block diagram showing a second configuration example of an imaging device according to a second embodiment.
  • 3 is a flowchart illustrating the operation of a check-dedicated processor.
  • FIG. 3 is a block diagram showing a third configuration example of an imaging device according to a second embodiment.
  • 3 is a flowchart illustrating the operation of a check-dedicated processor.
  • FIG. 2 is a diagram illustrating an application example of the technology according to the present disclosure.
  • FIG. 2 is a diagram illustrating an application example of the technology according to the present disclosure.
  • FIG. 1 is a block diagram showing an example of the configuration of a conventional imaging device.
  • the imaging device IS shown in FIG. 1 is, for example, a CMOS (Complementary Metal Oxide Semiconductor) image sensor configured with one chip, which receives incident light from an optical system (not shown), performs photoelectric conversion, and Outputs image data corresponding to the incident light from the system.
  • CMOS Complementary Metal Oxide Semiconductor
  • the imaging device IS also performs signal processing, such as recognition processing for recognizing a predetermined recognition target, using the output image data, and outputs the signal processing result of the signal processing.
  • signal processing such as recognition processing for recognizing a predetermined recognition target
  • the imaging device IS includes an imaging block 10, a signal processing block 20, a selector 30, and an external I/F 40.
  • the imaging block 10 and the signal processing block 20, and the signal processing block 20 and the selector 30 are electrically connected by connection lines. Further, the imaging block 10 and the selector 30 are directly electrically connected by a connecting line.
  • the imaging block 10 includes a pixel array 11, a column ADC (Analog to Digital Converter) 12, and a control unit 13.
  • the pixel array 11 is configured as an imaging unit in which a plurality of pixels are arranged two-dimensionally.
  • the pixel array 11 is driven by the control unit 13 and captures an image. Specifically, the pixel array 11 receives incident light from an optical system (not shown) in each pixel, performs photoelectric conversion, and outputs an analog image signal corresponding to the incident light.
  • the column ADC 12 reads analog image signals output from the pixel array 11 and performs AD conversion under the control of the control unit 13.
  • the column ADC 12 outputs a digital image signal obtained by AD conversion of an analog image signal as image data (RAW data).
  • the RAW data output by the column ADC 12 is supplied to the signal processing block 20 and also to the selector 30.
  • the control unit 13 controls the operations of the pixel array 11 and column ADC 12. Specifically, the control unit 13 controls driving of each pixel in the pixel array 11, reading of image signals by the column ADC 12, and AD conversion.
  • the signal processing block 20 includes an ISP (Image Signal Processing) section 21, a DSP (Digital Signal Processor) 22, a processing result I/F 23, and an external I/F 24.
  • ISP Image Signal Processing
  • DSP Digital Signal Processor
  • the units constituting the signal processing block 20 are connected to each other via a bus, and can exchange information as necessary.
  • the signal processing block 20 includes a CPU (Central Processing Unit) that performs various processes including overall control of the signal processing block 20 by executing programs stored in a memory (not shown). ) is provided.
  • CPU Central Processing Unit
  • the ISP unit 21 performs various image processing on the image data (RAW data) from the imaging block 10.
  • the ISP section performs HDR (High Dynamic Range) conversion processing, defect correction, development processing, etc. on the image data from the imaging block 10.
  • HDR High Dynamic Range
  • the DSP 22 functions as a signal processing unit that performs signal processing using image data after image processing by the ISP unit 21.
  • the processing result I/F 23 supplies the selector 30 with the signal processing results of the signal processing using the image data by the DSP 22 and the results of various image processing performed on the image data by the ISP section 21.
  • the external I/F 24 outputs the signal processing result of the signal processing performed by the DSP 22 using the image data to the outside.
  • the external I/F 24 is configured by, for example, a serial communication I/F such as SPI (Serial Peripheral Interface).
  • the selector 30 performs output control to selectively output the image data (RAW data) from the imaging block 10 and the signal processing result of the signal processing from the signal processing block 20 to the outside from the external I/F 40. That is, the selector 30 selects the image data from the imaging block 10, the signal processing result from the signal processing block 20, or both, and supplies them to the external I/F 40.
  • Intermediate data obtained during signal processing using image data by the DSP 22 may be supplied to the selector 30 via the processing result I/F 23.
  • the selector 30 transfers either the image data from the imaging block 10, the signal processing result from the signal processing block 20, both of them, or intermediate data obtained during signal processing from the external I/F 40. Performs output control to selectively output.
  • intermediate data obtained during signal processing for example, recognition processing
  • the intermediate data can be used for debugging a program for performing signal processing.
  • the external I/F 40 is an I/F that outputs image data and signal processing results supplied from the selector 30 to the outside.
  • a relatively high-speed parallel communication I/F such as MIPI (Mobile Industry Processor Interface) can be employed.
  • the image data from the imaging block 10 or the signal processing result by the DSP 22 from the signal processing block 20 is output to the outside according to the output control of the selector 30. Therefore, for example, if only the signal processing result from the signal processing block 20 by the DSP 22 is required externally, and the image data (captured image) itself is not required, only the signal processing result can be output, and the external The amount of data output from the I/F 40 to the outside can be reduced. Further, if the image data (captured image) itself is not required, only the signal processing result from the signal processing block 20 by the DSP 22 can be outputted to the outside from the external I/F 24 as well.
  • the imaging device IS can output only the signal processing results to the outside, so it can be said that there is an advantage in terms of privacy protection.
  • image data cannot be output, there is a risk of privacy violation.
  • FIG. 2 is a block diagram illustrating a configuration example of an imaging device to which the technology according to the present disclosure is applied.
  • the imaging device 1 shown in FIG. 2 is also, for example, a CMOS image sensor configured with one chip, which receives incident light from an optical system (not shown), performs photoelectric conversion, and converts the incident light from the optical system. Output the image data corresponding to.
  • CMOS image sensor configured with one chip, which receives incident light from an optical system (not shown), performs photoelectric conversion, and converts the incident light from the optical system. Output the image data corresponding to.
  • the imaging device 1 also performs signal processing, such as recognition processing for recognizing a predetermined recognition target, using the output image data, and outputs the signal processing result of the signal processing.
  • signal processing such as recognition processing for recognizing a predetermined recognition target
  • the imaging device 1 includes an imaging block 10 and a signal processing block 20.
  • the imaging device 1 in FIG. 2 differs from the imaging device IS in FIG. 1 in that it does not include the selector 30 and external I/F 40. That is, the imaging device 1 in FIG. 2 does not have an external I/F that is directly connected to the pixel array 11 (imaging block 10) as an imaging section.
  • the CPU 51 performs various processes including control of the entire signal processing block 20 by executing programs stored in a memory (not shown).
  • the external I/F 52 outputs the signal processing result of the signal processing performed on the image data by the DSP 22 to the outside.
  • the external I/F 52 is not limited to a serial communication I/F such as SPI, but may be configured by a parallel communication I/F such as MIPI or other communication I/F.
  • the external I/F 52 is configured as an output unit that is not directly connected to the pixel array 11 (imaging block 10) as an imaging unit and is capable of outputting only the signal processing result of signal processing to the outside.
  • the DSP 22 may be configured to delete image data after signal processing, or the CPU 51 may check that the output of the DSP 22 does not include image data before signal processing. It is possible to adopt a configuration that
  • the imaging device 1 can realize a configuration in which image data cannot be output to the outside.
  • FIG. 3 is a perspective view showing an outline of an example of the external configuration of the imaging device 1 of FIG. 2.
  • the imaging device 1 can be configured, for example, as a one-chip semiconductor device having a stacked structure in which a plurality of dies are stacked.
  • the imaging device 1 is configured by stacking two dies, a die 61 and a die 62.
  • the pixel array 11 is mounted on the upper die 61, and the column ADC 12, the control section 13, the ISP section 21, the DSP 22, the CPU 51, and the external I/F 52 are mounted on the lower die 62. There is.
  • the upper die 61 and the lower die 62 can be connected, for example, by forming a through hole that penetrates the die 61 and reaches the die 62, or by connecting the Cu wiring exposed on the lower surface side of the die 61 and the die 62.
  • the electrical connection is made by performing a Cu--Cu bond that is exposed on the upper surface side and directly connected to the Cu wiring.
  • the column ADC 12 as a method for AD converting the image signal output from the pixel array 11, for example, a column parallel AD method or an area AD method can be adopted.
  • ADCs are provided for the columns of pixels constituting the pixel array 11, and the ADCs in each column are responsible for AD conversion of the pixel signals of the pixels in that column. AD conversion of image signals of pixels in each column is performed in parallel.
  • a part of the column ADC 12 that performs AD conversion in the column-parallel AD method may be mounted on the upper die 61.
  • the pixels forming the pixel array 11 are divided into a plurality of blocks, and an ADC is provided for each block. Then, the ADC of each block takes charge of AD conversion of the pixel signals of the pixels of that block, so that AD conversion of the image signals of the pixels of the plurality of blocks is performed in parallel.
  • AD conversion reading and AD conversion
  • AD conversion of an image signal can be performed only for necessary pixels among the pixels forming the pixel array 11, using a block as the minimum unit.
  • the imaging device 1 can be configured with one die.
  • the two dies 61 and 62 are stacked to form the one-chip imaging device 1, but the one-chip imaging device 1 is constructed by stacking three or more dies. can be configured.
  • the signal processing performed by the imaging device 1 that is, the signal processing by the DSP 22, for example, recognition processing for recognizing a predetermined recognition target from image data can be adopted.
  • the imaging device 1 can receive, at the external I/F 52, the output of a distance sensor such as a ToF (Time of Flight) sensor that is arranged in a predetermined positional relationship with the imaging device 1.
  • a distance sensor such as a ToF (Time of Flight) sensor that is arranged in a predetermined positional relationship with the imaging device 1.
  • the signal processing of the DSP 22 includes, for example, processing that uses the captured image to remove noise in the distance image obtained from the output of the distance sensor received by the external I/F 52. It is possible to employ fusion processing that integrates images to obtain highly accurate distances.
  • the imaging device 1 can receive, through the external I/F 52, images output by other image sensors arranged in a predetermined positional relationship with the imaging device 1.
  • the signal processing of the DSP 22 includes, for example, self-position estimation processing (SLAM (Simultaneously Localization and Mapping)) that uses the image received by the external I/F 52 and the image captured by the imaging device 1 as a stereo image. Can be adopted.
  • SLAM Simultaneously Localization and Mapping
  • FIG. 4 is a block diagram showing an example of the configuration of the imaging device according to the first embodiment.
  • the imaging device 100 shown in FIG. 4 includes a pixel array 111, a column ADC 112, an ISP block 113, a DSP 114, a program memory 115, a work memory 116, a data transfer block 117, and an output I/F 118.
  • the pixel array 111, the column ADC 112, and the ISP block 113 have the same functions as the pixel array 11, the column ADC 12, and the ISP section 21 included in the imaging device 1 in FIG. 2, so their description will be omitted.
  • the DSP 114 executes recognition processing for recognizing a predetermined recognition target on the image data as signal processing using the image data after the image processing by the ISP block 113.
  • the program memory 115 stores a program for the DSP 114 to execute recognition processing.
  • the program memory 115 is constituted by a ROM (Read Only Memory), and the program for executing the recognition process cannot be rewritten.
  • the program is a program in which processing contents are described that enable execution of signal processing using a neural network, for example, recognition processing using a DNN.
  • the DSP 114 can execute recognition processing using DNN according to the program stored in the program memory 115.
  • the work memory 116 is composed of a RAM (Random Access Memory), and temporarily stores data related to recognition processing executed by the DSP 114. Specifically, in the work memory 116, the DSP 114 writes image data used in recognition processing and metadata of image data output as a recognition result of the recognition processing. Further, after the recognition process by the DSP 114, the image data written in the work memory 116 is deleted by the DSP 114.
  • RAM Random Access Memory
  • the data transfer block 117 is configured by, for example, a DMA (Direct Memory Access) controller, and controls the transfer of data from the work memory 116 to the output I/F 118. Specifically, the data transfer block 117 transfers only the metadata (recognition results) written in the work memory 116 to the output I/F 118.
  • DMA Direct Memory Access
  • the output I/F 118 has the same function as the external I/F 52 included in the imaging device 1 in FIG. 2, is not directly connected to the pixel array 11, and can output only metadata (recognition results) to the outside. Configured as an output section.
  • FIG. 5 is a diagram showing an example of access restriction to the work memory 116 of the data transfer block 117.
  • the data transfer block 117 is provided with an access restriction that allows access only to the storage area in which metadata (recognition results) are written among the storage areas of the work memory 116. You can.
  • the data transfer block 117 is provided with a read restriction such that data cannot be read from the storage area of the work memory 116 where image data is written. Good too.
  • DSP operation The operation of the DSP 114 will be described with reference to the flowchart in FIG. The process in FIG. 6 is repeated for each frame of the image captured by the pixel array 111.
  • step S111 the DSP 114 captures the image data processed by the ISP block 113.
  • step S112 the DSP 114 writes the captured image data into the work memory 116.
  • step S113 the DSP 114 executes DNN processing (recognition processing using DNN) on the image data written to the work memory 116.
  • DNN processing recognition processing using DNN
  • step S114 the DSP 114 deletes the image data written in the work memory 116.
  • step S115 the DSP 114 outputs metadata that is the recognition result of the recognition process to the work memory 116.
  • image data is once written to the work memory 116 for recognition processing, it is deleted after the recognition processing, and only metadata (recognition results) are output to the outside. As a result, image data will not be output to the outside, making it possible to reduce the risk of privacy violation.
  • the program for executing recognition processing is not rewritable, it is possible to reliably realize a structure in which image data is not output to the outside.
  • Second embodiment> a configuration will be described in which it is checked that image data is not included in the output of a DSP, which is a recognition processing unit.
  • FIG. 7 is a block diagram showing a first configuration example of an imaging device according to the second embodiment.
  • the imaging device 200 shown in FIG. 7 includes a pixel array 211, a column ADC 212, an ISP block 213, a DSP 214, a program memory 215, a work memory 216, and an output I/F 217.
  • These configurations basically have the same functions as the pixel array 111, column ADC 112, ISP block 113, DSP 114, program memory 115, work memory 116, and output I/F 118 included in the imaging device shown in FIG. , the explanation thereof will be omitted.
  • the DSP 214 does not write the metadata as the recognition result of the recognition process into the work memory 216, but directly outputs it to the output I/F 217.
  • the data transfer block 117 shown in FIG. 4 may be provided in the imaging device 200.
  • the DSP 214 may write metadata as a recognition result of the recognition process to the work memory 216, and the data transfer block 117 may transfer the metadata written to the work memory 216 to the output I/F 217. .
  • the program memory 215 is configured of a RAM instead of a ROM, and a program for executing the recognition process loaded from the outside is written in the program memory 215.
  • the imaging device 200 further includes an input I/F 218, a network inspection block 219, and an overall control CPU 220.
  • the input I/F 218 receives input of a program (hereinafter also referred to as a DSP program) for operating an AI model or the like in the DSP 214, provided by the user of the imaging device 200.
  • the DSP program is also a program in which processing contents are described that enable execution of signal processing using a neural network, for example, recognition processing using a DNN.
  • the network inspection block 219 is configured by a predetermined processor or realized by predetermined software, and executes a check process to check that image data is not included in the output of the DSP 214 before the recognition process by the DSP 214. It has the function of a processing section.
  • the network inspection block 219 loads a DSP program from the outside via the input I/F 218 and writes it into the program memory 215.
  • the network check block 219 executes the above-described checking process on the DSP program written in the program memory 215. More specifically, the network inspection block 219 checks that the output of the DSP 214 does not include image data by inspecting that the network structure of the neural network (DNN) is a structure that cannot output image data. Then, the test results are output to the overall control CPU 220.
  • DNN network structure of the neural network
  • the overall control CPU 220 performs various processes including overall control of the imaging device 200 by executing programs stored in a memory (not shown). For example, the overall control CPU 220 controls execution of recognition processing by the DSP 214 based on the test results from the network test block 219. The overall control CPU 220 also functions as an alert generation unit that generates alert information based on the test results from the network test block 219.
  • step S211 the network inspection block 219 loads the DSP program from the outside via the input I/F 218.
  • step S212 the network inspection block 219 writes the externally loaded DSP program into the program memory 215.
  • step S213 the network inspection block 219 checks (inspects) that the DSP program written in the program memory 215 has a network structure in which image data is not included in the output of the DNN.
  • the network inspection block 219 determines that there is a problem with the network structure of the DNN (NG).
  • the network inspection block 219 determines that there is no problem with the network structure of the DNN (OK).
  • the DNN network structure is converted to ROM, and no changes other than weights (parameters) can be made. You can do it like this.
  • the weights (parameters) may also be stored in a ROM so that only fixed processing is performed.
  • step S214 the network test block 219 outputs a test result indicating whether there is a problem with the DNN network structure to the overall control CPU 220.
  • step S221 the column ADC 212 reads RAW data from the pixel array 211 and outputs it to the ISP block 213.
  • the ISP block 213 performs various image processing on the RAW data from the column ADC 212.
  • step S222 the overall control CPU 220 obtains the test results from the network test block 219.
  • step S223 the overall control CPU 220 determines whether there is any problem with the DNN network structure based on the test results from the network test block 219. In other words, it is determined whether the user can obtain image data from the output of the DSP 214.
  • step S223 If it is determined in step S223 that there is no problem with the DNN network structure, that is, if it is determined that the user cannot obtain image data from the output of the DSP 214, the process advances to step S224.
  • step S224 the overall control CPU 220 transmits an enable signal (ON) to execute DNN processing (recognition processing using DNN) to the DSP 214. Thereby, the DSP 214 starts executing recognition processing on the image data after the image processing by the ISP block 213.
  • step S223 if it is determined in step S223 that there is a problem with the DNN network structure, that is, if it is determined that the user can obtain image data from the output of the DSP 214, the process advances to step S225.
  • step S225 the overall control CPU 220 transmits to the DSP 214 an enable signal (OFF) to stop execution of the DNN process.
  • the DSP 214 stops executing the recognition process on the image data after the image processing by the ISP block 213 .
  • step S226 the overall control CPU 220 generates alert information and supplies it to the output I/F 217, so that the output I/F 217 outputs an alert message.
  • FIG. 11 is a block diagram showing a second configuration example of the imaging device according to the second embodiment.
  • the same components as those included in the imaging device 200 in FIG. 7 are denoted by the same reference numerals, and the description thereof will be omitted.
  • the imaging device 200A in FIG. 11 is different from the imaging device 200 in FIG. 7 in that it includes a check-dedicated processor 231, a check-dedicated program memory 232, and a check-dedicated work memory 233 instead of the network inspection block 219 and the overall control CPU 220. different from.
  • the check-dedicated processor 231 is configured, for example, by a dedicated CPU, and executes a check process to check that the output of the DSP 214 does not include image data before the recognition process by the DSP 214. Note that the check-dedicated processor 231 may further have a function similar to that of the overall control CPU 220 in FIG. 7 in addition to the function of a check processing unit that executes a check process.
  • the check-only program memory 232 stores a program for the check-only processor 231 to execute check processing.
  • the check-only program memory 232 is constituted by a ROM, and the program for executing the check process cannot be rewritten.
  • the check-only work memory 233 is composed of a RAM, and a program (DSP program) loaded from the outside for executing recognition processing is written in the check-only work memory 233.
  • step S231 the check-only processor 231 loads a DSP program from the outside via the input I/F 218.
  • step S232 the check-only processor 231 writes the externally loaded DSP program into the check-only work memory 233.
  • step S233 the check-only processor 231 checks (inspects) that the DSP program written in the check-only work memory 233 has a network structure in which image data is not included in the output of the DNN.
  • step S234 the check-only processor 231 determines whether there is any problem with the DNN network structure.
  • step S234 If it is determined in step S234 that there is no problem with the DNN network structure, the process proceeds to step S235, where the check-only processor 231 transfers the DSP program written in the check-only work memory 233 to the program memory 215.
  • step S235 the check-only processor 231 transmits an enable signal (ON) to execute DNN processing (recognition processing using DNN) to the DSP 214.
  • the DSP 214 starts executing recognition processing on the image data after the image processing by the ISP block 213.
  • step S234 determines whether there is a problem with the DNN network structure. If it is determined in step S234 that there is a problem with the DNN network structure, the process advances to step S237, and the check-dedicated processor 231 transmits an enable signal (OFF) to stop the DNN processing execution to the DSP 214.
  • the DSP 214 stops executing the recognition process on the image data after the image processing by the ISP block 213 .
  • step S238 the check-only processor 231 generates alert information and supplies it to the output I/F 217, so that the output I/F 217 outputs an alert message.
  • FIG. 13 is a block diagram showing a third configuration example of the imaging device according to the second embodiment.
  • the same components as those included in the imaging device 200A in FIG. 11 are denoted by the same reference numerals, and the description thereof will be omitted.
  • the imaging device 200B in FIG. 13 differs from the imaging device 200A in FIG. 11 in that it does not include the check-only work memory 233.
  • step S251 the check-only processor 251 loads a DSP program from the outside via the input I/F 218.
  • step S252 the check-only processor 231 directly writes the externally loaded DSP program into the program memory 215.
  • step S253 the check-only processor 231 checks (inspects) that the DSP program written in the program memory 215 has a network structure in which image data is not included in the output of the DNN.
  • step S254 the check-only processor 231 determines whether there is any problem with the DNN network structure.
  • step S254 If it is determined in step S254 that there is no problem with the DNN network structure, the process proceeds to step S255, and the check-dedicated processor 231 sends an enable signal (ON) to the DSP 214 to execute DNN processing (recognition processing using DNN). Send. Thereby, the DSP 214 starts executing recognition processing on the image data after the image processing by the ISP block 213.
  • step S254 determines whether there is a problem with the DNN network structure. If it is determined in step S254 that there is a problem with the DNN network structure, the process proceeds to step S256, and the check-dedicated processor 231 transmits an enable signal (OFF) to stop the DNN processing execution to the DSP 214.
  • the DSP 214 stops executing the recognition process on the image data after the image processing by the ISP block 213 .
  • step S257 the check-only processor 231 generates alert information and supplies it to the output I/F 217, so that the output I/F 217 outputs an alert message.
  • the DSP program loaded from the outside is written directly to the program memory 215, so it is possible to check the network structure of the DNN without providing the check-only work memory 233. It becomes possible.
  • Each of the imaging devices of the second embodiment described above can also be configured in combination with the imaging device of the first embodiment. That is, it is possible to check that the output of the DSP 214 does not include image data, and then delete the image data after the recognition process, so that only the metadata (recognition results) are output to the outside.
  • the technology according to the present disclosure can be applied to a monitoring camera for watching over elderly people and people in need of care.
  • the technology according to the present disclosure can be applied to consumer behavior analysis at commercial facilities such as department store shopping centers.
  • the customer's "gender” is determined as a recognition result of the recognition process for image data IMG21 and IMG22. Metadata representing things like age, time spent at the store, and purchased products is output.
  • the present disclosure can take the following configuration.
  • an imaging unit that captures an image, in which a plurality of pixels are arranged two-dimensionally; a signal processing unit that performs signal processing on image data based on the output of the imaging unit; an output section that is not directly connected to the imaging section and is capable of outputting only the signal processing result of the signal processing to the outside;
  • the signal processing unit deletes the image data after the signal processing.
  • the imaging device (2) further comprising a first storage section into which data related to the signal processing is temporarily written, The imaging device according to (1), wherein the signal processing unit deletes the image data written to the first storage unit for the signal processing after the signal processing.
  • the chip includes a first substrate and a second substrate bonded to each other, The first substrate includes the imaging section, The imaging device according to (6), wherein the second substrate includes the signal processing section.
  • an imaging unit that captures an image, in which a plurality of pixels are arranged two-dimensionally; a signal processing unit that performs signal processing on image data based on the output of the imaging unit; an output unit capable of outputting the signal processing result of the signal processing to the outside;
  • An imaging device comprising: a check processing unit that executes a check process to check that the image data is not included in the output of the signal processing unit before the signal processing.
  • the imaging device (9) The imaging device according to (8), wherein the signal processing unit executes the signal processing when it is determined that the user cannot acquire the image data from the output of the signal processing unit.
  • a first program for executing the signal processing is loaded from the outside, The imaging device according to (9), wherein the check processing unit executes the check process on the first program.
  • the first program enables execution of the signal processing using a neural network, The imaging device according to (10), wherein the check processing unit checks whether the network structure of the neural network is a structure that cannot output the image data.
  • the check processing unit instructs the signal processing unit to execute the signal processing when the network structure is such that the image data cannot be restored from the network structure and the output result. Imaging device.
  • the imaging device further comprising a storage unit storing a second program for executing the check process, The imaging device according to any one of (8) to (12), wherein the storage unit makes the second program non-rewritable.
  • the signal processing unit stops execution of the signal processing when it is determined that the image data can be acquired by the user from the output of the signal processing unit, according to any one of (8) to (13). imaging device.
  • the imaging device further comprising an alert generation unit that generates alert information when it is determined that the image data is included in the output of the signal processing unit.
  • the imaging device according to any one of (8) to (15), wherein the output section is not directly connected to the imaging section.
  • the imaging device according to (8) to (16), wherein the imaging section, the signal processing section, and the check processing section are arranged in one chip.
  • the chip includes a first substrate and a second substrate bonded to each other, The first substrate includes the imaging section, The imaging device according to (17), wherein the second board includes the signal processing section and the check processing section.
  • an imaging unit that captures an image, in which a plurality of pixels are arranged two-dimensionally; a signal processing unit that performs signal processing on image data based on the output of the imaging unit;
  • the data output method according to (19), wherein, before the signal processing, a check process is performed to check that the image data is not included in the output of the signal processing section.
  • Imaging device 10 Imaging block, 11 Pixel array, 12 Column ADC, 13 Control unit, 20 Signal processing block, 21 ISP unit, 22 DSP, 51 CPU, 52 External I/F, 10 0 Imaging device, 111 Pixel array, 112 Column ADC, 113 ISP block, 114 DSP, 115 program memory, 116 work memory, 117 data transfer block, 118 output I/F, 200, 200A, 200B imaging device, 211 pixel array, 2 12 Column ADC, 213 ISP block, 214 DSP, 215 Program memory, 216 Work memory, 217 Output I/F, 218 Input I/F, 219 Network inspection block, 220 Overall control CPU, 231 Check-only processor, 232 Check-only program memory, 233 Check-only work memory

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Software Systems (AREA)
  • Bioethics (AREA)
  • Medical Informatics (AREA)
  • Databases & Information Systems (AREA)
  • Evolutionary Computation (AREA)
  • Computer Hardware Design (AREA)
  • General Engineering & Computer Science (AREA)
  • Computer Security & Cryptography (AREA)
  • Artificial Intelligence (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Computing Systems (AREA)
  • Signal Processing (AREA)
  • Studio Devices (AREA)
PCT/JP2023/023702 2022-07-11 2023-06-27 撮像装置およびデータ出力方法 Ceased WO2024014278A1 (ja)

Priority Applications (3)

Application Number Priority Date Filing Date Title
JP2024533621A JPWO2024014278A1 (https=) 2022-07-11 2023-06-27
US18/880,967 US20260017403A1 (en) 2022-07-11 2023-06-27 Imaging apparatus and data output method
CN202380051951.9A CN119631418A (zh) 2022-07-11 2023-06-27 成像装置和数据输出方法

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2022-111008 2022-07-11
JP2022111008 2022-07-11

Publications (1)

Publication Number Publication Date
WO2024014278A1 true WO2024014278A1 (ja) 2024-01-18

Family

ID=89536678

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2023/023702 Ceased WO2024014278A1 (ja) 2022-07-11 2023-06-27 撮像装置およびデータ出力方法

Country Status (4)

Country Link
US (1) US20260017403A1 (https=)
JP (1) JPWO2024014278A1 (https=)
CN (1) CN119631418A (https=)
WO (1) WO2024014278A1 (https=)

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2008152611A (ja) * 2006-12-19 2008-07-03 Seiko Epson Corp 画像認識装置、電子機器、画像認識方法及び画像認識プログラム
WO2016114392A1 (ja) * 2015-01-15 2016-07-21 日本電気株式会社 情報出力装置、カメラ、情報出力システム、情報出力方法及びプログラム
JP2016189135A (ja) * 2015-03-30 2016-11-04 パイオニア株式会社 認識装置、認識方法及び認識プログラム
WO2018051809A1 (ja) * 2016-09-16 2018-03-22 ソニーセミコンダクタソリューションズ株式会社 撮像装置、及び、電子機器
WO2021171295A1 (en) * 2020-02-25 2021-09-02 Ira Dvir Identity-concealing motion detection and portraying device
WO2022196214A1 (ja) * 2021-03-18 2022-09-22 矢崎総業株式会社 見守りシステム

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2008152611A (ja) * 2006-12-19 2008-07-03 Seiko Epson Corp 画像認識装置、電子機器、画像認識方法及び画像認識プログラム
WO2016114392A1 (ja) * 2015-01-15 2016-07-21 日本電気株式会社 情報出力装置、カメラ、情報出力システム、情報出力方法及びプログラム
JP2016189135A (ja) * 2015-03-30 2016-11-04 パイオニア株式会社 認識装置、認識方法及び認識プログラム
WO2018051809A1 (ja) * 2016-09-16 2018-03-22 ソニーセミコンダクタソリューションズ株式会社 撮像装置、及び、電子機器
WO2021171295A1 (en) * 2020-02-25 2021-09-02 Ira Dvir Identity-concealing motion detection and portraying device
WO2022196214A1 (ja) * 2021-03-18 2022-09-22 矢崎総業株式会社 見守りシステム

Also Published As

Publication number Publication date
US20260017403A1 (en) 2026-01-15
CN119631418A (zh) 2025-03-14
JPWO2024014278A1 (https=) 2024-01-18

Similar Documents

Publication Publication Date Title
US11315233B2 (en) Learning model generating device, and type identification system for generating learning model and using generated learning model to infer type of image defect
EP4064679A1 (en) Imaging device and electronic device
EP1788802A1 (en) Image pickup device, image pickup result processing method and integrated circuit
US20240251090A1 (en) Video compression apparatus, electronic apparatus, and video compression program
CN110338803B (zh) 物件监控方法及其运算装置
JP7250454B2 (ja) 撮像装置、撮像システム、移動体
US12536443B2 (en) Smart sensor
KR100920432B1 (ko) 화상 처리 장치 및 화상 처리 방법
EP3606052A1 (en) Train image monitoring system
JP2021136666A (ja) 撮像装置、デバイス、制御方法、およびプログラム
WO2024014278A1 (ja) 撮像装置およびデータ出力方法
JP4096513B2 (ja) 画像処理検査装置および検査方法
JP6072088B2 (ja) 放射線撮像システム、放射線撮像方法およびプログラム
JP6693764B2 (ja) 処理装置、分散処理システム及び分散処理方法
CN113711233A (zh) 用于汽车的感测设备和控制系统
US20240394905A1 (en) Image processing device and image processing method
WO2023179520A1 (zh) 成像方法及装置、图像传感器、成像设备和电子设备
US11729515B2 (en) Imaging apparatus, information processing method, and recording medium for determining whether an image analysis of a predetermined image is complete
JP2006318364A (ja) 画像処理装置
TW200806019A (en) Image sensors
CN113660455A (zh) 一种基于dvs数据的跌倒检测方法、系统、终端
CN116095520B (zh) 图像传感器、传感器架构、摄像模组及电子设备
JPH07141498A (ja) 医療用画像ファイル装置
US20200322627A1 (en) Video compression apparatus, electronic apparatus, and video compression program
CN118872282A (zh) 成像装置、成像方法和成像程序

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23839452

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 2024533621

Country of ref document: JP

WWE Wipo information: entry into national phase

Ref document number: 202380051951.9

Country of ref document: CN

Ref document number: 18880967

Country of ref document: US

NENP Non-entry into the national phase

Ref country code: DE

WWP Wipo information: published in national office

Ref document number: 202380051951.9

Country of ref document: CN

122 Ep: pct application non-entry in european phase

Ref document number: 23839452

Country of ref document: EP

Kind code of ref document: A1

WWP Wipo information: published in national office

Ref document number: 18880967

Country of ref document: US