WO2022070947A1 - 信号処理装置、撮像装置、信号処理方法 - Google Patents

信号処理装置、撮像装置、信号処理方法 Download PDF

Info

Publication number
WO2022070947A1
WO2022070947A1 PCT/JP2021/034103 JP2021034103W WO2022070947A1 WO 2022070947 A1 WO2022070947 A1 WO 2022070947A1 JP 2021034103 W JP2021034103 W JP 2021034103W WO 2022070947 A1 WO2022070947 A1 WO 2022070947A1
Authority
WO
WIPO (PCT)
Prior art keywords
input data
product
processing unit
data
signal processing
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2021/034103
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
克彦 半澤
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Sony Semiconductor Solutions Corp
Original Assignee
Sony Semiconductor Solutions Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Sony Semiconductor Solutions Corp filed Critical Sony Semiconductor Solutions Corp
Priority to JP2022553812A priority Critical patent/JPWO2022070947A1/ja
Priority to CN202180065035.1A priority patent/CN116210228A/zh
Priority to DE112021005190.3T priority patent/DE112021005190T5/de
Priority to US18/042,395 priority patent/US20230333816A1/en
Publication of WO2022070947A1 publication Critical patent/WO2022070947A1/ja
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F7/00—Methods or arrangements for processing data by operating upon the order or content of the data handled
    • G06F7/38—Methods or arrangements for performing computations using exclusively denominational number representation, e.g. using binary, ternary, decimal representation
    • G06F7/48—Methods or arrangements for performing computations using exclusively denominational number representation, e.g. using binary, ternary, decimal representation using non-contact-making devices, e.g. tube, solid state device; using unspecified devices
    • G06F7/544—Methods or arrangements for performing computations using exclusively denominational number representation, e.g. using binary, ternary, decimal representation using non-contact-making devices, e.g. tube, solid state device; using unspecified devices for evaluating functions by calculation
    • G06F7/5443—Sum of products
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H04N25/70—SSIS architectures; Circuits associated therewith
    • H04N25/76—Addressed sensors, e.g. MOS or CMOS sensors
    • H04N25/78—Readout circuits for addressed sensors, e.g. output amplifiers or A/D converters
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F17/00—Digital computing or data processing equipment or methods, specially adapted for specific functions
    • G06F17/10—Complex mathematical operations
    • G06F17/16—Matrix or vector computation, e.g. matrix-matrix or matrix-vector multiplication, matrix factorization
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0464—Convolutional networks [CNN, ConvNet]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/06—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons
    • G06N3/063—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons using electronic means
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/06—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons
    • G06N3/067—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons using optical means
    • G06N3/0675—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons using optical means using electro-optical, acousto-optical or opto-electronic means
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/80—Camera processing pipelines; Components thereof
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H04N25/70—SSIS architectures; Circuits associated therewith
    • H04N25/76—Addressed sensors, e.g. MOS or CMOS sensors
    • H04N25/77—Pixel circuitry, e.g. memories, A/D converters, pixel amplifiers, shared circuits or shared components

Definitions

  • This technology relates to a signal processing device that performs product-sum calculation, an image pickup device, and a signal processing method.
  • processing related to DNN such as image recognition processing for a subject may be performed.
  • DNN Deep Neural Network
  • many product-sum operations are required.
  • the product-sum calculation two types of input data such as image data and weight data are used.
  • the two types of input data may contain many zero values, and in that case, there is a problem that unnecessary operations are performed and the memory cannot be effectively used.
  • Patent Document 1 discloses a technique for generating an index including one or more memory address positions having input data (input activation value) which is a non-zero value. It is described that the input data can be compressed by storing only the input data having a non-zero value in the memory, and that the calculation efficiency is improved.
  • This technology was made in view of the above circumstances, and aims to improve the calculation efficiency of the product-sum calculation process.
  • the signal processing device has a product-sum calculator that is arranged in a one-dimensional or two-dimensional array and is capable of multiply-accumulate operations in a neural network, and input data used for the calculation by the product-sum calculator is a predetermined threshold. It is provided with a threshold determination processing unit for determining whether or not it is less than, and an avoidance processing unit for avoiding the product-sum operation processing for the input data when the input data is less than the predetermined threshold. ..
  • the input data less than a predetermined threshold value is, for example, input data having a zero value or input data close to a zero value.
  • the input data includes type 1 input data and type 2 input data
  • the threshold determination processing unit performs the determination on the type 1 input data and performs the avoidance processing.
  • the unit may avoid the product-sum calculation process for the type 1 input data when the type 1 input data is less than the predetermined threshold value.
  • the product-sum calculation unit multiplies the type 1 input data and the type 2 input data. That is, when either one of the type 1 input data and the type 2 input data has a zero value, the product also has a zero value. According to this configuration, the product-sum operation processing is avoided when the type 1 input data has a zero value.
  • the type 2 input data may be weight data which is information on weights to be multiplied by the type 1 input data.
  • the weight data is, for example, a coefficient of a filter applied to image data in a predetermined range in a CNN (Convolutional Neural Network). It is unlikely that a filter will have all zero filter coefficients.
  • the threshold value determination processing unit in the signal processing device described above may be provided once for each of the plurality of product / sum calculators. It is determined whether or not each of the plurality of input data input to the plurality of product-sum calculators is less than a predetermined threshold value, for example, whether or not the value is zero.
  • the avoidance processing unit in the signal processing apparatus described above changes the input data input to the product-sum calculator when the input data is less than the predetermined threshold value, so that the input data is set to be less than the predetermined threshold value.
  • the product-sum operation processing for the data may be avoided.
  • input data of a predetermined threshold value or more is input to the product-sum calculator.
  • the above-mentioned signal processing apparatus includes a product-sum calculation control unit that manages input data and output data of the product-sum calculation processing, and the avoidance processing unit receives the input data in which the product-sum calculation processing is avoided. Information for identification may be notified to the product-sum calculation control unit. As a result, the product-sum calculation control unit can grasp the correspondence between the input data used in the product-sum calculation and the product-sum calculation result.
  • the avoidance processing unit in the above-mentioned signal processing device may be provided for each product-sum calculation unit. Since the avoidance processing unit is provided for each product-sum calculation unit, the processing load of the determination processing executed by one avoidance processing unit is light. In this determination process, it is determined whether or not the input data is less than a predetermined threshold value, for example, whether or not it is a zero value.
  • the avoidance processing unit in the signal processing device may avoid the product-sum calculation process for the input data that is less than the predetermined threshold value, and output a zero value as the processing result of the product-sum calculation process. For example, when the input data has a zero value, it is obvious that the calculation result becomes a zero value. Therefore, the output data is forcibly set to a zero value after avoiding the multiply-accumulate operation process.
  • the input data includes type 1 input data and type 2 input data
  • the avoidance processing unit determines that the type 1 input data is less than the first threshold value.
  • the product-sum calculation control unit may be notified of information for changing the type 1 input data input to the product-sum calculation unit and identifying the changed type 1 input data.
  • a comparison process with a predetermined threshold value targeting only one of the type 1 input data and the type 2 input data as input data for example, a process of determining whether or not the value is zero is executed. It is possible.
  • the avoidance processing unit in the signal processing device described above changes the type 2 input data input to the product-sum calculator when the type 2 input data is less than the second threshold value.
  • the product-sum calculation control unit obtains information for changing the type 1 input data corresponding to the type 2 input data and specifying the changed type 1 input data and the type 2 input data. May be notified to.
  • the corresponding data is a number to be multiplied with respect to the number to be multiplied in the product-sum operation.
  • the result is a zero value regardless of the value of the multiplication number.
  • processing is performed in which the number to be multiplied (type 2 input data) set to zero value is omitted and the corresponding number to be multiplied is omitted.
  • the product-sum calculation control unit in the above-mentioned signal processing device manages the product-sum calculation result of the first-class input data and the second-class input data, and compensates for the avoided product-sum calculation result with a zero value. You may.
  • the avoided product-sum operation process that is, the skipped product-sum operation process can be specified by receiving the information for specifying the corresponding type 1 input data and the type 2 input data.
  • the image pickup apparatus includes a pixel array unit in which photoelectric conversion elements are arranged in a one-dimensional or two-dimensional array, and a signal processing unit in which input data based on the output signal of the pixel array unit is input.
  • the signal processing unit is arranged in a one-dimensional or two-dimensional array and has a product-sum calculation unit capable of performing a product-sum calculation in a neural network, and the input data used for the calculation by the product-sum calculation unit has a predetermined threshold value. It is provided with a threshold determination processing unit for determining whether or not it is less than, and an avoidance processing unit for avoiding the product-sum operation processing for the input data when the input data is less than the predetermined threshold. ..
  • the signal processing unit included in the image pickup apparatus is required to save power due to problems such as a battery.
  • the pixel array unit and the signal processing unit may be integrally formed. By forming them integrally, the size of the image pickup apparatus can be reduced.
  • feature data extracted based on the output signal of the pixel array unit may be input as the input data.
  • the feature data often includes data having a zero value or less than a predetermined threshold.
  • the signal processing method determines whether or not the input data used for the product-sum calculation in the neural network is less than a predetermined threshold value, and when the input data is less than the predetermined threshold value, the input data is described.
  • This is a signal processing method in which a signal processing device executes a process of avoiding a product-sum calculation process. Even with such a signal processing method, the same operation as that of the signal processing apparatus according to the present technology can be obtained.
  • FIG. 9 is a diagram for explaining a process in which pixel data as input data is exchanged in the configuration example 1 of the signal processing unit, and this figure is a diagram showing a state before the exchange.
  • FIG. 14 is a diagram for explaining a process of exchanging weight data as input data in the configuration example 2 of the signal processing unit together with FIGS. 14 and 15, and this figure is a diagram showing a state before the exchange. It is a figure which shows the weight data of the exchange target. It is a figure which shows the state after exchange of the weight data as input data.
  • FIG. 19 is a diagram for explaining a process of exchanging pixel data as input data in the configuration example 3 of the signal processing unit, and this figure is a diagram showing a state before the exchange. It is a figure which shows the state after exchange of the pixel data as input data. It is a figure which shows the structural example 4 of a signal processing part. It is a figure which shows the configuration example of MAC in the configuration example 4 of a signal processing unit. It is a flowchart which shows the 1st processing example. It is a flowchart which shows the 2nd processing example.
  • the signal processing device of the present technology is capable of executing various operations related to image recognition processing by DNN (Deep Neural Network).
  • DNN Deep Neural Network
  • a signal processing device that performs product-sum operation processing as image recognition processing by CNN (Convolutional Neural Network), which is a kind of DNN, will be described.
  • the image pickup apparatus 1 includes an image pickup lens 2, a sensor unit 3, a control unit 4, and a recording unit 5.
  • the image pickup device 1 is assumed to have various forms such as a camera mounted on an industrial robot, an in-vehicle camera, and a surveillance camera.
  • the image pickup lens 2 collects the incident light and guides it to the sensor unit 3.
  • the image pickup lens 2 may be composed of a plurality of lenses.
  • the sensor unit 3 is configured to include a plurality of light receiving elements, and outputs a signal obtained by photoelectric conversion.
  • the control unit 4 controls the shutter speed of the sensor unit 3, gives instructions for various signal processing in each unit of the image pickup device 1, captures and records operations according to user operations, reproduces recorded image files, and captures a lens.
  • 2 Drive control for example, zoom control, focus control, aperture control, etc.
  • user interface control etc.
  • the recording unit 5 stores information and the like used for processing by the control unit 4.
  • the recording unit 5 comprehensively shows, for example, a ROM (Read Only Memory), a RAM (Random Access Memory), a flash memory, and the like.
  • the recording unit 5 may be a memory area built in the microcomputer chip as the control unit 4, or may be configured by a separate memory chip.
  • the control unit 4 controls the entire image pickup apparatus 1 by executing a program stored in the ROM, flash memory, or the like of the recording unit 5.
  • the sensor unit 3 will be specifically described with reference to FIG.
  • the sensor unit 3 includes a pixel array unit 11, an arbiter 12, a reading unit 13, a signal processing unit 14, and an output unit 15 that function as a so-called DVS (Dynamic Vision Sensor).
  • DVS Dynamic Vision Sensor
  • the sensor unit 3 is not limited to the DVS and may be configured as various image sensors.
  • the pixel array unit 11 is formed by arranging pixels 16 provided with photoelectric conversion elements in a two-dimensional array in the row direction (horizontal direction) and the column direction (vertical direction). Each pixel 16 detects the presence or absence of an event depending on whether or not the amount of change in the amount of received light exceeds a predetermined threshold value, and outputs a request to the arbiter 12 when the event occurs.
  • the arbiter 12 arbitrates the request from each pixel 16 and controls the reading operation by the reading unit 13.
  • the reading unit 13 performs a reading operation for each pixel 16 of the pixel array unit 11 based on the control of the arbiter 12.
  • Each pixel 16 outputs a signal based on the difference between the reference level and the current received signal level according to the reading operation by the reading unit 13.
  • the signal read from each pixel 16 is stored in the memory as a difference signal.
  • the pixel 16 resets the reference level to the current level of the received light signal according to the output of the difference signal. This makes it possible to detect the amount of change in the amount of received light with respect to the reference level again.
  • the difference signal is not read out and the reference level is not reset until the amount of change in the amount of received light exceeds a predetermined threshold value.
  • the signal processing unit 14 executes various signal processing (preprocessing and the like) for image data input from the reading unit 13 as feature amount data, image recognition processing by DNN, and the like.
  • image recognition processing by CNN which is a kind of DNN, will be taken as an example.
  • the image recognition process for example, it is possible to execute an arithmetic process related to a convolution process by a convolution layer, a max pooling process by a pooling layer, a classification process by a fully connected layer and an output layer, and the like.
  • an arithmetic process related to a convolution process by a convolution layer e.g., a convolution layer
  • a max pooling process by a pooling layer e.g., a max pooling process by a pooling layer
  • a classification process by a fully connected layer and an output layer
  • the output unit 15 outputs the classification result by CNN to the control unit 4 in the subsequent stage based on a predetermined interface standard (for example, MIPI (Mobile Industry Processor Interface)).
  • the control unit 4 receives the classification result by CNN and uses it for various processes.
  • the signal processing unit 14 executes only a part of various processes related to the CNN, the processing result in the signal processing unit 14, that is, the intermediate processing result in the CNN is output from the output unit 15.
  • the signal processing unit 14 includes a MAC array unit 17, a signal processing control unit 18, and a memory unit 19 in order to execute the product-sum calculation process.
  • the MAC array unit 17 is composed of a multiply-accumulate unit (MAC) arranged in a two-dimensional array in the row direction (horizontal direction) and the column direction (vertical direction).
  • the product-sum calculator may be arranged in a one-dimensional array along either the row direction or the column direction.
  • the product-sum calculator is also referred to as MAC20.
  • Each MAC 20 is formed with a circuit for performing multiplication processing and addition processing on the data input from the memory unit 19.
  • the input data input to one MAC 20 is, for example, data for one pixel of image data output from the pixel array unit 11 or weight data to be multiplied by the data for the one pixel.
  • the weight data is used as a filter coefficient of a filter applied to the image data.
  • the image data input to the MAC 20 may be not only the image data output from the pixel array unit 11 but also the output image data in another convolution layer or pooling layer. In the following description, such image data will be referred to as “processing target data”.
  • An example of the operation performed by the MAC 20 will be described using a process target data represented by binary values (0 and 1) and a filter having two pixels both vertically and horizontally applied to the process target data.
  • FIG. 4 is a diagram showing the target area AR1 which is the area to which the processing target data and the filter are applied.
  • the value of the upper left pixel data a11 and the value of the upper right pixel data a12 are both set to "1"
  • the value of the lower left pixel data a21 and the value of the lower right pixel data a22 are both. It is set to "0".
  • FIG. 5 is a diagram showing a filter F1 applied to the target region AR1.
  • Each coefficient of the filter F1 has weight data w11, w12, w21, and w22.
  • the values of the upper left weight data w11 and the lower right weight data w22 in the filter F1 are set to "1", and the values of the upper right weight data w12 and the lower left weight data w21 are set to "0".
  • Equation (1) can be performed using four MAC20s.
  • the pixel data a11 and the wait data w11 are input to the MAC 20a.
  • the multiplication process of the pixel data a11 and the weight data w11 is performed, and the multiplication result is output as the output OP1.
  • the MAC 20b performs a multiplication process of the pixel data a12 and the coefficient w12, and further performs an addition process of the result of the multiplication process and the output OP1.
  • the addition result is output as output OP2.
  • Pixel data a21, wait data w21, and output OP2 are input to the MAC20c.
  • the MAC 20c performs a multiplication process of the pixel data a21 and the weight data w21, and performs an addition process of the result of the multiplication process and the output OP2.
  • the addition result is output as output OP3.
  • Pixel data a22, wait data w22, and output OP3 are input to the MAC 20d.
  • the MAC 20d performs a multiplication process of the pixel data a22 and the weight data w22, and performs an addition process of the result of the multiplication process and the output OP3.
  • the addition result is output as output OP4.
  • the calculation result of the equation (1) is output from the MAC 20d as the output OP4.
  • MAC20a, 20b, 20c, and 20d may be controlled so as to perform only multiplication processing.
  • a process of adding the outputs OP1, OP2, OP3, and OP4 may be executed in the MAC20 other than the MAC20a, 20b, 20c, and 20d.
  • the output OP4 may be configured to be the calculation result of the equation (1) by performing the process of adding the outputs OP1, OP2, OP3 to the multiplication result in the MAC 20d.
  • the signal processing control unit 18 reads out the processing target data (pixel data) and the filter coefficient (wait data) stored in the memory unit 19 and inputs them to each MAC 20 of the MAC array unit 17. Further, the signal processing control unit 18 has a function of avoiding an operation such that the operation result becomes a zero value. Specifically, it will be described later.
  • the signal processing control unit 18 performs a process of storing the calculation result of the MAC array unit 17 in the memory unit 19. In addition, processing such as transmitting the calculation result to the outside of the signal processing control unit 18 is performed.
  • the image pickup apparatus 1 shown in FIGS. 1, 2 and 3 is an example including an image sensor in which a pixel array unit 11 and a signal processing unit 14 are integrally formed.
  • the pixel array unit 11 or the like is arranged on the front surface, and the GPU or DSP as the signal processing unit 14 is formed on the back surface.
  • the image sensor does not have to include the signal processing unit 14. That is, the image sensor and the signal processing unit 14 may be provided separately.
  • FIG. 7 shows a specific configuration of the signal processing unit 14A in the configuration example 1.
  • the signal processing unit 14A in the configuration example 1 is provided with an avoidance processing unit 21 on either one of the two data input to the multiplication circuit of the MAC 20, specifically the pixel data and the weight data (filter coefficient) described above. Has been done. Further, one avoidance processing unit 21 is provided for each of the plurality of MAC 20s. In the example shown in FIG. 7, one avoidance processing unit 21 is provided for one MAC array unit 17 having a plurality of MAC 20s.
  • the signal processing unit 14A includes an avoidance processing unit 21, a first memory 22, a second memory 23, a third memory 24, a product-sum operation control unit 25, a first local memory 26, and a second local memory. 27.
  • a plurality of MAC 20s arranged in a two-dimensional array to form the MAC array unit 17 are provided.
  • the avoidance processing unit 21 and the product-sum calculation control unit 25 are the signal processing control units 18 shown in FIG. Further, the first memory 22, the second memory 23, and the third memory 24 are the memory unit 19 shown in FIG. The first memory 22, the second memory 23, and the third memory 24 may be provided as physically different memories, or may be provided as different areas of one memory.
  • Image data as processing target data is stored in the first memory 22.
  • Weight data is stored in the second memory 23.
  • the calculation result is stored in the third memory 24.
  • the calculation result stored in the third memory 24 may be output from the signal processing unit 14 or may be output to the first memory 22 as the processing target data input to the MAC array unit 17.
  • the calculation result stored in the third memory 24 may be input to the MAC array unit 17 from the third memory 24 without going through the first memory 22.
  • the avoidance processing unit 21 reads the processing target data from the first memory 22 and inputs it to each MAC 20 of the MAC array unit 17 via the first local memory 26.
  • the wait data stored in the second memory 23 is temporarily stored in the second local memory 27, and then input to each MAC 20 of the MAC array unit 17.
  • each MAC20 the pixel data for one pixel and the weight data in the input processing target data are multiplied.
  • the product-sum operation in MAC 20 may be wasted depending on the input processing target data. For example, in the examples shown in FIGS. 4, 5 and 6, when all of the pixel data a11, a12, a21 and a22 have zero values, the weight data w11, w12, w21 and w22 do not matter. Since the operation result of the equation (1) is always a zero value, it is not necessary to perform the product-sum operation.
  • the avoidance processing unit 21 performs processing for avoiding such unnecessary operations. Specifically, it will be described with reference to FIGS. 8 and 9.
  • FIG. 8 is an excerpt of a part of the MAC array unit 17 shown in FIG. 7. Specifically, eight MAC20-1, MAC20-2, MAC20-3, MAC20-4, MAC20-5, MAC20-6, MAC20-7, and MAC20-8 are shown among a plurality of MAC20s. ..
  • the four MAC20s, MAC20-1, MAC20-2, MAC20-3, and MAC20-4, are multiply-accumulate calculators that perform convolution processing for the target area AR1 to which the filter is applied in the processing target data.
  • the four MAC20s, MAC20-5, MAC20-6, MAC20-7, and MAC20-8, are multiply-accumulate calculators that perform convolution processing for the target area AR2 to which the filter is applied in the processing target data.
  • the pixel data in the target area AR2 have zero values. That is, the pixel data b11, b12, b21, and b22 are all zero values.
  • the four MAC20s, MAC20-5, MAC20-6, MAC20-7, and MAC20-8, do not need to perform multiply-accumulate processing.
  • the avoidance processing unit 21 avoids the convolution processing (product-sum operation processing) for the target area AR2, and instead performs the convolution processing for the target area AR3. That is, the pixel data c11, c12, c21, and c22 of the target region AR3 are input to the four MAC20s of MAC20-5, MAC20-6, MAC20-7, and MAC20-8, respectively (see FIG. 9).
  • the product-sum calculation process for the target area AR is canceled, and the MAC 20 is used for the product-sum calculation process for the other target area AR. To use.
  • the target areas AR1, AR2, and AR3 are shown not to overlap each other for the sake of simplicity, but some of them may overlap depending on the stride amount (shift amount) of the filter. In some cases. For example, when the stride amount is "1", the pixel data a12 in the target area AR1 and the pixel data b11 in the target area AR2 are the same pixel data.
  • the product-sum calculation control unit 25 performs a process of storing the calculation result output from the MAC array unit 17 in the third memory 24. At this time, if the relationship between the calculation result output from the MAC array unit 17 and the target area AR is not correctly linked, the result of the convolution process cannot be handled appropriately.
  • the avoidance processing unit 21 performs the processing for avoiding unnecessary operations as described above, the information for specifying the avoided operations for the product-sum operation control unit 25, or the MAC array unit 17 Notifies information for identifying which target area AR the operation performed using is.
  • the product-sum calculation control unit 25 stores the product-sum calculation result in the third memory 24. At this time, the zero value is stored in the third memory 24 for the avoided product-sum calculation result.
  • the product-sum calculation control unit 25 can appropriately handle the calculation result output from the MAC array unit 17.
  • FIG. 10 shows a specific configuration of the signal processing unit 14B in the configuration example 2.
  • the signal processing unit 14B in the configuration example 2 has a configuration that avoids the product-sum operation related to the weight data w when a part of the weight data w in the filter F has a zero value. That is, the signal processing unit 14B includes a second avoidance processing unit 21b.
  • the filter F2 in this example is shown in FIG. 11, and the processing target data and the target areas AR4, AR5, AR6 are shown in FIG.
  • the filter F2 has 3 pixels both vertically and horizontally.
  • the target areas AR4, AR5, and AR6 are also set to be areas with three vertical and horizontal pixels.
  • the values of the weight data w11, w12, w13, w22, w31, w32, and w33 in the filter F2 are set to "1", and the values of the weight data w21 and w23 are set to "0".
  • the target area AR4 is pixel data d11, d12, d13, d21, d22, d23, d31, d32, d33.
  • the target area AR5 is pixel data e11, e12, e13, e21, e22, e23, e31, e32, e33.
  • the target area AR6 has pixel data f11, f12, f13, f21, f22, f23, f31, f32, and f33.
  • the processing target data stored in the first memory 22 is input to each MAC 20 of the MAC array unit 17 via the first avoidance processing unit 21a (see FIG. 10).
  • the wait data stored in the second memory 23 is input to each MAC 20 of the MAC array unit 17 via the second avoidance processing unit 21b.
  • the second avoidance processing unit 21b cancels the product-sum calculation using the weight data w21, and instead performs the product-sum calculation using the weight data w22 (see FIG. 14). Along with this, the second avoidance processing unit 21b notifies the first avoidance processing unit 21a of the avoided weight data w21 and the newly adopted weight data w22 (see FIG. 10).
  • the first avoidance processing unit 21a cancels inputting the pixel data d21, e21, f21 scheduled to be used for the multiplication processing related to the weight data w22 into MAC20-4, MAC20-8, MAC20-12, and is adopted instead. It is determined that the pixel data d22, e22, and f22 used for the multiplication process related to the weight data w22 are input to MAC20-4, MAC20-8, and MAC20-12 (see FIG. 14).
  • the first avoidance processing unit 21a notifies the product-sum calculation control unit 25 of the pixel data d22, e22, f22 used for the product-sum calculation instead of the pixel data d21, e21, f21 that avoided the product-sum calculation. This allows the product-sum calculation control unit 25 to appropriately handle the calculation result. Further, the first avoidance processing unit 21a may notify the product-sum calculation control unit 25 of the weight data w that avoids the product-sum calculation and the weight data w that is adopted instead of notifying the pixel data.
  • the product-sum calculation control unit 25 stores the product-sum calculation result output from the MAC array unit 17 in the third memory 24. At this time, the zero value is stored in the third memory 24 for the avoided product-sum calculation result. As a result, the product-sum calculation control unit 25 can appropriately handle the calculation result output from the MAC array unit 17.
  • the weight data set to a zero value and the corresponding pixel data are temporarily loaded in the first local memory 26 and the second local memory 27. ing. However, in reality, before being loaded into the first local memory 26 or the second local memory 27, a determination process of whether or not the value is zero and a process of determining whether or not the pixel data is corresponding to the zero value are performed. You may. In that case, the weight data set to the zero value and the corresponding pixel data are not loaded into the first local memory 26 or the second local memory 27.
  • Configuration example 3 The signal processing unit 14C in the configuration example 3 has a configuration for applying a plurality of filters F3, F4, and F5 to one target region AR.
  • the four target regions AR7, AR8, AR9, AR10 and the three filters F3, F4, and F5 will be described as examples.
  • the target areas AR7, AR8, AR9, and AR10 are two pixel areas both vertically and horizontally.
  • the target area AR7 is composed of pixel data g11, g12, g21, and g22.
  • the target area AR8 is composed of pixel data h11, h12, h21, h22
  • the target area AR9 is composed of pixel data i11, i12, i21, i22
  • the target area AR10 is composed of pixel data j11, j12, j21, j22. ..
  • the filters F3, F4, and F5 applied to each target area AR7, AR8, AR9, and AR10 are also set to have a size of two pixels both vertically and horizontally.
  • the filter F3 is composed of weight data wa11, wa12, wa21, wa22
  • the filter F4 is composed of weight data wb11, wb12, wb21, wb22
  • the filter F5 is composed of weight data wc11, wc12, wc21, wc22.
  • the filter F3 For example, by applying the filter F3 to the target area AR7, the calculation of g11 ⁇ wa11 + g12 ⁇ wa12 + g21 ⁇ wa21 + g22 ⁇ wa22 is performed. Further, by applying the filter F4 to the target region AR7, the calculation of g11 ⁇ wb11 + g12 ⁇ wb12 + g21 ⁇ wb21 + g22 ⁇ wb22 is performed. Then, by applying the filter F5 to the target region AR7, the calculation of g11 ⁇ wc11 + g12 ⁇ wc12 + g21 ⁇ wc21 + g22 ⁇ wc22 is performed.
  • one operation result is obtained by adding the operation result of applying the filter F3 to the target area AR7, the operation result of applying the filter F4, and the operation result of applying the filter F5.
  • FIG. 17 shows a configuration example of the signal processing unit 14C when performing such a convolution process.
  • the signal processing unit 14C includes a first memory 22 and an avoidance processing unit 21, and the avoidance processing unit 21 performs a process of loading the pixel data stored in the first memory 22 into the first local memory 26.
  • the pixel data g11 of the target area AR7, the pixel data h11 of the target area AR8, the pixel data i11 of the target area AR9, and the pixel data j11 of the target area AR10 are loaded into the first local memory 26.
  • the signal processing unit 14C includes a second memory 23 and a second local memory 27, and loads the wait data stored in the second memory 23 into the second local memory 27.
  • the weight data wa11 of the filter F3, the weight data wb11 of the filter F4, and the weight data wc11 of the filter F5 are loaded into the second local memory 27.
  • FIG. 18 is a second arithmetic process for the target area AR7 using the MAC array unit 17.
  • the convolution process in this example can be realized by repeating the product-sum operation using the MAC array unit 17.
  • the pixel data g11, h11, i11, and j11 shown in FIG. 17 were all "1".
  • the pixel data h12, i12, and j12 shown in FIG. 18 are "1", but the pixel data g12 has a zero value.
  • the avoidance processing unit 21 does not load the pixel data g12 into the first local memory 26, but loads the pixel data of the other target area AR into the first local memory 26. That is, the state is as shown in FIG.
  • the pixel data k12 is pixel data of the target area AR other than the target areas AR7, AR8, AR9, and AR10.
  • the data is loaded into the first local memory 26 while avoiding the pixel data set to the zero value.
  • the avoidance processing unit 21 notifies the product-sum operation control unit 25 of information for identifying the pixel data that has not been loaded into the first local memory 26.
  • the product-sum calculation control unit 25 compensates the avoided product-sum calculation result with a zero value and stores it in the third memory 24. As a result, the product-sum calculation control unit 25 can appropriately handle the calculation result output from the MAC array unit 17.
  • an avoidance processing unit 21 that performs a process of determining whether or not the pixel data is a zero value and selecting the pixel data to be loaded into the first local memory 26.
  • the avoidance processing unit 21 may be provided to perform a process of determining whether or not the weight data is a zero value and selecting the weight data to be loaded into the second local memory 27.
  • the avoidance processing unit 21 related to the pixel data and the avoidance processing unit 21 related to the weight data may be provided together, or only the avoidance processing unit 21 related to the wait data may be provided.
  • Configuration example 4 The signal processing unit 14D in the configuration example 4 is provided with an avoidance processing unit 21D for each MAC 20D.
  • pixel data is loaded from the first memory 22 into the first local memory 26 without going through the avoidance processing unit 21.
  • the wait data is loaded from the second memory 23 into the second local memory 27 without going through the avoidance processing unit 21.
  • Pixel data and wait data are input to the respective MAC 20Ds from the first local memory 26 and the second local memory 27.
  • the MAC 20D includes an avoidance processing unit 21D and a zero value output unit 28 as shown in FIG. 21 in addition to the addition circuit and the multiplication circuit.
  • the avoidance processing unit 21D determines whether or not the input pixel data has a zero value. When it is determined that the pixel data has a zero value, the clock applied to the MAC 20D is stopped, and the zero value output unit 28 is operated to output the zero value as output data.
  • the avoidance processing unit 21D and the zero value output unit 28 can be configured by a logic circuit or the like. For example, the zero value output unit 28 can forcibly set the output value to the zero value by using the zero value and the AND circuit.
  • the power consumption of the MAC20D can be suppressed, which can contribute to power saving.
  • the clock may be stopped and the zero value output process may be executed.
  • both the input pixel data and the wait data may be monitored, and if at least one of them has a zero value, the clock may be stopped and the zero value output process may be performed.
  • the result of the avoided product-sum calculation is output to the MAC 20D and the product-sum calculation control unit 25 in the next stage, so that the avoided product-sum operation can be specified. It is not necessary to notify the product-sum calculation control unit 25 of the information of.
  • First processing example> it is determined whether or not the pixel data has a zero value, and the product-sum operation is appropriately avoided. For example, by executing the first processing example, the configuration example 1 of the signal processing unit 14A can be realized.
  • step S100 of FIG. 22 the signal processing unit 14A acquires wait data from the second memory 23 and loads it into the second local memory 27.
  • the signal processing unit 14A acquires pixel data from the first memory 22 in step S101. Subsequently, in step S102, the signal processing unit 14A determines whether or not the predetermined pixel data group includes non-zero value data.
  • the predetermined pixel data group is, for example, the pixel data a11, a12, a21, a22 of the target area AR1 shown in FIG. 8, the pixel data b11, b12, b21, b22 of the target area AR2, and the like.
  • the signal processing unit 14A performs step S103.
  • the product / sum operation control unit 25 is notified of the information for specifying the avoided operation.
  • the product-sum operation control unit 25 is notified of vertical and horizontal position information (for example, x-coordinate and y-coordinate) for specifying the position of the controlled target area.
  • the signal processing unit 14A (avoidance processing unit 21) returns to the processing of step S101 and acquires the next pixel data.
  • step S102 when it is determined in step S102 that the predetermined pixel data group includes non-zero value data, the signal processing unit 14A (avoidance processing unit 21) loads the acquired pixel data into the first local memory 26 in step S104. do.
  • the signal processing unit 14A determines in step S105 whether or not the pixel data loading is completed. When it is determined that the loading of the pixel data is not completed, the signal processing unit 14A (avoidance processing unit 21) returns to the processing of step S101 and acquires the next pixel data.
  • step S105 when it is determined in step S105 that the loading of the pixel data is completed, the signal processing unit 14A executes the product-sum operation in step S106. This process is executed at the timing when the data required for the product-sum operation is prepared in each of the first local memory 26 and the second local memory 27.
  • the signal processing unit 14A transmits the calculation result to the product-sum calculation control unit 25 in step S107.
  • the signal processing unit 14A (product-sum operation control unit 25) compensates for the zero value as the operation result of the avoided operation in step S108. As a result, it is possible to prevent the operation result of the avoided operation from being missing.
  • the signal processing unit 14A (product-sum operation control unit 25) performs a process of storing the operation result in the third memory 24 in step S109.
  • the signal processing unit 14A determines in step S110 whether or not all the operations have been completed. If the calculation is not completed, a series of processes starting from step S100 are executed again for the new image data and the data as the calculation result stored in the third memory 24 in step S109.
  • step S110 when it is determined in step S110 that all the operations have been completed, the signal processing unit 14A (product-sum operation control unit 25) ends a series of processes shown in FIG. At this time, a process of outputting the final calculation result stored in the third memory 24 to the outside of the signal processing unit 14A may be executed.
  • Second processing example> it is determined whether or not the pixel data has a zero value and the product-sum operation is avoided as appropriate, and whether or not the weight data has a zero value is determined and the product-sum operation is avoided as appropriate. It is something to do.
  • the configuration example 2 of the signal processing unit 14B can be realized.
  • the signal processing unit 14B (second avoidance processing unit 21b) acquires wait data from the second memory 23 in step S201 of FIG.
  • the signal processing unit 14B determines in step S202 whether or not the acquired wait data has a zero value. If it is determined that the value is zero, the signal processing unit 14B (second avoidance processing unit 21b) notifies the product-sum operation control unit 25 of the position information of the wait data in step S203.
  • the signal processing unit 14B (second avoidance processing unit 21b) returns to the processing of step S201 and acquires the next pixel data.
  • the signal processing unit 14B (second avoidance processing unit 21b) loads the acquired weight data into the second local memory 27 in step S204.
  • the signal processing unit 14B determines in step S205 whether or not the load of the wait data is completed. When it is determined that the loading of the wait data is not completed, the signal processing unit 14B (second avoidance processing unit 21b) returns to the processing of step S201 and acquires the next weight data.
  • step S205 when it is determined in step S205 that the loading of the wait data is completed, the signal processing unit 14B (first avoidance processing unit 21a) acquires pixel data from the first memory 22 in step S101.
  • the signal processing unit 14B corresponds to the weight data determined to be a zero value, that is, the weight data not loaded in the second local memory 27. Judge whether or not.
  • the corresponding pixel data is, for example, the pixel data d21, the pixel data e21, the pixel data f21, etc. shown in FIG.
  • the signal processing unit 14B (first avoidance processing unit 21a) loads the acquired pixel data into the first local memory 26. Instead, new pixel data is acquired in step S101.
  • the signal processing unit 14B determines that the acquired pixel data is zero in step S207. Determine if it is a value.
  • the signal processing unit 14B notifies the product-sum calculation control unit 25 of the position information of the pixel data in step S208. That is, the acquired pixel data is not loaded into the first local memory 26.
  • the signal processing unit 14B (first avoidance processing unit 21a) transfers the acquired pixel data to the first local in step S104. Load into memory 26.
  • the signal processing unit 14B determines in step S105 whether or not the pixel data loading is completed. When it is determined that the loading of the pixel data is not completed, the signal processing unit 14B (first avoidance processing unit 21a) returns to the processing of step S101 and acquires the next pixel data.
  • step S105 when it is determined in step S105 that the loading of the pixel data is completed, the signal processing unit 14B executes the product-sum calculation in step S106 of FIG. 24, and transmits the calculation result to the product-sum calculation control unit 25 in step S107. ..
  • the signal processing unit 14B compensates for the zero value as the operation result of the avoided operation in step S108, and stores the operation result in the third memory 24 in step S109. conduct.
  • the signal processing unit 14B determines in step S110 whether or not all the operations have been completed. If the calculation is not completed, the process returns to the process of step S201 in order to perform a new product-sum calculation.
  • step S110 when it is determined in step S110 that all the operations have been completed, the signal processing unit 14B (product-sum operation control unit 25) ends a series of processes shown in FIGS. 23 and 24. At this time, a process of outputting the final calculation result stored in the third memory 24 to the outside of the signal processing unit 14B may be executed.
  • the operation using the same filter F may not be completed only by executing the product-sum operation process in step S106 once. In that case, after finishing the process of step S110, the process returns to step S101 of FIG. 23 without returning to step S201. As a result, the product-sum operation is properly executed.
  • the third processing example is an example of a flowchart for realizing the configuration example 4 of the signal processing unit 14D. That is, the third processing example is for realizing a configuration in which the avoidance processing unit 21D and the zero value output unit 28 are provided for each MAC 20D.
  • the signal processing unit 14D acquires weight data from the second memory 23 in step S100 of FIG. 25 and loads it into the second local memory 27.
  • the signal processing unit 14D acquires pixel data from the first memory 22 and loads it into the first local memory 26 in step S301.
  • the signal processing unit 14D determines in step S302 whether or not the input pixel data has a zero value. This process is performed for each MAC 20D.
  • the signal processing unit 14D performs clock stop processing in step S303. Further, the signal processing unit 14D (avoidance processing unit 21D) causes the zero value output unit 28 to execute the zero value output process in step S304. As a result, the multiply-accumulate operation is avoided and the power consumption is reduced in the MAC 20D. Further, a zero value is output from the MAC 20D as a calculation result.
  • the signal processing unit 14D executes the product-sum calculation process in step S106.
  • the product-sum operation related to the pixel data as the input data and the weight data is executed.
  • the signal processing unit 14D After finishing the processing of step S304 or after finishing the processing of step S106, the signal processing unit 14D transmits the calculation result to the product-sum calculation control unit 25 in step S107.
  • the signal processing unit 14D (product-sum operation control unit 25) performs a process of storing the operation result in the third memory 24 in step S109.
  • the signal processing unit 14D determines in step S110 whether or not all the operations have been completed. When the calculation is not completed, a series of processes starting from step S100 in FIG. 25 is executed again for the new image data and the data as the calculation result stored in the third memory 24 in step S109.
  • step S110 when it is determined in step S110 that all the operations have been completed, the signal processing unit 14D (product-sum operation control unit 25) ends a series of processes shown in FIG.
  • the input data is less than a predetermined threshold value
  • the pixel data is represented by 4 bits, that is, when the pixel data is a numerical value of any of 0 to 15, and the predetermined threshold value is "4" and the pixel data is 0 to 3. Avoid product-sum operations related to pixel data.
  • the predetermined threshold value "4" is an example, and may be any number such as "8" or "10".
  • the predetermined pixel data group is pixel data having a predetermined threshold value or more. It may be determined whether or not it contains.
  • the predetermined threshold value used for determining the pixel data and the predetermined threshold value used for determining the weight data may be different.
  • the predetermined threshold value used for determining the pixel data may be the first threshold value (for example, “4”), and the predetermined threshold value used for determining the weight data may be set as the second threshold value (for example, “2”).
  • step S202 it is determined in step S202 whether or not the weight data is less than a predetermined threshold value instead of determining whether or not the weight data is a zero value. Then, in step S206 of FIG. 23, it is determined whether or not it corresponds to the weight data determined to be less than the predetermined threshold value, and in step S207, it is determined whether or not the pixel data is less than the predetermined threshold value.
  • the MAC 20E may be capable of performing operations on a recurrent neural network (RNN). Specifically, the MAC 20E may be equipped with an LSTM (Long Short-Term Memory) (see FIG. 26).
  • RNN recurrent neural network
  • LSTM Long Short-Term Memory
  • a signal processing unit 14F provided with an avoidance processing unit 21 or the like may be provided outside the sensor unit 3.
  • the sensor unit 3F includes a pixel array unit 11, a reading unit 13, a preprocessing unit 29, and an output unit 15, and the output unit 15 is connected to the bus 30.
  • the pre-processing unit 29 is a portion that performs signal processing as pre-processing among various processes executed by the signal processing unit 14 in each of the above-mentioned examples.
  • a control unit 4 including a memory 31 and a signal processing unit 14F is connected to the bus 30. That is, a signal processing unit 14F provided with the above-mentioned avoidance processing unit 21 and the like is provided outside the sensor unit 3F.
  • the signal processing unit 14F provided with the avoidance processing unit 21 and the like may be provided outside the sensor unit 3F and outside the control unit 4.
  • the sensor unit 3F includes a pixel array unit 11, a reading unit 13, a preprocessing unit 29, and an output unit 15, and the output unit 15 is connected to the bus 30.
  • the control unit 4, the memory 31, and the signal processing unit 14F are connected to the bus 30.
  • the signal processing unit 14F includes a signal processing control unit 18 including a MAC array unit 17, an avoidance processing unit 21, and the like, and a memory unit 19.
  • a signal processing unit 14F provided with an avoidance processing unit 21 or the like may be provided in another signal processing device.
  • the image pickup device 1 including the sensor unit 3F, the control unit 4, the memory 31, and the communication unit 32, and another signal processing device 34 including the signal processing unit 14F and the communication unit 32 are described above. Various functions may be realized.
  • the communication unit 32 of the image pickup device 1 is capable of data communication by wire or wirelessly with the communication unit 33 of another signal processing device 34. By adopting such various configurations, it is possible to realize various functions as the above-mentioned signal processing unit.
  • the application target of the processing may be one-dimensional data.
  • the one-dimensional data is, for example, audio data, output data such as velocity data, acceleration data, angular velocity data, etc. output from a gyro sensor, position information, and the like.
  • These one-dimensional data may be made into two-dimensional data by arranging each predetermined amount of data in a different dimensional direction.
  • the image pickup device 1 as a signal processing device is a product-sum calculator (MAC20, 20D, 20E) arranged in a one-dimensional or two-dimensional array and capable of a product-sum operation in a neural network, and a product-sum.
  • Threshold determination processing unit (avoidance processing units 21 and 21D, first avoidance processing unit 21a, second avoidance) for determining whether or not the input data (pixel data, weight data) used for the calculation by the arithmetic unit is less than a predetermined threshold value.
  • avoidance processing units 21 and 21D that avoid product-sum operation processing for input data when the input data is less than a predetermined threshold. It is equipped with.
  • the input data less than a predetermined threshold value is, for example, input data having a zero value or input data close to a zero value.
  • the product-sum operation result has a zero value, and it can be calculated without executing the product-sum operation process.
  • the product-sum calculation is avoided when the input data is a zero value, so that the product-sum calculation unit is prevented from being used to execute a useless calculation, and the power consumption is reduced. It is possible to plan.
  • the input data includes the first type input data (pixel data) and the second type input data (wait data), and the threshold determination processing unit (avoidance processing unit).
  • 21,21D, 1st avoidance processing unit 21a, 2nd avoidance processing unit 21b) determines the type 1 input data, and avoidance processing units 21 and 21D (1st avoidance processing unit 21a, 2nd avoidance processing unit 21b). ) May avoid the product-sum calculation process for the type 1 input data when the type 1 input data is less than a predetermined threshold value.
  • the product-sum calculator multiplies the type 1 input data and the type 2 input data. That is, when either one of the type 1 input data and the type 2 input data has a zero value, the multiplication result also has a zero value. According to this configuration, the product-sum operation processing is avoided when the type 1 input data has a zero value. According to this configuration, since the product-sum operation process is avoided when the type 1 input data has a zero value, it is possible to efficiently avoid the product-sum operation in which the operation result is a zero value.
  • the type 2 input data may be weight data which is information on the weight to be multiplied by the type 1 input data (pixel data).
  • the weight data is, for example, a coefficient of a filter applied to image data in a predetermined range in CNN. It is unlikely that a filter will have all zero filter coefficients. Therefore, for example, by performing a determination process of determining whether or not the type 1 input data which is the image data of a predetermined area has a zero value and appropriately avoiding the product-sum operation process, unnecessary product-sum operation can be efficiently performed. It is possible to eliminate it and save power.
  • the threshold value determination processing unit (avoidance processing unit 21 and 21D, the first avoidance processing unit 21a, the second avoidance processing unit 21b) includes a plurality of multiply-accumulate units (MAC20, 20D, 20E). ) May be provided one by one. It is determined whether or not each of the plurality of input data input to the plurality of product-sum calculators is less than a predetermined threshold value, for example, whether or not the value is zero. This makes it possible to perform processing such as exchanging input data that is determined to be less than a predetermined threshold value, and the product-sum calculation unit can be used efficiently. That is, it is possible to reduce the total number of times the multiply-accumulate calculator is used until a predetermined result is obtained, and it is possible to contribute to the reduction of consumption reduction.
  • MAC20, 20D, 20E multiply-accumulate units
  • the avoidance processing units 21 and 21D are input data (pixel data, wait data).
  • the product-sum calculation unit MAC20, 20D, 20E
  • the product-sum calculation processing for the input data set to be less than the predetermined threshold may be avoided. ..
  • input data of a predetermined threshold value or more is input to the product-sum calculator. Therefore, the product-sum calculation unit can be effectively used and unnecessary product-sum calculation can be prevented from being executed.
  • the product-sum calculation control unit 25 that manages the input data (pixel data, wait data) and output data of the product-sum calculation process is provided, and avoidance processing is performed.
  • Units 21 and 21D may notify the product-sum calculation control unit 25 of information for identifying input data in which the product-sum calculation process has been avoided. ..
  • the product-sum calculation control unit 25 can grasp the correspondence between the input data used in the product-sum calculation and the product-sum calculation result. Therefore, the calculation result can be handled appropriately, and for example, the convolution process in CNN can be correctly executed.
  • unnecessary product-sum calculation processing such that the calculation result becomes a zero value is avoided, so that power saving can be achieved.
  • the avoidance processing units 21 and 21D may be provided for each product-sum calculation unit (MAC20, 20D, 20E). ..
  • the processing load of the determination processing executed by one avoidance processing unit 21 is light.
  • This determination process determines whether or not the input data (pixel data, weight data) is less than a predetermined threshold value, for example, whether or not it is a zero value. This makes it possible to avoid the product-sum operation process without performing a process such as replacing the input data with a non-zero value one. Therefore, power saving can be achieved by simple processing.
  • the avoidance processing unit 21D avoids the product-sum calculation process for the input data (pixel data, weight data) that is less than the predetermined threshold value, and the processing result of the product-sum calculation process is a zero value. May be output.
  • the output data is forcibly set to a zero value after avoiding the multiply-accumulate operation process. As a result, correct output data can be obtained as the product-sum calculation result, and the effect of reducing power consumption by avoiding the calculation process can be obtained.
  • the input data includes type 1 input data (pixel data) and type 2 input data (wait data), and the avoidance processing units 21 and 21D (first avoidance processing unit 21a and second avoidance processing unit 21b) are used.
  • the type 1 input data is less than the first threshold value
  • the type 1 input data input to the product-sum calculator (MAC20, 20D, 20E) is changed and the changed type 1 input data is specified.
  • the product / sum operation control unit 25 may be notified of the information for this purpose.
  • the avoidance processing unit (first avoidance processing unit 21a, second avoidance processing unit 21b) has a second type input data (wait data).
  • the type 2 input data input to the product-sum calculator (MAC20) is changed, and the type 1 input data (pixel data) corresponding to the changed type 2 input data is changed.
  • the corresponding data is a number to be multiplied with respect to the number to be multiplied in the product-sum operation.
  • the result is a zero value regardless of the value of the multiplication number.
  • processing is performed in which the number to be multiplied (type 2 input data) set to zero value is omitted and the corresponding number to be multiplied is omitted.
  • the product-sum operation control unit can grasp the avoided multiplication process and addition process, the operation result of the product-sum operation process can be appropriately handled. Further, since the number of multiplication processes and addition processes executed to obtain a specific result can be reduced, it is possible to contribute to power saving.
  • the product-sum calculation control unit 25 manages the product-sum calculation result of the first-class input data (pixel data) and the second-class input data (wait data), and avoids the product-sum calculation.
  • the calculation result may be supplemented with a zero value.
  • the avoided product-sum operation process that is, the skipped product-sum operation process can be specified by receiving the information for specifying the corresponding type 1 input data and the type 2 input data. Then, as the processing result of the specified product-sum calculation processing, the processing result of the product-sum calculation processing can be obtained so that there is no omission of data by compensating for the zero value and managing it. Therefore, the convolution operation in CNN or the like can be efficiently performed with low power consumption.
  • the image pickup apparatus 1 includes a pixel array unit 11 in which photoelectric conversion elements (pixels 16) are arranged in a one-dimensional or two-dimensional array, and a pixel array.
  • a signal processing unit 14 (14A, 14B, 14C, 14D, 14F) into which input data (pixel data, weight data) based on the output signal of the unit 11 is input, and the signal processing unit 14 is one-dimensional or two-dimensional.
  • the product-sum calculation unit (MAC20, 20D, 20E) arranged in a dimensional array and capable of product-sum calculation in a neural network, and whether or not the input data used for the calculation by the product-sum calculation unit is less than a predetermined threshold.
  • the signal processing unit 14 included in the image pickup apparatus 1 is required to save power due to problems such as a battery. According to this configuration, in an image pickup apparatus capable of carrying out at least a part of a convolution operation in a CNN or the like, the power consumed in the product-sum operation process can be reduced, which is suitable.
  • the pixel array unit 11 and the signal processing unit 14 may be integrally formed.
  • the image pickup device 1 can be downsized. Therefore, the ease of handling of the image pickup apparatus 1 can be improved.
  • the signal processing unit 14 may input feature data extracted based on the output signal of the pixel array unit 11 as input data. ..
  • the feature data often includes data having a zero value or less than a predetermined threshold. Therefore, in many cases, the product-sum calculation process can be performed with high efficiency, and the effect of reducing power consumption can be further enhanced.
  • a product-sum calculator that is arranged in a one-dimensional or two-dimensional array and is capable of product-sum operations in a neural network.
  • a threshold value determination processing unit that determines whether or not the input data used for the calculation by the product-sum calculation unit is less than a predetermined threshold value.
  • a signal processing device including an avoidance processing unit that avoids a product-sum calculation process for the input data when the input data is less than the predetermined threshold value.
  • the input data includes type 1 input data and type 2 input data.
  • the threshold value determination processing unit performs the determination on the type 1 input data, and then performs the determination.
  • the signal processing device wherein the avoidance processing unit avoids the product-sum operation processing for the type 1 input data when the type 1 input data is less than the predetermined threshold value.
  • the type 2 input data is weight data that is information on weights to be multiplied by the type 1 input data.
  • the threshold value determination processing unit is provided for each of the plurality of product / sum calculators.
  • the avoidance processing unit changes the input data input to the product-sum calculation unit when the input data is less than the predetermined threshold, so that the product-sum of the input data is set to be less than the predetermined threshold.
  • the signal processing device which avoids arithmetic processing.
  • a product-sum operation control unit that manages input data and output data of the product-sum operation process is provided.
  • the signal processing device according to (5) above, wherein the avoidance processing unit notifies the product-sum calculation control unit of information for identifying the input data in which the product-sum calculation processing has been avoided.
  • the signal processing device according to any one of (1) to (6) above, wherein the avoidance processing unit is provided for each product-sum calculation unit.
  • the avoidance processing unit avoids the product-sum calculation process for the input data that is less than the predetermined threshold value and outputs a zero value as the processing result of the product-sum calculation process. ..
  • the input data includes type 1 input data and type 2 input data.
  • the avoidance processing unit When the type 1 input data is less than the first threshold value, the type 1 input data input to the product-sum calculator is changed and the changed type 1 input data is specified.
  • the signal processing device according to (6) above, which notifies the information to the product-sum calculation control unit.
  • the avoidance processing unit changes the type 2 input data input to the product-sum calculator and also changes the type 2 input.
  • the above (9) that changes the type 1 input data corresponding to the data and notifies the product-sum calculation control unit of the information for specifying the changed type 1 input data and the type 2 input data. ).
  • the signal processing device When the type 1 input data is less than the first threshold value, the type 1 input data input to the product-sum calculator is changed and the changed type 1 input data is specified.
  • the signal processing device according to (6) above, which notifies the information to the product-sum calculation control unit.
  • the avoidance processing unit changes the type 2 input data input to the product-sum calculator and also
  • the product-sum calculation control unit manages the product-sum calculation result of the first-class input data and the second-class input data, and compensates for the avoided product-sum calculation result with a zero value.
  • Signal processing device (12) A pixel array unit in which photoelectric conversion elements are arranged in a one-dimensional or two-dimensional array, and A signal processing unit for inputting input data based on the output signal of the pixel array unit is provided.
  • the signal processing unit A product-sum calculator that is arranged in a one-dimensional or two-dimensional array and is capable of product-sum operations in a neural network.
  • a threshold value determination processing unit that determines whether or not the input data used in the calculation by the product-sum calculation unit is less than a predetermined threshold value.
  • An image pickup apparatus including an avoidance processing unit that avoids a product-sum calculation process for the input data when the input data is less than the predetermined threshold value.
  • the image pickup apparatus according to (12) above wherein the pixel array unit and the signal processing unit are integrally formed.
  • the signal processing unit inputs feature data extracted based on the output signal of the pixel array unit as the input data.
  • Imaging device (signal processing device) 20, 20D, 20E MAC (multiply-accumulate calculator) 20-1, 20-2, 20-3, 20-4 MAC (multiply-accumulate calculator) 20-5, 20-6, 20-7, 20-8 MAC (multiply-accumulate calculator) 20-9, 20-10, 20-11, 20-12 MAC (multiply-accumulate calculator) 21,21D Avoidance processing unit (threshold value determination processing unit) 21a First avoidance processing unit (threshold determination processing unit) 21b Second avoidance processing unit (threshold value determination processing unit) 25 Multiply-accumulate operation control unit

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Mathematical Physics (AREA)
  • Computing Systems (AREA)
  • Biophysics (AREA)
  • General Engineering & Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Biomedical Technology (AREA)
  • Data Mining & Analysis (AREA)
  • Signal Processing (AREA)
  • Multimedia (AREA)
  • Software Systems (AREA)
  • Mathematical Analysis (AREA)
  • Artificial Intelligence (AREA)
  • Computational Linguistics (AREA)
  • Pure & Applied Mathematics (AREA)
  • Evolutionary Computation (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Mathematical Optimization (AREA)
  • Computational Mathematics (AREA)
  • Neurology (AREA)
  • Algebra (AREA)
  • Databases & Information Systems (AREA)
  • Image Processing (AREA)
PCT/JP2021/034103 2020-09-30 2021-09-16 信号処理装置、撮像装置、信号処理方法 Ceased WO2022070947A1 (ja)

Priority Applications (4)

Application Number Priority Date Filing Date Title
JP2022553812A JPWO2022070947A1 (https=) 2020-09-30 2021-09-16
CN202180065035.1A CN116210228A (zh) 2020-09-30 2021-09-16 信号处理装置、成像装置和信号处理方法
DE112021005190.3T DE112021005190T5 (de) 2020-09-30 2021-09-16 Signalverarbeitungsvorrichtung, bildaufnahmevorrichtung und signalverarbeitungsverfahren
US18/042,395 US20230333816A1 (en) 2020-09-30 2021-09-16 Signal processing device, imaging device, and signal processing method

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2020166280 2020-09-30
JP2020-166280 2020-09-30

Publications (1)

Publication Number Publication Date
WO2022070947A1 true WO2022070947A1 (ja) 2022-04-07

Family

ID=80950317

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2021/034103 Ceased WO2022070947A1 (ja) 2020-09-30 2021-09-16 信号処理装置、撮像装置、信号処理方法

Country Status (5)

Country Link
US (1) US20230333816A1 (https=)
JP (1) JPWO2022070947A1 (https=)
CN (1) CN116210228A (https=)
DE (1) DE112021005190T5 (https=)
WO (1) WO2022070947A1 (https=)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2024162015A1 (ja) * 2023-02-03 2024-08-08 ソニーセミコンダクタソリューションズ株式会社 撮像装置、データ処理方法、及び、記録媒体
WO2024236748A1 (ja) * 2023-05-17 2024-11-21 三菱電機株式会社 異常判定装置、および、異常判定方法

Families Citing this family (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20220150319A (ko) * 2020-03-06 2022-11-10 가부시키가이샤 한도오따이 에네루기 켄큐쇼 촬상 장치 및 전자 기기
US20220253692A1 (en) * 2021-02-05 2022-08-11 Samsung Electronics Co., Ltd. Method and apparatus of operating a neural network
US11979674B2 (en) * 2022-09-08 2024-05-07 Micron Technology, Inc. Image enhancement using integrated circuit devices having analog inference capability
US12266184B2 (en) * 2022-09-08 2025-04-01 Micron Technology, Inc. Surveillance cameras implemented using integrated circuit devices having analog inference capability
US12538048B2 (en) * 2022-09-08 2026-01-27 Micron Technology, Inc. Image sensor with analog inference capability
US12437810B2 (en) * 2022-11-29 2025-10-07 Micron Technology, Inc. Memory device performing multiplication using logical states of memory cells
US20240304255A1 (en) * 2023-03-09 2024-09-12 Micron Technology, Inc. Memory device for multiplication using memory cells with different thresholds based on bit significance

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2005346472A (ja) * 2004-06-03 2005-12-15 Canon Inc 情報処理方法、情報処理装置、撮像装置
US20180285715A1 (en) * 2017-03-28 2018-10-04 Samsung Electronics Co., Ltd. Convolutional neural network (cnn) processing method and apparatus
CN111669527A (zh) * 2020-07-01 2020-09-15 浙江大学 一种cmos图像传感器内的卷积运算架构

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
GB2474901B (en) * 2009-10-30 2015-01-07 Advanced Risc Mach Ltd Apparatus and method for performing multiply-accumulate operations
US10360163B2 (en) 2016-10-27 2019-07-23 Google Llc Exploiting input data sparsity in neural network compute units
CN107622305A (zh) * 2017-08-24 2018-01-23 中国科学院计算技术研究所 用于神经网络的处理器和处理方法
US10943652B2 (en) * 2018-05-22 2021-03-09 The Regents Of The University Of Michigan Memory processing unit
US11663001B2 (en) * 2018-11-19 2023-05-30 Advanced Micro Devices, Inc. Family of lossy sparse load SIMD instructions

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2005346472A (ja) * 2004-06-03 2005-12-15 Canon Inc 情報処理方法、情報処理装置、撮像装置
US20180285715A1 (en) * 2017-03-28 2018-10-04 Samsung Electronics Co., Ltd. Convolutional neural network (cnn) processing method and apparatus
CN111669527A (zh) * 2020-07-01 2020-09-15 浙江大学 一种cmos图像传感器内的卷积运算架构

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
YASUHIRO NAKAHARA , JUNTARO CHIKA , TAIKI AMAGASAKI, KEN ZHAO, MASAHIRO IIDA: "DNN accelerator for AI edge computing", IEICE TECHNICAL REPORT; RECONF, vol. 119, no. 287 (RECONF2019-38), 7 November 2019 (2019-11-07), pages 15 - 20, XP009535627 *

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2024162015A1 (ja) * 2023-02-03 2024-08-08 ソニーセミコンダクタソリューションズ株式会社 撮像装置、データ処理方法、及び、記録媒体
WO2024236748A1 (ja) * 2023-05-17 2024-11-21 三菱電機株式会社 異常判定装置、および、異常判定方法
JPWO2024236748A1 (https=) * 2023-05-17 2024-11-21
JP7696532B2 (ja) 2023-05-17 2025-06-20 三菱電機モビリティ株式会社 異常判定装置、および、異常判定方法

Also Published As

Publication number Publication date
JPWO2022070947A1 (https=) 2022-04-07
CN116210228A (zh) 2023-06-02
DE112021005190T5 (de) 2023-09-14
US20230333816A1 (en) 2023-10-19

Similar Documents

Publication Publication Date Title
WO2022070947A1 (ja) 信号処理装置、撮像装置、信号処理方法
US12190226B2 (en) Method for accelerating operations and accelerator apparatus
US11467969B2 (en) Accelerator comprising input and output controllers for feeding back intermediate data between processing elements via cache module
US10210419B2 (en) Convolution operation apparatus
JP2024096786A (ja) ニューラルネットワーク計算ユニットにおける入力データのスパース性の活用
Li et al. A high performance FPGA-based accelerator for large-scale convolutional neural networks
JP2020074099A (ja) 処理装置と処理方法
JP2010134697A (ja) コンボリューション演算回路、階層的コンボリューション演算回路及び物体認識装置
EP3633559B1 (en) Information processing apparatus, information processing method, non-transitory computer-readable storage medium
JP2009116400A (ja) 情報処理装置および情報処理方法
JP6195342B2 (ja) 情報処理装置およびメモリアクセス制御方法
JP5335356B2 (ja) 信号処理装置、信号処理方法、及び撮像装置
JP2021515339A (ja) 機械知覚および高密度アルゴリズム集積回路
US7298898B2 (en) System and method for image segmentation
KR102557572B1 (ko) 인공 신경망 장치 및 그 동작 방법
US11704546B2 (en) Operation processing apparatus that calculates addresses of feature planes in layers of a neutral network and operation processing method
JP7493380B2 (ja) 機械学習システム、並びに、機械学習システムを構成する方法、コンピュータプログラム及び装置
Liu et al. Direct servo control from in-sensor cnn inference with a pixel processor array
US12475542B2 (en) Image processing system and image processing method
JP5099641B2 (ja) 飛跡認識装置および飛跡認識方法
US20100128989A1 (en) Correlation processing apparatus and medium readable by correlation processing apparatus
JP7648352B2 (ja) データ処理方法、データ処理装置、データ処理システムおよびデータ処理プログラム
JP7348805B2 (ja) 認識装置、認識方法
Furuta et al. An efficient implementation of fpga-based object detection using multi-scale attention
JP4621944B2 (ja) 画像フィルタ装置、方法およびコンピュータプログラム

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21875249

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2022553812

Country of ref document: JP

Kind code of ref document: A

WWE Wipo information: entry into national phase

Ref document number: 112021005190

Country of ref document: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21875249

Country of ref document: EP

Kind code of ref document: A1