US12536420B2 - Low power generative adversarial network accelerator and mixed-signal time-domain MAC array - Google Patents

Low power generative adversarial network accelerator and mixed-signal time-domain MAC array

Info

Publication number
US12536420B2
US12536420B2 US17/347,017 US202117347017A US12536420B2 US 12536420 B2 US12536420 B2 US 12536420B2 US 202117347017 A US202117347017 A US 202117347017A US 12536420 B2 US12536420 B2 US 12536420B2
Authority
US
United States
Prior art keywords
time
domain
edge device
training
tmu
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active, expires
Application number
US17/347,017
Other versions
US20210390380A1 (en
Inventor
Jie Gu
Zhengyu Chen
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Northwestern University
Original Assignee
Northwestern University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Northwestern University filed Critical Northwestern University
Priority to US17/347,017 priority Critical patent/US12536420B2/en
Assigned to NORTHWESTERN UNIVERSITY reassignment NORTHWESTERN UNIVERSITY ASSIGNMENT OF ASSIGNOR'S INTEREST Assignors: CHEN, ZHENGYU, GU, JIE
Publication of US20210390380A1 publication Critical patent/US20210390380A1/en
Assigned to NATIONAL SCIENCE FOUNDATION reassignment NATIONAL SCIENCE FOUNDATION CONFIRMATORY LICENSE Assignors: NORTHWESTERN UNIVERSITY
Application granted granted Critical
Publication of US12536420B2 publication Critical patent/US12536420B2/en
Active legal-status Critical Current
Adjusted expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/084—Backpropagation, e.g. using gradient descent
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/06—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons
    • G06N3/063—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons using electronic means
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0464—Convolutional networks [CNN, ConvNet]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/047—Probabilistic or stochastic networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0475—Generative networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0495—Quantised networks; Sparse networks; Compressed networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/088—Non-supervised learning, e.g. competitive learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/094—Adversarial learning

Definitions

  • the present disclosure generally relates to machine learning, and more specifically relates to low power generative adversarial network accelerators and mixed-signal time-domain MAC arrays.
  • a conventional generative adversarial network In deep learning applications, a conventional generative adversarial network (GAN) is often implemented with a high end central processing unit (CPU) or a high end graphics processing unit (GPU).
  • CPU central processing unit
  • GPU high end graphics processing unit
  • the high end CPU or GPU e.g., in desktop environments
  • GAN generative adversarial network
  • a system includes a memory and a training management unit (TMU) in communication with the memory.
  • TMU training management unit
  • the TMU is configured to manage a training sequence.
  • the system includes a time-domain multiplication-accumulation (TDMAC) unit in communication with the TMU, wherein the TDMAC unit is configured to perform time-domain multiplier operations and time-domain accumulator operations.
  • TDMAC time-domain multiplication-accumulation
  • an edge device includes a memory and a mixed-signal generative adversarial network (GAN) accelerator in communication with the memory.
  • the GAN accelerator is configured to perform mixed-signal time-domain training.
  • FIGS. 1 A- 1 D illustrate an example GAN algorithm and an example hardware implementation.
  • FIG. 2 illustrate an example GAN training sequence and an example ASIC TMU core design.
  • FIGS. 3 A- 3 F illustrate an example top-level architecture diagram with an example MAC array and an example TD-Accumulator, an example TD ReLU circuit, and an example TD MAC unit.
  • FIGS. 4 A- 4 E illustrate an example TD sub-vth multiplier and linearization implementation.
  • FIGS. 5 A- 5 G illustrate example measurement results.
  • FIGS. 6 A- 6 D illustrate an example die photo and comparison table.
  • Conventional GAN is rendered as one of the most interesting and challenging applications in deep learning space. Despite its broad real-time applications in gaming, authentication, and VR, for example, there is a lack of a dedicated low power GAN accelerator due to the tremendous challenges on resource-limited edge devices.
  • conventional GAN is extremely difficult to train due to model collapses from unbalanced models and high sensitivity to hyper-parameters.
  • conventional GAN involves two deep neural networks (DNNs) with complex training sequences, e.g. 41 different training stages as in this work.
  • DNNs deep neural networks
  • the typical floating-point training and complex calculation e.g. batch normalization and optimizers, are very expensive for a resource-limited edge device.
  • the disclosed systems and methods through significant architecture improvement and hardware adaptation, provide a mixed-signal GAN accelerator with 8-bit resolution for cost-effective implementation on edge device.
  • Certain aspects of the disclosed technology include: (1) for the first time, a complete GAN training core implemented on an 8-bit low-power application-specific integrated circuit (ASIC) chip consuming only 39 mW; (2) an efficient subthreshold time-domain (TD) multiplier designed with significant area saving compared to digital design; (3) On-chip training performed in mixed-signal TD for the first time.
  • the disclosed systems and methods eliminated 94% overhead from domain conversion, leading to the state-of-art throughput for a mixed-signal based accelerator which normally suffers from slow operation speed.
  • the disclosed systems and methods provide, at algorithm level, an adaptive training strategy and hardware simplification which makes the training process hardware friendly for low power edge devices.
  • the disclosed systems and methods provide, at hardware level, an ASIC training engine that supervise the training sequence. Such aspects, use very low power compared with previous conventional dependence on CPU to do the job.
  • the disclosed systems and methods provide, at circuit level, a time-domain multiplication-accumulation (MAC) circuit which uses time-domain multiplier and time-domain accumulator to save the cost and power compared with digital counterpart.
  • MAC time-domain multiplication-accumulation
  • the disclosed systems and methods provide GAN that consumes 39 mW and finishes tasks much faster.
  • FIGS. 1 A- 1 D show the implemented GAN architecture 8 with model compression and hardware adaptation techniques used in this work.
  • GAN deep convolutional GAN
  • model balancing and adaptive training were utilized to enable 8-bit training versus conventional floating-point training, leading to a 5 ⁇ reduction in hardware cost;
  • the challenging and memory consuming operations of batch normalization were simplified by disabling low-impact runtime operations, rendering a 77% removal of the associated operations;
  • the expensive ADAM optimizer was replaced by a succinct momentum stochastic gradient descent optimizer suitable for integer implementation with an 11 ⁇ reduction of the optimizer's computation;
  • the number of layers and channels were further minimized to reduce the computation load by 6 ⁇ to 9 ⁇ .
  • a 6 ⁇ reduction of training complexity, a 6.5 ⁇ hardware cost reduction, and an 11 ⁇ reduction of on-chip memory were achieved through the algorithm simplification with about a 3% loss of accuracy.
  • FIG. 2 shows the training sequence 10 .
  • Each training iteration consists of 7 unique phases (e.g. forward prop., loss cal.) with 5 phases for the generator 12 and 4 phases for the discriminator 14 .
  • Each phase also contains 4 to 6 sub-tasks 15 (e.g. Conv, FC, pooling, etc.).
  • an adaptive training 16 and model strength control scheme 18 was implemented which ceases the training of discriminator 14 if its strength is too high and adaptively increases the magnitude of the gradients during back propagation.
  • the training sequence 10 is managed by an ASIC training management unit (TMU) 20 as shown in FIG. 2 .
  • TMU ASIC training management unit
  • FIGS. 3 A- 3 F show the test chip architecture diagram 22 including the TMU 20 , a 10 ⁇ 10 time-domain (TD) MAC matrix 24 , static random-access memory (SRAM) modules 26 and supporting blocks. All the MAC operations of convolutional neural network (CNN) 28 (shown in FIG. 2 ) and Transpose-CNN (T-CNN) 30 are performed by a TD MAC matrix 24 to improve area and energy efficiency.
  • the time pulses generated from digital-to-time converters (DTC) 32 are processed by the subsequent multiplication-accumulation circuit (MAC) 34 and activation unit 36 all in time domain and are finally converted back into digital domain using time-to-digital converters (TDC) 38 .
  • DTC digital-to-time converters
  • MAC multiplication-accumulation circuit
  • TDC time-to-digital converters
  • a special 16 b time-pulse based time-domain accumulator (TD-ACC) 40 is designed using four 4-b ring-based time accumulators with carry propagation to realize accumulation efficiently.
  • the TDC 38 is only activated once every 25 MAC operations, removing 94% of the time and power overhead from the expensive TDC operations. Pushing all operations in time domain significantly reduces the cross-domain data conversion, rendering a 160 ⁇ speed-up in MAC operation compared with previous counter-based TD designs.
  • the 8-b TD multiplication is partitioned into four 4-bit multiplications to improve the computation accuracy and speed.
  • FIGS. 4 A- 4 E show the detailed circuit design featuring a subthreshold (sub-vth) TD multiplier (TD-MUL) 42 and a DTC-based linearization technique 44 .
  • the TD-MUL 42 takes input time pulses and generates output pulses of the multiplication results.
  • the current starving PMOS transistor is pre-biased at subthreshold region and generates a delay equals to the multiplication results through charge accumulation at the gate with logarithmic addition, i.e. a multiplication is addition in log domain.
  • the implemented sub-vth multiplier renders a 4.3 ⁇ reduction of area.
  • FIGS. 5 A- 5 G show the measured linearity from both the TD-MUL 42 and TD-ACC 40 .
  • For the multiplier although up to 4% error is seen in the result, most of the error is just a small scaling factor shift. Less than 1 b error is observed in the TD-ACC design.
  • the accuracy of the generated images with conditional GAN from 3 databases shows less than 1% error compared to the ideal integer 8-bit training on CPU and 3% compared with ideal floating-point training (1.6% comes from quantization loss and the rest from process variation of TD circuit).
  • the chip is verified with supply voltages down to 0.7V with up to 5% degradation of accuracy compared with ideal GAN operation. Interestingly, a “self-healing” feature of GAN is observed, recovering most of the error loss from on-chip variations compared with no on-chip training.
  • This intrinsic resiliency presents a merit for training empowered design using mixed-signal circuits.
  • the chip consumes 39 mW power with TD-MAC at 90 MHz.
  • the total training time of MNIST database takes 4.5 minutes which is 82 ⁇ less than a high-performance CPU (2.6 GHz Intel i7 Quad-core with a power of 197 W).
  • the die photo 46 and comparison table 48 with prior analog mixed-signal (AMS) designs are shown in FIGS. 6 A- 6 D . As most of existing AMS designs suffer from low throughput, this work achieves the highest throughput of 18 ⁇ 5400 ⁇ with similar efficiency. In addition, a low-cost 8-bit on-chip training was realized for AMS design on the very challenging GAN

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • General Health & Medical Sciences (AREA)
  • General Physics & Mathematics (AREA)
  • Evolutionary Computation (AREA)
  • Computational Linguistics (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Artificial Intelligence (AREA)
  • Neurology (AREA)
  • Probability & Statistics with Applications (AREA)
  • Complex Calculations (AREA)
  • Power Sources (AREA)

Abstract

Systems and methods for a low-cost mixed-signal time-domain accelerator for generative adversarial network (GAN) are provided. In one aspect, a system includes a memory and a training management unit (TMU) in communication with the memory. The TMU is configured to manage a training sequence. The system includes a time-domain multiplication-accumulation (TDMAC) unit in communication with the TMU, wherein the TDMAC unit is configured to perform time-domain multiplier operations and time-domain accumulator operations.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS
The present application claims the benefit of priority under 35 U.S.C. § 119 from U.S. Provisional Patent Application Ser. No. 63/039,100 entitled “A Low Power Generative Adversarial Network Accelerator and Mixed-signal Time-domain MAC Array,” filed on Jun. 15, 2020, the disclosure of which is hereby incorporated by reference in its entirety for all purposes.
STATEMENT OF FEDERALLY FUNDED RESEARCH OR SPONSORSHIP
This invention was made with government support under grant number 1846424 awarded by the National Science Foundation. The government has certain rights in the invention.
TECHNICAL FIELD
The present disclosure generally relates to machine learning, and more specifically relates to low power generative adversarial network accelerators and mixed-signal time-domain MAC arrays.
BACKGROUND
In deep learning applications, a conventional generative adversarial network (GAN) is often implemented with a high end central processing unit (CPU) or a high end graphics processing unit (GPU). In such conventional implementations of the GAN, the high end CPU or GPU (e.g., in desktop environments) will typically use power in the order of Watts or hundreds of Watts.
SUMMARY
Systems and methods for a low-cost mixed-signal time-domain accelerator for generative adversarial network (GAN) are provided.
According to certain aspects of the present disclosure, a system is provided. The system includes a memory and a training management unit (TMU) in communication with the memory. The TMU is configured to manage a training sequence. The system includes a time-domain multiplication-accumulation (TDMAC) unit in communication with the TMU, wherein the TDMAC unit is configured to perform time-domain multiplier operations and time-domain accumulator operations.
According to certain other aspects of the present disclosure, an edge device is provided. The edge device includes a memory and a mixed-signal generative adversarial network (GAN) accelerator in communication with the memory. The GAN accelerator is configured to perform mixed-signal time-domain training.
It is understood that other configurations of the subject technology will become readily apparent to those skilled in the art from the following detailed description, wherein various configurations of the subject technology are shown and described by way of illustration. As will be realized, the subject technology is capable of other and different configurations and its several details are capable of modification in various other respects, all without departing from the scope of the subject technology. Accordingly, the drawings and detailed description are to be regarded as illustrative in nature and not as restrictive.
BRIEF DESCRIPTION OF DRAWINGS
The disclosure is better understood with reference to the following drawings and description. The elements in the figures are not necessarily to scale, emphasis instead being placed upon illustrating the principles of the disclosure. Moreover, in the figures, like-referenced numerals may designate to corresponding parts throughout the different views.
FIGS. 1A-1D illustrate an example GAN algorithm and an example hardware implementation.
FIG. 2 illustrate an example GAN training sequence and an example ASIC TMU core design.
FIGS. 3A-3F illustrate an example top-level architecture diagram with an example MAC array and an example TD-Accumulator, an example TD ReLU circuit, and an example TD MAC unit.
FIGS. 4A-4E illustrate an example TD sub-vth multiplier and linearization implementation.
FIGS. 5A-5G illustrate example measurement results.
FIGS. 6A-6D illustrate an example die photo and comparison table.
 DETAILED DESCRIPTION
The detailed description set forth below is intended as a description of various implementations and is not intended to represent the only implementations in which the subject technology may be practiced. As those skilled in the art would realize, the described implementations may be modified in various different ways, all without departing from the scope of the present disclosure. Accordingly, the drawings and description are to be regarded as illustrative in nature and not restrictive.
Conventional GAN is rendered as one of the most interesting and challenging applications in deep learning space. Despite its broad real-time applications in gaming, authentication, and VR, for example, there is a lack of a dedicated low power GAN accelerator due to the tremendous challenges on resource-limited edge devices. From the algorithm aspect, conventional GAN is extremely difficult to train due to model collapses from unbalanced models and high sensitivity to hyper-parameters. From the hardware aspect, conventional GAN involves two deep neural networks (DNNs) with complex training sequences, e.g. 41 different training stages as in this work. Moreover, the typical floating-point training and complex calculation, e.g. batch normalization and optimizers, are very expensive for a resource-limited edge device.
The disclosed systems and methods, through significant architecture improvement and hardware adaptation, provide a mixed-signal GAN accelerator with 8-bit resolution for cost-effective implementation on edge device. Certain aspects of the disclosed technology include: (1) for the first time, a complete GAN training core implemented on an 8-bit low-power application-specific integrated circuit (ASIC) chip consuming only 39 mW; (2) an efficient subthreshold time-domain (TD) multiplier designed with significant area saving compared to digital design; (3) On-chip training performed in mixed-signal TD for the first time. In certain aspects, the disclosed systems and methods eliminated 94% overhead from domain conversion, leading to the state-of-art throughput for a mixed-signal based accelerator which normally suffers from slow operation speed.
In certain aspects, the disclosed systems and methods provide, at algorithm level, an adaptive training strategy and hardware simplification which makes the training process hardware friendly for low power edge devices. In certain aspects, the disclosed systems and methods provide, at hardware level, an ASIC training engine that supervise the training sequence. Such aspects, use very low power compared with previous conventional dependence on CPU to do the job. In certain aspects, the disclosed systems and methods provide, at circuit level, a time-domain multiplication-accumulation (MAC) circuit which uses time-domain multiplier and time-domain accumulator to save the cost and power compared with digital counterpart. In certain aspects, the disclosed systems and methods provide GAN that consumes 39 mW and finishes tasks much faster.
FIGS. 1A-1D show the implemented GAN architecture 8 with model compression and hardware adaptation techniques used in this work. For fitting with a small chip budget on edge device, we targeted a low-budget architecture implementation of deep convolutional GAN (DCGAN) using greyscale image with a size of 28×28 pixels. The following techniques were specially developed: (1) model balancing and adaptive training were utilized to enable 8-bit training versus conventional floating-point training, leading to a 5× reduction in hardware cost; (2) The challenging and memory consuming operations of batch normalization were simplified by disabling low-impact runtime operations, rendering a 77% removal of the associated operations; (3) The expensive ADAM optimizer was replaced by a succinct momentum stochastic gradient descent optimizer suitable for integer implementation with an 11× reduction of the optimizer's computation; (4) The number of layers and channels were further minimized to reduce the computation load by 6× to 9×. Overall, a 6× reduction of training complexity, a 6.5× hardware cost reduction, and an 11× reduction of on-chip memory were achieved through the algorithm simplification with about a 3% loss of accuracy.
FIG. 2 shows the training sequence 10. Each training iteration consists of 7 unique phases (e.g. forward prop., loss cal.) with 5 phases for the generator 12 and 4 phases for the discriminator 14. Each phase also contains 4 to 6 sub-tasks 15 (e.g. Conv, FC, pooling, etc.). To avoid model collapsing, an adaptive training 16 and model strength control scheme 18 was implemented which ceases the training of discriminator 14 if its strength is too high and adaptively increases the magnitude of the gradients during back propagation. The training sequence 10 is managed by an ASIC training management unit (TMU) 20 as shown in FIG. 2 . A total of 41 training stages were implemented in the TMU as a finite state machine. Special operations such as pooling, sigmoid, data transpose etc. were handled by the dedicated hardware modules inside the TMU 20. Register files were used to store temporary weights and feature map outputs, bridging the throughput mismatch between SRAM and MAC arrays.
FIGS. 3A-3F show the test chip architecture diagram 22 including the TMU 20, a 10×10 time-domain (TD) MAC matrix 24, static random-access memory (SRAM) modules 26 and supporting blocks. All the MAC operations of convolutional neural network (CNN) 28 (shown in FIG. 2 ) and Transpose-CNN (T-CNN) 30 are performed by a TD MAC matrix 24 to improve area and energy efficiency. The time pulses generated from digital-to-time converters (DTC) 32 are processed by the subsequent multiplication-accumulation circuit (MAC) 34 and activation unit 36 all in time domain and are finally converted back into digital domain using time-to-digital converters (TDC) 38. A special 16 b time-pulse based time-domain accumulator (TD-ACC) 40 is designed using four 4-b ring-based time accumulators with carry propagation to realize accumulation efficiently. With the special TD-ACC 40, the TDC 38 is only activated once every 25 MAC operations, removing 94% of the time and power overhead from the expensive TDC operations. Pushing all operations in time domain significantly reduces the cross-domain data conversion, rendering a 160× speed-up in MAC operation compared with previous counter-based TD designs. The 8-b TD multiplication is partitioned into four 4-bit multiplications to improve the computation accuracy and speed.
FIGS. 4A-4E show the detailed circuit design featuring a subthreshold (sub-vth) TD multiplier (TD-MUL) 42 and a DTC-based linearization technique 44. The TD-MUL 42 takes input time pulses and generates output pulses of the multiplication results. As in FIGS. 4A-4E, the current starving PMOS transistor is pre-biased at subthreshold region and generates a delay equals to the multiplication results through charge accumulation at the gate with logarithmic addition, i.e. a multiplication is addition in log domain. Compared to the digital implementation, the implemented sub-vth multiplier renders a 4.3× reduction of area. However, as shown in simulation, significant nonlinearity is observed in sub-vth multiplication. The nonlinearity is compensated by a logarithmic encoding of DTC. As shown in both equation and the simulated waveforms in FIGS. 4A-4E, the linearization technique elegantly removes nonlinearity with negligible overhead. After the multiplication, the resulting time pulses are sent into TD-ACC 40 for accumulation of 25 cycles avoiding time-consuming digitalization. Simple TD ReLU function is also implemented at each CNN layer except the final layer which uses digital sigmoid function.
FIGS. 5A-5G show the measured linearity from both the TD-MUL 42 and TD-ACC 40. For the multiplier, although up to 4% error is seen in the result, most of the error is just a small scaling factor shift. Less than 1 b error is observed in the TD-ACC design. We trained the GAN with 3 databases, i.e. a digit-MNIST, a fashion, and an emoji database. The accuracy of the generated images with conditional GAN from 3 databases shows less than 1% error compared to the ideal integer 8-bit training on CPU and 3% compared with ideal floating-point training (1.6% comes from quantization loss and the rest from process variation of TD circuit). The chip is verified with supply voltages down to 0.7V with up to 5% degradation of accuracy compared with ideal GAN operation. Interestingly, a “self-healing” feature of GAN is observed, recovering most of the error loss from on-chip variations compared with no on-chip training. This intrinsic resiliency presents a merit for training empowered design using mixed-signal circuits. The chip consumes 39 mW power with TD-MAC at 90 MHz. The total training time of MNIST database takes 4.5 minutes which is 82× less than a high-performance CPU (2.6 GHz Intel i7 Quad-core with a power of 197 W). The die photo 46 and comparison table 48 with prior analog mixed-signal (AMS) designs are shown in FIGS. 6A-6D. As most of existing AMS designs suffer from low throughput, this work achieves the highest throughput of 18˜5400× with similar efficiency. In addition, a low-cost 8-bit on-chip training was realized for AMS design on the very challenging GAN operation.
Other systems, methods, features and advantages will be, or will become, apparent to one with skill in the art upon examination of the figures and detailed description. It is intended that all such additional systems, methods, features and advantages be included within this description, be within the scope of the disclosure, and be protected by the following claims.

Claims (9)

What is claimed is:
1. An edge device, comprising:
a memory; and
a mixed-signal generative adversarial network (GAN) accelerator in communication with the memory,
wherein the GAN accelerator is configured to perform mixed-signal time-domain training,
wherein the GAN accelerator comprises an application-specific integrated circuit (ASIC) training management unit (TMU) in communication with the memory, wherein the ASIC TMU is configured to perform a training sequence comprising an adaptive training scheme, and comprises a time-domain multiplication-accumulation (TDMAC) unit in communication with the ASIC TMU,
wherein the TDMAC unit is configured to perform multiplication-accumulation operations of a convolutional neural network and a transpose convolutional neural network.
2. The edge device of claim 1, wherein the GAN accelerator is implemented on an 8-bit low-power application-specific integrated circuit (ASIC) chip.
3. The edge device of claim 2, wherein the 8-bit low-power ASIC chip comprises power consumption of less than 39 mW.
4. The edge device of claim 1, wherein the TDMAC unit comprises a 16-bit time-pulse based time-domain accumulator (TD-ACC) configured to perform the time-domain accumulator operation.
5. The edge device of claim 4, wherein the 16-bit time-pulse based TD-ACC comprises four 4-bit ring-based time accumulators.
6. The edge device of claim 1, wherein the TDMAC unit comprises an 8-bit time-domain multiplier (TD-MUL) configured to perform the time-domain multiplier operations wherein the TD-MUL comprises a subthreshold TD-MUL.
7. The edge device of claim 6, wherein the TD-MUL comprises four 4-bit multipliers.
8. The edge device of claim 1, wherein the ASIC TMU is configured as a finite state machine comprising 41 training stages.
9. The edge device of claim 1, wherein the ASIC TMU comprises modules for performing pooling operations, sigmoid operations, and data transpose operations.
US17/347,017 2020-06-15 2021-06-14 Low power generative adversarial network accelerator and mixed-signal time-domain MAC array Active 2044-11-28 US12536420B2 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US17/347,017 US12536420B2 (en) 2020-06-15 2021-06-14 Low power generative adversarial network accelerator and mixed-signal time-domain MAC array

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US202063039100P 2020-06-15 2020-06-15
US17/347,017 US12536420B2 (en) 2020-06-15 2021-06-14 Low power generative adversarial network accelerator and mixed-signal time-domain MAC array

Publications (2)

Publication Number Publication Date
US20210390380A1 US20210390380A1 (en) 2021-12-16
US12536420B2 true US12536420B2 (en) 2026-01-27

Family

ID=78825518

Family Applications (1)

Application Number Title Priority Date Filing Date
US17/347,017 Active 2044-11-28 US12536420B2 (en) 2020-06-15 2021-06-14 Low power generative adversarial network accelerator and mixed-signal time-domain MAC array

Country Status (1)

Country Link
US (1) US12536420B2 (en)

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9820074B2 (en) * 2013-03-15 2017-11-14 Apple Inc. Memory management techniques and related systems for block-based convolution
US20190138267A1 (en) * 2017-11-06 2019-05-09 Google Llc Parsing electronic conversations for presentation in an alternative interface
US10387122B1 (en) * 2018-05-04 2019-08-20 Olsen Ip Reserve, Llc Residue number matrix multiplier
US20200311535A1 (en) * 2019-03-25 2020-10-01 Northeastern University Self-powered analog computing architecture with energy monitoring to enable machine-learning vision at the edge
US20210142467A1 (en) * 2019-11-13 2021-05-13 Software Ag Systems and methods of generating datasets for training neural networks

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9820074B2 (en) * 2013-03-15 2017-11-14 Apple Inc. Memory management techniques and related systems for block-based convolution
US20190138267A1 (en) * 2017-11-06 2019-05-09 Google Llc Parsing electronic conversations for presentation in an alternative interface
US10387122B1 (en) * 2018-05-04 2019-08-20 Olsen Ip Reserve, Llc Residue number matrix multiplier
US20200311535A1 (en) * 2019-03-25 2020-10-01 Northeastern University Self-powered analog computing architecture with energy monitoring to enable machine-learning vision at the edge
US20210142467A1 (en) * 2019-11-13 2021-05-13 Software Ag Systems and methods of generating datasets for training neural networks

Non-Patent Citations (36)

* Cited by examiner, † Cited by third party
Title
Amravati et al., "A 55 nm Time-Domain Mixed-Signal Neuromorphic Accelerator with Stochastic Synapses and Embedded Reinforcement Learning for Autonomous Micro-Robots", (2018) (Year: 2018). *
Cao, Ningyuan et al., "A 65nm 1.1-to-9.1TOPS/W Hybrid-Digital-Mixed-Signal Computing Platform for Accelerating Model-Based and Model-Free Swarm Robotics," IEEE International Solid-State Circuits Conference, Feb. 2019, 3 Pages.
Chen, Zhengyu et al., "A Scalable Pipelined Time-Domain DTW Engine for Time-Series Classification Using Multibit Time Flip-Flops with 140Giga-Cell-Updates/s Throughput," IEEE International Solid-State Circuits Conference, Feb. 2019, 3 Pages.
Du, "Time Domain Multiply and Accumulate Engine for Convolutional Neural Networks", Thesis, The Ohio State University 2020. (Year: 2020). *
Ekroot et al., "A GaAs 4-bit Adder-Accumulator Circuit for Direct Digital Synthesis", (1988) (Year: 1988). *
EMOJI database retrieved on Oct. 11, 2021 from <getemoji.com>.
Fashion database retrieved on Oct. 11, 2021, <www.kaggle.comzalando-researchfashionmnist>.
Kashinath et al., "High-Speed and Energy-Efficient MAC design using Vedic Multiplier and Carry Skip Adder", (Oct. 2017) (Year: 2017). *
Lee, Edward H. et al., "A 2.5GHz 7.7TOPS/W Switched-Capacitor Matrix Multiplier with Co-designed Local Memory in 40nm," IEEE International Solid-State Circuits Conference, Feb. 2016, 3 Pages.
Liu, Muqing et al., "A Scalable Time—based Integrate-and-Fire Neuromorphic Core with Brain—Inspired Leak and Local Lateral Inhibition Capabilities," IEEE International Custom Integrated Circuits Conference (CICC), 2017, 4 Pages.
Mao et al., "AdaLearner: An Adaptive Distributed Mobile Learning System for Neural Networks", (2017) (Year: 2017). *
Masadeh et al., "Input-Conscious Approximate Multiply-Accumulate (MAC) Unit for Energy-Efficiency", (Oct. 2019) (Year: 2019). *
Radford, Alec et al., "Unsupervised Representation Learning With Deep Convolutional Generative Adversarial Networks," arXiv preprint arXiv:1511.06434v2, Jan. 2016, 16 Pages.
Reynders et al., "Variation-Resilient Sub-Threshold Circuit Solutions for Ultra-Low-Power Digital Signal Processors with 10MHz Clock Frequency", (2012) (Year: 2012). *
Roohi et al., "ApGAN: Approximate GAN for Robust Low Energy Learning From Imprecise Components", (Mar. 2020) (Year: 2020). *
Sayal, Aseem et al., "All-Digital Time-Domain CNN Engine Using Bidirectional Memory Delay Lines for Energy-Efficient Edge Computing," IEEE International Solid-State Circuits Conference, Feb. 2019, 3 Pages.
Yin et al., "A 2.6 TOPS/W 16-bit Fixed-Point Convolutional Neural Network Learning Processor in 65-nm CMOS", (Nov. 20, 2019) (Year: 2019). *
Yoshioka, Kentaro et al., "PhaseMAC: A 14 TOPS/W 8bit GRO based Phase Domain MAC Circuit for In-Sensor-Computed Deep Learning Accelerators," VLSI, 2018, 2 Pages.
Amravati et al., "A 55 nm Time-Domain Mixed-Signal Neuromorphic Accelerator with Stochastic Synapses and Embedded Reinforcement Learning for Autonomous Micro-Robots", (2018) (Year: 2018). *
Cao, Ningyuan et al., "A 65nm 1.1-to-9.1TOPS/W Hybrid-Digital-Mixed-Signal Computing Platform for Accelerating Model-Based and Model-Free Swarm Robotics," IEEE International Solid-State Circuits Conference, Feb. 2019, 3 Pages.
Chen, Zhengyu et al., "A Scalable Pipelined Time-Domain DTW Engine for Time-Series Classification Using Multibit Time Flip-Flops with 140Giga-Cell-Updates/s Throughput," IEEE International Solid-State Circuits Conference, Feb. 2019, 3 Pages.
Du, "Time Domain Multiply and Accumulate Engine for Convolutional Neural Networks", Thesis, The Ohio State University 2020. (Year: 2020). *
Ekroot et al., "A GaAs 4-bit Adder-Accumulator Circuit for Direct Digital Synthesis", (1988) (Year: 1988). *
EMOJI database retrieved on Oct. 11, 2021 from <getemoji.com>.
Fashion database retrieved on Oct. 11, 2021, <www.kaggle.comzalando-researchfashionmnist>.
Kashinath et al., "High-Speed and Energy-Efficient MAC design using Vedic Multiplier and Carry Skip Adder", (Oct. 2017) (Year: 2017). *
Lee, Edward H. et al., "A 2.5GHz 7.7TOPS/W Switched-Capacitor Matrix Multiplier with Co-designed Local Memory in 40nm," IEEE International Solid-State Circuits Conference, Feb. 2016, 3 Pages.
Liu, Muqing et al., "A Scalable Time—based Integrate-and-Fire Neuromorphic Core with Brain—Inspired Leak and Local Lateral Inhibition Capabilities," IEEE International Custom Integrated Circuits Conference (CICC), 2017, 4 Pages.
Mao et al., "AdaLearner: An Adaptive Distributed Mobile Learning System for Neural Networks", (2017) (Year: 2017). *
Masadeh et al., "Input-Conscious Approximate Multiply-Accumulate (MAC) Unit for Energy-Efficiency", (Oct. 2019) (Year: 2019). *
Radford, Alec et al., "Unsupervised Representation Learning With Deep Convolutional Generative Adversarial Networks," arXiv preprint arXiv:1511.06434v2, Jan. 2016, 16 Pages.
Reynders et al., "Variation-Resilient Sub-Threshold Circuit Solutions for Ultra-Low-Power Digital Signal Processors with 10MHz Clock Frequency", (2012) (Year: 2012). *
Roohi et al., "ApGAN: Approximate GAN for Robust Low Energy Learning From Imprecise Components", (Mar. 2020) (Year: 2020). *
Sayal, Aseem et al., "All-Digital Time-Domain CNN Engine Using Bidirectional Memory Delay Lines for Energy-Efficient Edge Computing," IEEE International Solid-State Circuits Conference, Feb. 2019, 3 Pages.
Yin et al., "A 2.6 TOPS/W 16-bit Fixed-Point Convolutional Neural Network Learning Processor in 65-nm CMOS", (Nov. 20, 2019) (Year: 2019). *
Yoshioka, Kentaro et al., "PhaseMAC: A 14 TOPS/W 8bit GRO based Phase Domain MAC Circuit for In-Sensor-Computed Deep Learning Accelerators," VLSI, 2018, 2 Pages.

Also Published As

Publication number Publication date
US20210390380A1 (en) 2021-12-16

Similar Documents

Publication Publication Date Title
Wang et al. Low power convolutional neural networks on a chip
US20210216280A1 (en) Switched capacitor vector-matrix multiplier
WO2021050590A1 (en) Systems and methods for modifying neural networks for binary processing applications
Huynh Deep neural network accelerator based on FPGA
Chen et al. A mixed-signal time-domain generative adversarial network accelerator with efficient subthreshold time multiplier and mixed-signal on-chip training for low power edge devices
CN111931938B (en) System and method for accelerating inference operation based on structured sparse recurrent neural network
CN108446761A (en) A kind of neural network accelerator and data processing method
Moon et al. Memory-reduced network stacking for edge-level CNN architecture with structured weight pruning
Dorrance et al. An energy-efficient bayesian neural network accelerator with cim and a time-interleaved hadamard digital grng using 22-nm finfet
Wu et al. A 3.89-GOPS/mW scalable recurrent neural network processor with improved efficiency on memory and computation
US12536420B2 (en) Low power generative adversarial network accelerator and mixed-signal time-domain MAC array
Chen et al. FPGA implementation of neural network accelerator for pulse information extraction in high energy physics
Zheng et al. A high energy-efficiency FPGA-based LSTM accelerator architecture design by structured pruning and normalized linear quantization
Ando et al. Dither nn: An accurate neural network with dithering for low bit-precision hardware
Niu et al. Spec2: Spectral sparse cnn accelerator on fpgas
Byun et al. Hardware-friendly hessian-driven row-wise quantization and FPGA acceleration for transformer-based models
Zhang et al. An energy-efficient mixed-signal parallel multiply-accumulate (MAC) engine based on stochastic computing
Sharma et al. Hardware accelerator for object detection using tiny YOLO-v3
Ji et al. Compacc: Efficient hardware realization for processing compressed neural networks using accumulator arrays
Lahdhiri et al. DNNZip: selective layers compression technique in deep neural network accelerators
Zhao et al. Fpga-based cnn quantization method and hardware acceleration module
Jo et al. Bit-serial multiplier based neural processing element with approximate adder tree
Kumar et al. Complex multiplier: implementation using efficient algorithms for signal processing application
Li et al. FPGA implementation of LSTM based on automatic speech recognition
EP4505286B1 (en) Multi-bit analog multiply-accumulate operations with memory crossbar arrays

Legal Events

Date Code Title Description
AS Assignment

Owner name: NORTHWESTERN UNIVERSITY, ILLINOIS

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:GU, JIE;CHEN, ZHENGYU;SIGNING DATES FROM 20200628 TO 20200629;REEL/FRAME:056591/0561

FEPP Fee payment procedure

Free format text: ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: SMALL ENTITY

FEPP Fee payment procedure

Free format text: ENTITY STATUS SET TO SMALL (ORIGINAL EVENT CODE: SMAL); ENTITY STATUS OF PATENT OWNER: SMALL ENTITY

STPP Information on status: patent application and granting procedure in general

Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION

STPP Information on status: patent application and granting procedure in general

Free format text: NON FINAL ACTION MAILED

STPP Information on status: patent application and granting procedure in general

Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER

AS Assignment

Owner name: NATIONAL SCIENCE FOUNDATION, VIRGINIA

Free format text: CONFIRMATORY LICENSE;ASSIGNOR:NORTHWESTERN UNIVERSITY;REEL/FRAME:070059/0328

Effective date: 20220529

STPP Information on status: patent application and granting procedure in general

Free format text: FINAL REJECTION MAILED

STPP Information on status: patent application and granting procedure in general

Free format text: RESPONSE AFTER FINAL ACTION FORWARDED TO EXAMINER

STPP Information on status: patent application and granting procedure in general

Free format text: ALLOWED -- NOTICE OF ALLOWANCE NOT YET MAILED

STPP Information on status: patent application and granting procedure in general

Free format text: NOTICE OF ALLOWANCE MAILED -- APPLICATION RECEIVED IN OFFICE OF PUBLICATIONS

STPP Information on status: patent application and granting procedure in general

Free format text: PUBLICATIONS -- ISSUE FEE PAYMENT RECEIVED

STPP Information on status: patent application and granting procedure in general

Free format text: PUBLICATIONS -- ISSUE FEE PAYMENT VERIFIED

STCF Information on status: patent grant

Free format text: PATENTED CASE