CN109116967B

CN109116967B - Neural network predictor based on time multiplexing and electronic equipment suitable for neural network predictor

Info

Publication number: CN109116967B
Application number: CN201810883953.5A
Authority: CN
Inventors: 马恺声
Original assignee: Tsinghua University
Current assignee: Tsinghua University
Priority date: 2018-08-06
Filing date: 2018-08-06
Publication date: 2020-11-24
Anticipated expiration: 2038-08-06
Also published as: CN109116967A

Abstract

The application provides a neural network predictor, a neural network chip, a nonvolatile processor, an energy management system and applicable electronic equipment and devices thereof based on time multiplexing, wherein a neural network module of the neural network predictor performs prediction calculation based on at least one of received power supply information, power storage information and power failure information at least one moment, and outputs at least one instruction or/and service quality prediction information of a data bit width instruction, a starting instruction or a write strategy instruction; and the time sequence control module is used for controlling the time sequence of the prediction calculation of the neural network module. The neural network predictor realizes various prediction calculations in different time periods through a hardware architecture of the neural network predictor, thereby achieving the purpose of saving hardware cost and area.

Description

Neural network predictor based on time multiplexing and electronic equipment suitable for neural network predictor

Technical Field

The present disclosure relates to the field of energy management technologies, and in particular, to a neural network predictor, a neural network chip, a non-volatile processor, an energy management system, and an electronic device and an apparatus suitable for the same based on time multiplexing.

Background

The rapid development of internet of things devices and wireless communication technologies enables, for example, wearable devices or implantable devices with high performance and small size to be developed rapidly, the demand for high performance means the increase of energy consumption of a system, the development speed of the current battery is far behind the increase of the demand for energy consumption, and the problems of large volume, heavy weight and high maintenance cost still exist in battery power supply. For this reason, the wearable device or the implantable device usually realizes self-powered by collecting external energy, however, self-powered has the defects of limited energy, drastic change and difficult prediction, and therefore, the nodes of the internet of things need to optimize energy utilization efficiency by reasonably storing and utilizing the limited collected energy and reasonably managing energy according to the energy requirements of different loads.

In the nodes of the internet of things, besides signal processing and control operations inside the processor, there are operations of data communication and information interaction between the processor and peripheral devices, such as reading sensor information from a sensor back into the processor, writing and reading data in a memory chip, and transmitting and receiving data through a radio frequency unit. These result in a high demand for power from the processor. In a self-powered system, each atomic operation must be guaranteed to be completed with sufficient energy. Therefore, the energy management devices of the system need to be able to provide such support. Therefore, scientific and reasonable energy management is very important.

Disclosure of Invention

In view of the above-mentioned disadvantages of the related art, it is an object of the present invention to provide a time-multiplexed neural network predictor, a neural network chip, a non-volatile processor, an energy management system, and an electronic device and apparatus suitable for the same, so as to perform energy management in a low-cost manner.

To achieve the above and other related objects, a first aspect of the present application provides a neural network predictor based on time multiplexing, the neural network predictor including a neural network module and a timing control module; the neural network module performs prediction calculation based on at least one of the received power supply information, the received power storage information and the received power failure information at least one moment, and outputs at least one of a data bit width instruction, a starting instruction or a write strategy instruction or/and service quality prediction information; and a timing control module for controlling the timing of the prediction calculation of the neural network module outputting the at least one instruction or/and the qos prediction information based on the received at least one information.

A second aspect of the present application provides a neural network chip including the time-multiplexing-based neural network predictor described in the first aspect above.

A third aspect of the present application provides an electronic device comprising the time-multiplexing-based neural network predictor described in the first aspect above.

A fourth aspect of the present application provides an electronic device, including a feature extraction module, which is specifically configured to extract at least one of power supply information, power storage information, and power failure information of the electronic device at least one time, the neural network predictor based on time multiplexing according to the first aspect, and an execution module, which performs energy management on an operation according to at least one of a data bit width instruction, a start instruction, or a write strategy instruction output by the neural network predictor, or/and service quality prediction information.

A fifth aspect of the present application provides a nonvolatile processor, including the neural network predictor based on time multiplexing according to the first aspect, and an execution module that performs energy management on an operation according to at least one instruction of a data bit width instruction, a start instruction, or a write policy instruction output by the neural network predictor, or/and service quality prediction information.

A sixth aspect of the present application provides an energy management system, which is applied to an electronic device with a processor, and includes the neural network predictor based on time multiplexing according to the first aspect, and an execution module that performs energy management on an operation according to at least one instruction of a data bit width instruction, a start instruction, or a write policy instruction output by the neural network predictor, or/and service quality prediction information.

As described above, the data transmission bit width or the data write strategy and the operation starting time are obtained by predicting the future electric energy input or power-off time, so that the operation of the processor can be ensured to be matched with the obtained expected energy, the retention time of the nonvolatile element is dynamically adjusted to be matched with the electric energy condition according to the write strategy, and the service quality is matched with the minimum service quality requested in advance; the method and the device realize various prediction calculations in different time periods through a hardware architecture of the neural network predictor, in other words, the method and the device realize the prediction calculations of a plurality of small-scale neural networks in different time periods, thereby achieving the purpose of saving hardware cost and area.

Drawings

FIG. 1 is a block diagram of a neural network predictor according to an embodiment of the present disclosure.

Fig. 2 is a circuit block diagram of a feature extraction module according to an embodiment of the present application.

FIG. 3 is a diagram of a neural network predictor according to an embodiment of the present invention.

FIG. 4 is a diagram of another neural network predictor according to an embodiment of the present application.

FIG. 5 is a block diagram of a hardware architecture of a neural network module according to an embodiment of the present invention

FIG. 6 is a diagram of a hardware architecture of a neural network predictor according to another embodiment of the present application

FIG. 7 is a diagram illustrating a hardware architecture of a neural network predictor according to another embodiment of the present application.

Fig. 8 is a schematic diagram of a functional module in the present application in which a neural network module is configured with a variety of predictors.

FIG. 9 is a flow chart illustrating the predicted computation timing of a power-up cycle according to an embodiment of the present application.

FIG. 10 is a flow chart illustrating the predicted computation timing for a power down cycle according to one embodiment of the present application.

FIG. 11 is a diagram illustrating the relationship between the write current and the write pulse width of the write strategy according to an embodiment of the present invention.

Fig. 12 is a schematic structural diagram of a neural network chip according to the present application.

FIG. 13 is a block diagram of a non-volatile processor according to an embodiment of the present application.

FIG. 14 is a block diagram of a non-volatile processor according to another embodiment of the present application.

Fig. 15 is a schematic diagram of an electronic device according to an embodiment of the present application.

Fig. 16 is a schematic diagram of an electronic device according to another embodiment of the present application.

Fig. 17 is a schematic diagram of an energy management system according to an embodiment of the present invention.

Fig. 18 is a schematic diagram of an energy management system according to another embodiment of the present application.

FIG. 19 is a block diagram illustrating an approximate computing architecture according to an embodiment of the present application.

FIG. 20 is a circuit diagram illustrating a write operation according to an embodiment of the present application.

Detailed Description

The following description of the embodiments of the present application is provided for illustrative purposes, and other advantages and capabilities of the present application will become apparent to those skilled in the art from the present disclosure. In the following description, reference is made to the accompanying drawings that describe several embodiments of the application. It is to be understood that other embodiments may be utilized and that compositional and operational changes may be made without departing from the spirit and scope of the present application. The following detailed description is not to be taken in a limiting sense, and the scope of embodiments of the present application is defined only by the claims of the patent of the present application. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application.

Although the terms first, second, etc. may be used herein to describe various elements in some instances, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first threshold may be referred to as a second threshold, and similarly, a second threshold may be referred to as a first threshold, without departing from the scope of the various described embodiments.

Also, as used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context indicates otherwise. For example, the term "at least one time" in this application includes a time and a plurality of times. As used herein, a phrase referring to "at least one of a list of items refers to any combination of those items, including a single member. By way of example, "at least one of a, b, or c" is intended to encompass: a. b, c, a-b, a-c, b-c and a-b-c.

It will be further understood that the terms "comprises," "comprising," "includes" and/or "including," when used in this specification, specify the presence of stated features, steps, operations, elements, components, items, species, and/or groups, but do not preclude the presence, or addition of one or more other features, steps, operations, elements, components, species, and/or groups thereof. The terms "or" and/or "as used herein are to be construed as inclusive or meaning any one or any combination. It should be understood that the term "and/or" herein is merely one type of association relationship that describes an associated object, meaning that three relationships may exist, e.g., a and/or B may mean: a exists alone, A and B exist simultaneously, and B exists alone. In addition, the character "/" herein generally indicates that the former and latter related objects are in an "or" relationship.

It should be understood that the term "instructions" may also be broadly interpreted in this application to mean instructions, data, information, signals, or any combination thereof, and the like. The "Nonvolatile" or "Nonvolatile" are both expressed by the same concept, and the corresponding english language is Nonvolatile.

In internet of things devices such as wearable devices or implantable devices, for example, the cost of replacing batteries, the safety of batteries, the volume of battery compartments, the charging time and the timeliness are all factors to be considered, and many devices desire the volume of batteries in the system to be as small as possible or even no batteries, so self-powered systems including an ambient energy harvesting power supply or a data acquisition device have come to be developed, especially with the development of Nonvolatile Processors (NVPs), so that the ambient energy harvesting power supply is popularized in the use of wearable devices. Non-volatile processors can handle unstable input power by backing up the computational state, and can ensure that systems using these processors are allowed to operate without a battery or super capacitor in a very short time frame as compared to a battery.

Although the non-volatile processor can ensure that the program is continuously executed under the condition of unstable power supply to some extent, when the power supply of the data acquisition device is unstable, the processor in the existing data acquisition device cannot process the latest acquired data, ensure the data accuracy, recover calculation or backup calculation and the like. Of course, the full use of energy may increase the number of backup operations, but also result in more energy being wasted in unnecessary backup and restore operations, and if an energy saving strategy is adopted, this may result in unnecessary leakage of the capacitor, and in addition, the capacitor cannot store newly collected energy even at full charge, and will also delay the service response time. There is a need for optimization of energy management, such as predicting future energy input to better allocate resources for a subsequent task, and predicting outage duration to reduce reserve time and power for backup operations.

The present application provides a neural network predictor based on time Multiplexing, which may also be referred to as time-series Multiplexing (tdm) or time-division Multiplexing, and is used to implement multiple kinds of prediction calculations in different time periods through a hardware architecture of one neural network predictor, that is, implement prediction calculations of multiple small-scale neural networks in different time periods, thereby achieving the purpose of saving hardware cost and area.

The neural network predictor of the present application is used for energy management of electronic devices with processors for high efficiency of power usage, computational efficiency of the processors and reduction of backup energy while maintaining the most basic quality of service, etc. The energy management system is applied to electronic equipment with a processor. In one embodiment, the processor is, for example, but not limited to, a non-volatile processor (NVP), and in other embodiments, the processor may be a general processor, such as any commercially available processor, controller, microcontroller or state machine without departing from the inventive concepts and concepts disclosed herein. A processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

In embodiments provided herein, the electronic device is an internet of things device, e.g., a wearable device or an implantable device, such as a wearable electronic device may include any type of electronic device that may be worn on a limb of a user. The wearable electronic device may be secured to a human limb such as a wrist, ankle, arm, or leg. Such electronic devices include, but are not limited to, health or fitness assistant devices, digital music players, smart phones, computing devices or display exercise or other activity monitors, time-tellable devices, devices capable of measuring biometric parameters of a wearer or user, and the like. Such as a blood glucose test device or the like.

As one example, the wearable electronic device may be implemented in the form of a wearable health assistant that provides health-related information (real-time or non-real-time) to the user, an authorized third party, and/or an associated monitoring device. The device may be configured to provide health-related information or data, such as, but not limited to, heart rate data, blood pressure data, temperature data, blood oxygen saturation data, diet/nutrition information, medical reminders, health-related reminders or information, or other health-related data. The associated monitoring device may be, for example, a tablet computing device, a telephone, a personal digital assistant, a computer, or the like.

As another example, the electronic device may be configured in the form of a wearable communication device. The wearable communication device may include a processor coupled to or in communication with a memory, one or more communication interfaces, output devices (such as a display and speakers), and one or more input devices. One or more communication interfaces may provide electronic communication between the communication device and any external communication network, device, or platform, such as, but not limited to, a wireless interface, a bluetooth interface, a USB interface, a Wi-Fi interface, a TCP/IP interface, a network communication interface, or any conventional communication interface. In addition to communication, the wearable communication device may provide information, messages, video, operational commands, etc. regarding the time, health, status, or externally connected or communicating devices and/or software running on such devices (and may receive any of the above from an external device).

Referring to fig. 1, which is a block diagram of a neural network predictor 1 according to an embodiment of the present invention, as shown in the figure, the neural network predictor includes a neural network module 11 and a timing control module 12.

The neural network module 11 performs prediction calculation based on at least one of the received Power supply information (Power Sensing), Power storage information (Stored Energy Sensing), and Power failure information (Power output Sensing) at least one time, outputs at least one of a data bit width instruction (Bitwidth), a System Start instruction, and a Write policy instruction (Write Configuration) or/and quality of service prediction information (Predicted QoS), the data transmission bit width or the data write strategy and the operation starting time are obtained by predicting the future electric energy input or power-off time, and to give the prediction to the processor to ensure that the processor's operation matches the expected energy it has acquired, and dynamically adjusting the retention time of the nonvolatile element to match the power condition according to the write strategy, and matches the Quality of Service (QoS) with the lowest (most basic) Quality of Service requested in advance.

The neural network module 11 of the time-multiplexed based neural network predictor 1 predicts the prediction module 10 based on receiving power supply information, power storage information, and power failure information of the electronic device at one or more times, such as power supply information of the last 10 historical times, power storage information of the last 10 historical times, or power failure information of the last 10 historical times of the electronic device in one embodiment. In an example, the power supply information of the last 10 historical moments of the electronic device is the last 10 continuous power-on times, or the power storage information of the last 10 historical moments of the electronic device is the remaining power of the last 10 moments, or the power failure information of the last 10 historical moments of the electronic device is the power failure duration of the last 10 moments.

In the embodiment, the time may be different time periods, and may be divided into a plurality of time levels according to different requirements, such as 10ms, 100ms, 1s, 2s, 3s, 10s, 1 minute, 10 minutes, 1 hour, 1 day, etc., it should be noted that the above time is exemplary, and in different implementation situations, the time is not limited thereto.

In an embodiment, the power supply information is power supply information of the electronic device, the power supply information is generated by a self-powered system of the electronic device, such as an energy harvester, which obtains energy from human body movement, for example, vibration energy from actions or behaviors of walking or limb swinging, jumping, pressing (such as pressure obtained by a small energy harvester implanted in a shoe during running), breathing, and the like, and converts the vibration energy into electric energy, and in other cases, the energy may also come from a natural environment, such as solar energy and the like. The energy harvested by the energy harvester needs to be processed from AC to DC or DC to DC and then the harvested energy is temporarily stored in off-chip or even on-chip capacitors, which are mainly used to support data rather than to store energy.

In an embodiment, the power storage information is power information stored in a battery or a power storage capacitor of the electronic device, such as the stored power information is obtained in real time or intermittently under the assumption that power consumption is constant.

In an embodiment, the power failure information is information of interruption of power input in the electronic device due to insufficient power supply, energy exhaustion, human factors (such as human setting or human damage) or unforeseen accidents, and the like, such as information of time of power failure and duration of power failure. In one embodiment, 10 levels of power down time may be set, for example, 10ms, 100ms, 1s, 2s, 3s, 10s, 1 minute, 10 minutes, 1 hour, 1 day, etc. for different periods of time.

In an embodiment, the neural network predictor 1 receives at least one of Power supply information (Power Sensing), Power storage information (Stored Energy Sensing), and Power failure information (Power output Sensing) at least one time from a feature extraction module 10, referring to fig. 2, which is a circuit block diagram of the feature extraction module in an embodiment of the present application, and as shown in the figure, the feature extraction module 10 includes: the system comprises a detection unit, an energy storage unit and a power failure sensing unit, wherein the characteristic extraction module 10 is in communication connection with the neural network predictor 11.

In this embodiment, the feature extraction module 10, which is a front-end circuit, includes a power supply, such as a battery or a power supply including a charging device, one end of the power supply is grounded, the other end of the power supply is connected to an Rs resistor, the Rs resistor serves as a detection element, and the Rs resistor and a 6-bit ADC converter in fig. 2 constitute a detection unit in this embodiment, which is configured to acquire the power supply information (Input power sensing) provided by the electronic device by collecting a current value flowing through the Rs resistor or a voltage value at two ends of the Rs resistor, that is, the power supply information is, in the embodiment, power supply information of the electronic device, and the power supply information is generated by a self-power supply system of the electronic device, as shown by a solid arrow in fig. 2.

In this embodiment, the energy storage unit is configured to acquire Stored energy information (Stored energy sensing) by acquiring a voltage difference (voltage drop) between two ends of the energy storage element, which is indicated by a dotted arrow in the diagram, that is, the remaining energy of the energy storage element; the energy storage element is a grounding capacitor C1 in FIG. 2. The capacitor C1 and the ADC converter in fig. 2 constitute the detection unit described in this embodiment.

In this embodiment, the Power-off sensing unit is configured to acquire Power-off information (Power-off sensing) including Power-off time by collecting a voltage difference between two ends of the discharging element, which is shown by a dotted line arrow in fig. 2. The discharging element is a leakage capacitor C2 in fig. 2, one end of the leakage capacitor C2 is connected via a DC-DC converter, a Charge Breaker (Charge Breaker) and an LDO device, and the other end of the leakage capacitor C2 is grounded. In a specific implementation, the leakage capacitor C2 is charged each time during a recovery operation controlled by a charge-breaker. The DC-DC converter, LDO device, charge breaker, leakage capacitor C2 and ADC converter in fig. 2 constitute an inductance-breaking sensing unit. As shown in fig. 2. By charging the leakage capacitor each time during a recovery operation controlled by the LDO device as shown in fig. 2, the voltage of the capacitor is checked when the electronic device recovers from a power outage, and the system power down time can be calculated from the voltage drop detected by the ADC.

In an embodiment, the Neural Network module 11 is, for example, a feed forward Neural Network (feed forward Neural Network), which is a Neural Network trained offline or a Neural Network trained online in a back propagation manner. Referring to a neural network shown in fig. 3, fig. 3 is a schematic diagram of a neural network in an embodiment of the neural network predictor of the present application, as shown in the drawing, in this embodiment, the neural network is, for example, a feedforward neural network, and the feedforward neural network includes 1 Input Layer (Input Layer), 2 Hidden layers (Hidden layers 1, 2), and 1 Output Layer (Output Layer), each Layer has 10 neurons, each neuron has 10 outputs, and the power supply information, the power storage information, or the power failure information of 10 time instants (from On time _1 to On time _10) received from a non-volatile Shifter (NV Shifter) is used to predict the future generated power or the time of possible future break. Without being limited thereto, in other possible embodiments, as shown in fig. 4, which is another neural network diagram of the neural network predictor of the present application in one embodiment, the feed-forward neural network may include more Hidden layers (

Hidden Layer

1, 2 … … N) as shown in the figure, and each Layer may also include more or less neurons (N). The neural network shown in fig. 4 is an off-line trained neural network or an on-line back propagation trained neural network.

Referring to fig. 5, which is a schematic diagram illustrating a hardware architecture of a neural network module in an embodiment of the present disclosure, as shown in the drawing, in the embodiment, the neural network module 11 includes a neural network unit 111 and a one-time prediction state machine 112, wherein the neural network unit 111 includes a neuron register 1110, a weight register 1112 storing a plurality of weights, a plurality of selectors 1113 for selecting data input or output, and a multiply-accumulate unit 1114. The one-time prediction state machine 112 is used for controlling the timing of the neural network unit 111 receiving the at least one information for one-time prediction calculation. The neural network module 11 shown in fig. 5 is a serial architecture, and the one-time prediction state machine 112 controls the corresponding selector 1113 to select a weight value and the source neuron and the target neuron that need to be activated from the weight value register 1112 according to the input of the neural network unit 111, and then delivers the weight value and the source neuron and the target neuron to a Multiply-and-Accumulate (MAC) unit 1114 after calculation, and then writes back the neurons in the neural network unit until all the neurons in the input layer, the hidden layer and the output layer are processed. The weights pre-stored in the weight register 1112 are obtained by training.

In this embodiment, the weight register 1112 includes a Nonvolatile shift unit or a Nonvolatile memory unit for storing the weight corresponding to each prediction calculation, where the Nonvolatile shift unit is a Nonvolatile Shifter (NV shift), and the Nonvolatile memory unit is a Nonvolatile memory (NVM).

In the present embodiment, the one-time prediction state machine 112 has a nonvolatile shift unit or a nonvolatile storage unit for storing a timing control program, and specifically, for controlling the output timing of each selector 1113 by the timing control program. The nonvolatile shift unit is a nonvolatile shifter, and the nonvolatile storage unit is a nonvolatile memory.

The timing control module 12 is configured to control the neural network module 11 to output a timing of prediction calculation of a data bit width instruction (Bitwidth), a System Start instruction (System Start) or a Write policy instruction (Write Configuration) based on at least one of the received Power supply information (Power Sensing), the received Power storage information (Stored Energy Sensing), and the received Power failure information (Power output Sensing) at least one time, or/and prediction calculation of quality of service prediction information (Predicted QoS), so as to ensure that the plurality of types of prediction calculation can share one neural network module, in other words, one prediction hardware (one neural network architecture) can be used to complete functions of all the predictors.

Referring to fig. 6, a hardware architecture diagram of a neural network predictor according to another embodiment of the present invention is shown, in the embodiment shown in fig. 6, in order to enable multiple types of prediction calculations to be completed in one hardware architecture at different times, the hardware architecture is further specified in the present application. As shown in the figure, the neural network module 11 further includes a softmax state machine 113 and a judging unit 114 (i.e., a portion shown in fig. 6 where OR is 0) configured in the multiply-accumulate unit. In this embodiment, to normalize the hardware architecture, a number of virtual connections are constructed in the network of the neural network module to normalize the neural network topology by inserting 0 connection Weights (i.e., Weights 1, Weights 2 … … Weights 5 of the weight registers shown in FIG. 6).

In the embodiment shown in FIG. 6, the one-time predictive state machine 112 includes a non-volatile shift unit for storing a timing control program. The schematic of the one-shot prediction state machine 112 controlling the timing of the one-shot prediction computation of the information that the neural network element receives the input is represented by the dashed arrow shown in FIG. 6; the schematic of the timing control module 12 controlling the timing of the neural network modules is represented by the dotted arrows shown in fig. 6.

When the neural network module 11 performs one-time prediction (such as prediction of future on-off time or prediction of future power-off time), the one-time prediction state machine 112 controls the selector to output the selected weight value and the source neuron and the target neuron which need to be activated to the judging unit 114 after one-time calculation, whether any input of the judging unit 114 is 0 or not is judged by the judging unit 114, if any input is 0, the multiplier is bypassed, if any input is not 0, the multiplier multiplies and accumulates the inputs, and then the neurons in the neural network unit 111 are written back until all neurons in the input layer, the hidden layer and the output layer are processed. Finally, the softmax layer is executed under the control of the softmax state machine 113. After all steps are performed by the predictor, the output of the predictor is then selected to be stored in a non-volatile shift unit 115 in the execution module, and some of the output is updated in a non-volatile memory (NVM)115 shown in FIG. 6 (e.g., the timing control module also controls the neural network module to update the power down information in one power up cycle, or the timing control module controls the neural network module to update the power up information in one power down cycle) for the next other prediction, such as power down prediction confidence or power up confidence. In this embodiment, the multiplier and the adder in the multiply-accumulate unit are a floating-point multiplier and a floating-point adder.

In this embodiment, the timing control module 12 includes a nonvolatile shift unit or a nonvolatile memory unit for storing a timing control program, where the nonvolatile shift unit is a nonvolatile shifter and the nonvolatile memory unit is a nonvolatile memory.

In one embodiment, the neural network predictor 11 includes one or more nonvolatile shift units and nonvolatile storage units, and the nonvolatile shift units are used for storing power supply information, power storage information and power failure information of one or more moments of the electronic device received from the feature extraction module. Referring to fig. 7, which is a schematic diagram illustrating a hardware architecture of a neural network predictor in another embodiment of the present disclosure, as shown in the figure, the Power supply information (Power Sensing) and the Power failure information (Power output Sensing) are Stored in the nonvolatile shift units, the Power storage information (Stored Energy Sensing) is directly extracted from the feature extraction module 10 to the neural network module 111, and the nonvolatile memory stores updated information, for example, the timing control module 12 further controls the neural network module 11 to update the Power failure information in a Power-on cycle; or the timing control module 12 controls the neural network module 111 to update the power supply information in one power-off period for the next other prediction, such as the power-off prediction confidence or the power-on confidence.

In the embodiment shown in fig. 7, the neural network predictor 11 outputs at least one of the output data bit width command (Bitwidth), the Start command (System Start) and the Write strategy command (Write Configuration) or/and the Predicted QoS to an execution module after prediction calculation, and in this embodiment, the execution module 13 includes an approximate bit width controller (not numbered), a Start controller (not numbered) and a retention time controller (not numbered).

The approximate bit width controller is used for controlling the precision of operation according to a data bit width instruction (BitWidth) when receiving the data bit width instruction output by the neural network predictor 11; in this embodiment, the approximate bit width controller has one or more nonvolatile shift units, such as a nonvolatile Shifter (NV Shifter), into which the approximate bit width controller stores the received data bit width instruction. The processor is used for calculating and processing the sensing data or the interaction data acquired by the electronic equipment. In some examples, the processing of the sensed data, such as the wearable device, generates user data that may be transmitted by the wireless module or displayed by the display device by processing the collected heart rate data, blood pressure data, temperature data, blood oxygen saturation data, diet/nutrition information, medical reminders, health-related reminders or information, or other health-related data.

In some examples, the processing of the interaction data may be such as by a user operating a wearable device to respond to an event notification generated by a host device. The wearable device can receive notifications of events from the host device and present reminders and prompts for responses to the user. If the user responds to the prompt, the wearable device may transmit the response to the host device. For example, a user may respond to a phone call, text message, or other communication received at a host device.

The starting controller is used for starting the work of the nonvolatile processor when receiving a starting instruction output by the neural network predictor 11; in one embodiment, the start-up controller has one or more non-volatile shift cells, such as a non-volatile Shifter (NV Shifter), into which the start-up controller stores received start-up instructions. In this embodiment, the Start Controller is, for example, an NVP Start Trigger (NVP Start Trigger Controller) for controlling whether the nonvolatile processor starts to work.

The retention time controller is configured to execute a Write operation according to at least one of Write current and Write time included in a Write strategy command (Write Configuration) when receiving the Write strategy command output by the neural network predictor 11. In this embodiment, the retention time controller has one or more nonvolatile shift units, such as a nonvolatile Shifter (NV Shifter), into which the retention time controller stores the received write strategy instruction. The retention time controller executes a write operation of writing data according to the received write strategy instruction, and in an example, the data written to the memory of the electronic device by the write operation is, for example, calculation or processing data of the processor, or a backup calculation state is recorded.

The execution module also receives quality of service prediction information (Predicted QoS) output by the neural network predictor 11 to cause the processor to predict potential output quality of an electronic device running program based on dynamic bit width execution approximation and different approximation methods in dynamic backup data retention time, such that the results of the entire energy management system have quality of service control.

In an embodiment, the neural network module implements multiple prediction calculations through a multiplexing hardware architecture, and implements output of at least one of a data bit width instruction (Bitwidth), a Start instruction (System Start), or a Write policy instruction (Write Configuration) or/and Predicted QoS (Predicted QoS) to perform energy management on operations of the processor. In this embodiment, the neural network module 111 performs the following prediction calculation in a prediction calculation process based on the received at least one of the power supply information, the power storage information, and the power failure information of the electronic device at least one time:

(1) predicting and calculating to obtain future Power-on Time and Power-on Confidence (Confidence) of the electronic equipment based on the received Power supply information;

(2) performing prediction calculation based on the received future power-on time, power-on confidence and power storage information (Stored Energy Sensing) to obtain a data bit width instruction and a Start instruction (System Start);

(3) performing prediction calculation based on the received data bit width instruction and Power failure information (Power Outage Sensing) to obtain service quality prediction information;

(4) predicting and calculating the future Power-off time and the Power-off confidence of the electronic equipment based on the received Power-off information (Power Outage Sensing);

(5) and performing prediction calculation based on the received future power-off time and the power-off confidence coefficient to obtain a write strategy instruction and a write strategy confidence coefficient.

To further clarify the principle and efficacy of the present application, the 5 prediction calculations of the neural network module are respectively considered as 5 predictors, and the 5 predictors respectively perform the prediction calculations based on the same hardware architecture at different times. Referring to fig. 8, which is a functional block diagram of a neural network module configured with a plurality of predictors according to the present application, as shown in the figure, the neural network predictor 11 is connected to a feature extraction module 10 and an execution module 13, and the predictors for implementing 5 kinds of prediction calculations are respectively a future energy predictor (not numbered), an approximate bit width predictor (not numbered), a power-off predictor (not numbered), a backup time predictor (not numbered) and a quality of service predictor (not numbered).

In an embodiment, the timing control module (not shown in fig. 8) performs the predictive calculations for the power-up and power-down cycles, respectively, when the power module (e.g., self-powered system) of the electronic device is powered up, the energy system of the electronic device charges the energy storage capacitor until the stored energy reaches the starting threshold of the neural network predictive calculation, then first detects a leakage capacitor (e.g., leakage capacitor C2 in fig. 2) for measuring the power-down time, then charges again, and will remain charged until the next power failure (power-down cycle) occurs. The last power down time of the non-volatile shift cell in the neural network predictor is then updated and the sensed voltage in the leakage capacitor is detected before the leakage capacitor is charged again.

The time sequence control module is used for controlling the neural network module to perform prediction calculation based on at least one of the received power supply information and the power storage information in a power-on period and outputting at least one of a data bit width instruction and a start instruction or/and service quality prediction information.

Referring now to FIG. 9, therein is shown a flow chart of the predicted computation timing of a power-on cycle in one embodiment of the present application, as shown, for ease of understanding, in conjunction with the information (data) relationship shown between the predictors shown in FIG. 8. In this embodiment, the timing control module controls the neural network module to perform the prediction calculation timing in one power-on cycle, where the prediction calculation timing is as follows:

step S110 is executed first, and a first prediction calculation cycle is executed, in which a neural network module configured as a future energy predictor predicts a future Power-on Time (Power-on Time) and a Power-on Confidence (Confidence) of the electronic device based on the acquired Power supply information (Power Sensing) of the electronic device at a plurality of times. In the embodiment, the time may be different time periods, and may be divided into a plurality of time levels according to different requirements, such as 10ms, 100ms, 1s, 2s, 3s, 10s, 1 minute, 10 minutes, 1 hour, 1 day, etc., it should be noted that the above time is exemplary, and in different implementation situations, the time is not limited thereto.

In an embodiment, the power supply information of the electronic device is obtained by collecting a current value flowing through a detection element or a voltage value across the detection element at least one time, as described above with respect to the corresponding part in fig. 2.

In this embodiment, the future energy predictor has one or more nonvolatile shift units, such as a nonvolatile Shifter (NV Shifter), and stores the received power supply information of the electronic device at one or more time points into the nonvolatile shift units. In this embodiment, the future energy predictor stores the received power supply information of the electronic device at 10 moments in time into the nonvolatile shift unit. In the present embodiment, taking the last 10 consecutive power-on times of the electronic device received by the future energy predictor as an example, a counter (not shown) is used to calculate the power-on time, and the counter pushes the last calculated power-on time to the non-volatile shifter to replace the earliest power-on time.

In this embodiment, by configuring a fully connected feed forward neural network such that the neural network module has 2 hidden layers, 10 neurons are set per layer, 10 neurons of the output layer output 10 potential energy levels (energy levels), each energy level is converted to correspond to one energization time, each output indicates one energization confidence, where the highest value is output as the predicted energy level and its confidence when the energization confidence is very different from the other confidences. In one case, if the first few predicted energization confidences are within the 10% interval, the energy levels for the predicted future energization times are weighted averaged, and the corresponding energization confidences are also averaged.

In step S111, a second prediction calculation cycle is executed, and the neural network module configured as a bit width predictor performs prediction calculation based on the future power-on time, the power-on confidence, and the power storage information (Stored Energy Sensing) predicted and calculated by the future Energy predictor to obtain a data bit width instruction (Bitwidth) and a Start Threshold (System Start Stored Energy Threshold), and the bit width predictor dynamically collects the bit width required by the dynamic prediction execution module during each power-on interval, so that the electronic device can dynamically adjust the precision of processor calculation according to the expected Energy, thereby reducing the processor calculation and backup cost and shortening the response time.

In an embodiment, the power storage information (Stored power sensing) of the electronic device is obtained by the detection unit shown in fig. 2 by collecting a voltage drop of an energy storage element. In this embodiment, the energy storage unit is configured to acquire the electricity storage information by collecting a voltage difference (voltage drop) between two ends of an energy storage element, which is indicated by a dotted arrow in the figure, that is, the remaining capacity of the energy storage element, where the energy storage element is a ground capacitor C1 in fig. 2.

In one example, the data bit width instruction is calculated by prediction of 2 hidden layers upon receiving 10 potential energy levels (energy levels) and a power-on confidence for each energy level via a neural network configured as a bit width predictor. Specifically, the input layer of the feedforward neural network according to the bit width predictor receives 10 potential energy levels (energy levels) output by the future energy predictor and performs approximate calculation corresponding to the power-on confidence of each energy level to predict and determine the appropriate output data bit width to determine the precision of the processor at which energy level to perform operation.

In this embodiment, a feedforward neural network configured as a bit width predictor performs prediction calculation according to the future power-on time, the power-on confidence and the power storage information of the electronic device to obtain approximate configuration data (approximate Config); in one example, when 10 potential energy levels (energy levels) and a power-on confidence corresponding to each energy level are received via a neural network configured as a bit width predictor, the approximate configuration data is calculated by prediction of 2 hidden layers, and the approximate configuration data includes bit width information obtained by the bit width predictor via the neural network according to future power-on time, power-on confidence and power storage information prediction calculation, such as how many bits of bit width the processor needs to calculate to determine a correct or proper bit width configuration.

The feedforward neural network configured as a bit width predictor identifies the approximate configuration data (approximate Config) according to preset approximate indicating data (ACEN), and performs approximate calculation to generate a data bit width instruction (Bitwidth) when the approximate configuration data is judged to be approximately calculated. In one example, the approximation indicating data (ACEN) is preset, in particular, data that can be approximated via programmer identification setting, which includes a data buffer or an image (image) but does not include a basic variable, such as an index in a "for" loop.

Referring to FIG. 18, a schematic diagram of an approximation calculation architecture according to an embodiment of the present application is shown, wherein the approximation calculation architecture includes a 5-stage pipeline processor architecture, and a dynamic architecture approximation control unit controls the approximation calculation of the pipeline processor architecture. As shown in the figure, a neural network configured as a future Energy predictor is used for predicting and calculating the future Power-on time and the Power-on confidence coefficient of the electronic equipment according to the Power supply information (Input Power), then a neural network configured as a bit width predictor is used for predicting and calculating according to the future Power-on time, the Power-on confidence coefficient and the Power storage information (Stored Energy) of the electronic equipment to obtain approximate configuration data (approximate configuration), then a bit called ACEN is added to each approximate configuration data through an approximate control unit preset with an approximate designation data (ACEN) interface to identify whether the approximate configuration data can be approximated during operation, the dynamic architecture approximation control unit reads the approximate designation data (ACEN) and the obtained approximate configuration data (approximate configuration) from two operators in an instruction, if an operator is determined to be approximated, but the other operator can not be approximated, the instruction can not be approximated, if both operators can be approximated, the data bit width instruction (Bitwidth) is generated to enable the processor to utilize the processor architecture of the 5-stage pipeline to carry out approximation calculation, and further, the calculation (operation) precision of the processor is controlled. As can be seen from the above, a bit width predictor is configured to be activated at the beginning of a programmer-predefined loop in the main program, which is typically a loop for a new frame to be processed. The configured bit width predictor is used to determine the appropriate bit width to complete the loop operation of the entire program during this power-up cycle.

In an embodiment, the dynamic architecture approximation control unit is, for example, an approximation bit width controller disposed in a processor or an execution module, and the approximation bit width controller is configured to control the precision of an operation according to a data bit width instruction (Bitwidth) configured as a neural network output of a bit width predictor when the data bit width instruction is received; in this embodiment, the approximate bit width controller has one or more nonvolatile shift units, such as a nonvolatile Shifter (NV Shifter), into which the approximate bit width controller stores the received data bit width instruction.

The processor is used for calculating and processing the sensing data or the interaction data acquired by the electronic equipment. In some examples, the processing of the sensed data, such as the wearable device, generates user data that may be transmitted by the wireless module or displayed by the display device by processing the collected heart rate data, blood pressure data, temperature data, blood oxygen saturation data, diet/nutrition information, medical reminders, health-related reminders or information, or other health-related data.

Then, step S112 is executed, and when the start-up threshold satisfies the first threshold Y, a start-up instruction is output to the processor, and in this embodiment, when the neural network module configured as the bit width predictor receives 10 potential energy levels (energy levels) and the power-on confidence corresponding to each energy level, the neural network module outputs the start-up threshold as an indication of whether to start up or not by means of prediction calculation of 2 hidden layers. In this embodiment, the threshold Y is, for example, 20dB, but is not limited thereto, and different thresholds Y may be set in different implementation states, for example, for different types of electronic devices.

For example, if the predicted potential input energy level is relatively low and there is insufficient stored energy in the buffer capacitor, then no start-up is performed; if the predicted potential input energy is high, but the received energization confidence is low, then a startup is indicated only if the stored energy level is high enough to reach the prediction threshold; if the predicted potential input energy is high and the confidence of power-on is high, even if the stored energy is low, a startup is indicated (very low threshold of predictor output) for better forward progress and QoS satisfaction. In one example, the energization confidence ratio is relatively low, e.g., less than 30%, and the energization confidence ratio is relatively high, e.g., greater than 70%. But not limited to this, the threshold of the power-on confidence can be dynamically adjusted according to the actual situation under the circumstance of knowing the innovative idea of the present application.

Because more electric energy is dissipated in high bit width operation (high calculation precision of a processor) than in low bit width (low calculation precision of the processor), the starting time of the processor is controlled through controlling the processor to start and store an energy threshold value in the application, and the purpose of relieving the low-quality output problem is achieved under the condition that energy management is reasonable, the starting time of the processor is delayed through controlling the parameter of the threshold value Y until enough energy exists in an energy storage capacitor to restart the processor, and high-quality data operation or output is ensured under the reasonable energy management mechanism.

In an embodiment, the control of the processor Start may be implemented by a Start controller, the Start controller is configured to Start the processor when receiving a Start instruction (System Start); in one embodiment, the start-up controller has one or more non-volatile shift cells, such as a non-volatile Shifter (NV Shifter), into which the start-up controller stores received start-up instructions. In this embodiment, the Start Controller is, for example, an NVP Start Trigger (NVP Start Trigger Controller).

To avoid electronic devices producing low quality data output, such as data below 20dB, data of 20dB and above is generally considered reasonable quality data. In step S113, a third prediction calculation cycle is executed, and the neural network module configured as a quality of service predictor performs prediction calculation according to the data bit width instruction and outage information to obtain Predicted QoS (Predicted QoS), where the outage information includes an outage confidence level; in one embodiment, the outage confidence is obtained by prediction by an outage predictor shown in FIG. 8.

In the present embodiment, the approximate bit width and the average power outage prediction confidence are obtained during the frame data as input processing. In this embodiment, the first threshold is, for example, 20dB, but is not limited thereto, and different thresholds may be set in different implementation states, for example, for different types of electronic devices.

Step S114 is then executed to output the Predicted QoS to the execution module when the QoS prediction information satisfies a first threshold. Because more electric energy is dissipated in high bit width operation (high calculation precision of a processor) than in low bit width (low calculation precision of the processor), the processor is controlled to start and store an energy threshold value in the application, the purpose of relieving the problem of low-quality output is achieved under the condition of reasonable energy management, the starting time of the processor is delayed by controlling the parameter of the first threshold value until enough energy exists in the energy storage capacitor to restart the processor, and high-quality data operation or output is ensured under the mechanism of reasonable energy management.

In an embodiment, the timing control module is further configured to control the neural network module to update the power outage information in a power-up cycle, for example, update the power outage information in the last power-up cycle, and store the power outage information in a non-volatile Shifter (NV Shifter) for next other prediction, such as prediction of power outage time and power outage prediction confidence.

Although many electronic devices are powered by an unstable power source (e.g., an energy harvester in a self-powered system) with the aid of non-volatile elements, the backup operation of the electronic device while in operation still consumes a significant amount of power, especially when the power source is intermittent. Therefore, if the power-off information can be obtained in advance by a certain technique, the NVM (Non-volatile memory) retention time can be shortened from a very long time (e.g. more than 10 years) to a time only slightly longer than the power-off time, so that the necessary energy can be saved during the data backup operation. In addition, since the electronic device usually has a block/distributed non-volatile memory (NVM), this makes the system often consume a large amount of standby energy during data backup operation, and for these unnecessary consumption, the standby energy can be reduced by improving the backup retention time, thereby realizing the control of energy saving. For this reason, in the present application, a reasonable write strategy can be determined by predicting the power-off time, in other words, how much write current is used at what time and how long write time is used to decide the time for data backup.

Referring to FIG. 10, a flow chart of the timing of the prediction calculation of the power down cycle in one embodiment of the present application is shown, as shown, for ease of understanding, with continued reference to the information (data) relationship shown between the predictors shown in FIG. 8. In this embodiment, the timing control module is configured to control the neural network module to perform prediction calculation and output a write strategy instruction based on the received power outage information in a power outage period.

As shown in fig. 10, the timing sequence for controlling the neural network module to perform the prediction calculation in one power-off cycle by the timing sequence control module is as follows:

firstly, step S110' is executed, a first prediction calculation cycle is executed, and the neural network module configured as a Power-off predictor performs prediction calculation on the future Power-off Time (Power-off Time) and the Power-off Confidence (Confidence) of the electronic device based on the received Power-off information (Power-off Sensing); in one example, the power outage information is information of power input interruption in the electronic device due to factors such as insufficient energy supply, energy exhaustion, human setting or unpredictable accidents, and the like, such as information of a time node of the power outage and a duration of the power outage. In one embodiment, 10 levels of power down time may be set, for example, 10ms, 100ms, 1s, 2s, 3s, 10s, 1 minute, 10 minutes, 1 hour, 1 day, etc. for different periods of time.

In one example, the power outage information may be collected by a power outage sensing unit of the feature extraction module shown in fig. 2, for example, a DC-DC converter, an LDO device, a Charge Breaker (Charge Breaker), a leakage capacitor C2 and an ADC converter in fig. 2 constitute a power outage sensing unit, and the system power outage time may be calculated by a voltage drop detected by the ADC by charging the leakage capacitor each time during a recovery operation controlled by the Charge Breaker (Charge Breaker) as shown in fig. 2 and checking the voltage of the capacitor when the electronic device recovers from a power outage. In particular, as shown in the power-off sensing unit portion of fig. 2. Each time the system is powered up (before resuming operation), the leakage capacitor charges at a fully stable voltage. When a power failure occurs, the voltage on the leakage capacitor drops with the passage of time, and the detected power-off time can be obtained by measuring the remaining voltage in the leakage capacitor after the next power-on. In this embodiment, the received Power-off information (Power output Sensing) of the electronic device at one or more time points is stored in the nonvolatile shift unit. In one example, the last 10 consecutive power-off messages received from the electronic device are used to calculate the power-off time by a counter (not shown), which pushes the last calculated power-off time to the non-volatile shifter to replace the earliest power-off time.

In this embodiment, by configuring the Power-off predictor of the neural network module as a fully connected feedforward neural network including 1 input layer, 2 hidden layers, and 1 output layer, each layer is provided with 10 neurons, the input layer is used for receiving Power-off information such as 10 historical Power-off times from the nonvolatile shift unit, and future Power-off times (Power-off times) of, for example, 10 Power-off Time levels and Power-off Confidence degrees (confidences) corresponding to each future Power-off Time are predicted by the output layer after calculation through the 2 hidden layers of the feedforward neural network.

In step S111', a second prediction calculation cycle is executed, and the neural network module configured as a time backup predictor performs prediction calculation based on the received future power failure time and power failure Confidence to obtain a Write strategy command (Write Configuration) and a Write strategy Confidence (Confidence). The write strategy command includes at least one of write current and write time for writing data. The writing time is a data writing time duration, in a specific implementation, the data writing time duration is determined by a writing pulse width, and the writing current and the writing pulse width can affect a retention (holding or backup) time of the written data. In this embodiment, the write strategy command is configured for a retention time including write data or is a retention time strategy.

Then, the process proceeds to step S112', and the write strategy instruction is output to the processor when the write strategy confidence satisfies a second threshold. In this embodiment, the write strategy instruction is output to the execution module when the write strategy confidence level satisfies a second threshold value X (X shown in fig. 8), where the second threshold value X is a preset outage confidence level, for example, the outage confidence level is 80%, and when the outage confidence level obtained through the neural network prediction configured as the time backup predictor is greater than the preset outage confidence level X, the write strategy instruction is sent to the execution module. The second threshold X may be set to different values in different implementation states, including the medium of the storage device in which data is written, such as STT-ram (shared transmitter Technology Random Access memory), etc.; in addition, these different implementation states may be different in the data content that the processor requires to record, and so on.

The write current and write pulse width can affect the retention (hold or backup) time of the written data. Referring to fig. 11, which shows a graph illustrating the relationship between the write current and the write pulse width of the write strategy according to an embodiment of the present invention, as shown in the figure, the abscissa represents the write pulse width, and the ordinate represents the magnitude of the write current, taking the STT-RAM as an example of the medium of the storage device for writing data in the figure, since the data can be written with different currents to determine the data retention time,

as shown in fig. 11, when the write strategy determines that the retention time of the written data is 10ms, the write current and the write pulse width are distributed by using a curve composed of square points in fig. 11, and the trade-off between the write current and the write pulse width is the write current and the write pulse width using the coordinates of the vertex of the upper right corner of the small square illustrated in fig. 11 (i.e., the point a indicated by the arrow in fig. 11); accordingly, when the write strategy determines that the retention time of the written data is 1 day, the write current and the write pulse width distributed by a curve composed of dots in fig. 11 are adopted, and the trade-off between the write current and the write time in the curve is the write current and the write pulse width which are adopted by the vertex coordinate point (i.e., the point b indicated by the arrow in fig. 11) at the upper right corner of the large square illustrated in fig. 11; similarly, when the write strategy determines that the retention time of the written data is 1 minute or 1 second, the write current and the write pulse width distributed by a curve composed of a regular triangle point and an inverted triangle point in fig. 11 are adopted, and the trade-off write current and the write pulse width are respectively adopted in the curve at the vertex of the upper right corner of the square illustrated in fig. 11 (i.e., the point c or the point d indicated by the arrow in fig. 11).

Since the write current and write pulse width of the data can affect the retention (hold or backup) time of the written data, a write current is provided in this application, see FIG. 20, which is a schematic diagram of a write operation circuit in one embodiment of this application, as shown, different currents are generated by a current mirror, different times are determined by a counter, and the selection of the currents is controlled by the illustrated write data MUX array to determine how much current to write the data, such as the illustrated I₁To I₈The 8-way circuit shown selects different time lengths through the illustrated write time comparator to determine how much time is used to write data.

In FIG. 20, I_refIs the base current of the current mirror, and the secondary I is generated by changing the W/L ratio (the width-length ratio of the transistor channel can determine the amplification factor of the current mirror) of the current mirror composed of PMOS transistors₁To I₈In the present embodiment, the maximum current change rate is from 1 day to 10 ms. According to what is shown in the figurePredicting the write current configuration, different currents can be selected in the MUX array, the write current is connected to "Bit" or "Bit B" (the write data can be changed by flipping the current direction of "Bit" or "Bit B"), depending on the input of the "write data" signal. The other row of "Bit" or "Bit B" controls the write time, as compared to the predicted write time configuration according to the illustration by a high frequency 4-Bit counter (sub ns per cycle) that is disconnected from Ground (GND) to terminate the write operation once the counter reaches the preset write time. In this embodiment, data is written into a memory by performing the write operation, where the memory is a Non-volatile memory (NVM), such as STT-ram (shared memory Technology Random Access memory).

In one embodiment, when the power down time is predicted to be short or the power down confidence is low (small), such as the power down time is less than 50ms or the power down confidence is less than 80%, the system may run using the stored energy without a backup operation, i.e., without sending write strategy instructions to the processor.

In another embodiment, when a Memory in the electronic device is, for example, a Static Random-Access Memory (SRAM) with low standby power consumption, since the standby power consumption of the Memory is relatively low, when the power-off time is, for example, between 50ms and 0.2s, the backup may not be needed, that is, the write strategy instruction is not sent to the execution module.

In an embodiment, the timing control module is further configured to control the neural network module to update the power supply information in a power-off period. For example, the last power supply (power-on) information is updated in the current power-off cycle, and the power supply information is stored in a non-volatile Shifter (NV Shifter) for the next other prediction, such as prediction of future power-on time and power-on prediction confidence.

The neural network predictor based on time multiplexing obtains data transmission bit width or a data write strategy and operation starting time by predicting future electric energy input or power-off time, so that the operation of a processor can be ensured to be matched with expected energy obtained by the processor, the retention time of a nonvolatile element is dynamically adjusted to be matched with the electric energy condition according to the write strategy, and the service quality is matched with the minimum service quality requested in advance; in addition, the neural network predictor based on time multiplexing is used for realizing various prediction calculations in different time periods through the hardware architecture of one neural network predictor, in other words, the neural network predictor based on time multiplexing realizes the prediction calculations of a plurality of small-scale neural networks in different time periods, thereby achieving the purpose of saving hardware cost and area.

The present application further provides a neural network chip, which includes the neural network predictor based on time multiplexing described in the above embodiments. Please refer to fig. 12, which is a schematic structural diagram of a neural network chip according to the present application. As shown in the figure, the neural network chip 2 is integrated with the neural network predictor 20 based on time multiplexing for performing energy management on an electronic device having a processor, and the neural network predictor 20 includes a neural network module and a timing control module.

Wherein, the neural network module carries out prediction calculation based on at least one of the received Power supply information (Power Sensing), Power storage information (Stored Energy Sensing) and Power failure information (Power Outage Sensing) at least one moment, and outputs at least one of a data bit width instruction (BitWidth), a Start instruction (System Start) or a Write strategy instruction (Write Configuration) or/and quality of service prediction information (Predicted QoS), the data transmission bit width or the data write strategy and the operation starting time are obtained by predicting the future electric energy input or power-off time, and to give the prediction to the processor to ensure that the processor's operation matches the expected energy it has acquired, and dynamically adjusting the retention time of the nonvolatile element to match the power condition according to the write strategy, and matches the Quality of Service (QoS) with the lowest (most basic) Quality of Service requested in advance.

The timing control module is configured to control a timing of prediction calculation of at least one of a data bit width instruction (Bitwidth), a System Start instruction (System Start) or a Write policy instruction (Write Configuration) output by the neural network module based on at least one of received Power supply information (Power Sensing), Power storage information (Stored Energy Sensing), and Power failure information (Power output Sensing) at least one time, so as to ensure that the plurality of types of prediction calculation can share one neural network module, in other words, the functions of all the predictors can be completed by using one piece of prediction hardware (a neural network architecture). The hardware architecture of the neural network predictor is referred to the states shown in fig. 1, fig. 5, or fig. 6, and will not be described herein again. Under different implementation scenarios, the neural network chip can present various packaging structures according to the requirements of application to different electronic devices.

The present application also provides an electronic device (not shown) that includes the time-multiplexing-based neural network predictor described in the embodiments above. In one embodiment, the electronic device is, for example, a circuit board or a card on which an integrated circuit or a chip is disposed. The circuit board is, for example, a double-layer PCB board or a multi-layer PCB board.

Referring to fig. 13, which is a schematic diagram illustrating an architecture of a Nonvolatile processor (NVP) according to an embodiment of the present disclosure, as shown in the figure, the Nonvolatile processor 3 includes a time-multiplexing-based neural network predictor 30 and an execution module 31. In different implementation scenarios, the non-volatile processor may have a variety of packaging configurations depending on the requirements of different electronic devices.

The non-volatile processor 3 is integrated with the neural network predictor 30 based on time multiplexing, and is used for performing energy management on the electronic equipment with the processor, and comprises a neural network module and a time sequence control module.

The timing control module is configured to control a timing of prediction calculation of at least one of a data bit width instruction (Bitwidth), a System Start instruction (System Start) or a Write policy instruction (Write Configuration) output by the neural network module based on at least one of received Power supply information (Power Sensing), Power storage information (Stored Energy Sensing), and Power failure information (Power output Sensing) at least one time, so as to ensure that the plurality of types of prediction calculation can share one neural network module, in other words, the functions of all the predictors can be completed by using one piece of prediction hardware (a neural network architecture). The hardware architecture of the neural network predictor refers to the states described above with respect to fig. 1, fig. 5, or fig. 6, which are not repeated herein.

The execution module 31 is configured to perform energy management on an operation according to at least one instruction of a data bit width instruction, a start instruction, or a write strategy instruction output by the neural network predictor, or/and service quality prediction information. Referring to fig. 14, which is a schematic diagram illustrating an architecture of the nonvolatile processor according to another embodiment of the present application, as shown in the drawing, the execution module 31 includes: a start controller 310, an approximate bit width controller 311, and a retention time controller 312.

The approximate bit width controller 311 is configured to control precision of an operation according to a data bit width instruction (Bitwidth) when receiving the data bit width instruction output by the neural network predictor; in this embodiment, the approximate bit width controller 311 has one or more nonvolatile shift units, such as a nonvolatile Shifter (NV Shifter), into which it stores the received data bit width instruction. The processor is used for calculating and processing the sensing data or the interaction data acquired by the electronic equipment. In some examples, the processing of the sensed data, such as the wearable device, generates user data that may be transmitted by the wireless module or displayed by the display device by processing the collected heart rate data, blood pressure data, temperature data, blood oxygen saturation data, diet/nutrition information, medical reminders, health-related reminders or information, or other health-related data.

The boot controller 310 is configured to boot the nonvolatile processor when receiving a boot instruction output by the neural network predictor 30; in one embodiment, the start-up controller has one or more non-volatile shift cells, such as a non-volatile Shifter (NV Shifter), into which the start-up controller stores received start-up instructions. In this embodiment, the Start Controller is, for example, an NVP Start Trigger (NVP Start Trigger Controller) for controlling whether the nonvolatile processor starts to work.

The retention time controller is configured to execute a Write operation according to at least one of Write current and Write time included in a Write strategy command (Write Configuration) when receiving the Write strategy command output by the neural network predictor. In this embodiment, the retention time controller has one or more nonvolatile shift units, such as a nonvolatile Shifter (NV Shifter), into which the retention time controller stores the received write strategy instruction. The retention time controller executes a write operation of writing data according to the received write strategy instruction, and in an example, the data written to the memory of the electronic device by the write operation is, for example, calculation or processing data of the processor, or a backup calculation state is recorded.

The execution module 31 also receives quality of service prediction information (Predicted QoS) output by the neural network predictor to cause the processor to predict potential output quality of an electronic device running program based on dynamic bit width execution approximation and different approximation methods in dynamic backup data retention time, such that the results of the entire energy management system have quality of service control.

Referring to fig. 15, which is a schematic diagram of an embodiment of the electronic device 4 of the present application, as shown in the figure, the electronic device includes a feature extraction module 41, a neural network predictor 42 based on time multiplexing, and an execution module 43.

The feature extraction module 41 is configured to extract at least one of power supply information, power storage information, and power failure information of the electronic device at least one time, for example, the embodiment described with reference to fig. 2 is not repeated herein.

The neural network predictor 42 based on time multiplexing is used for energy management of an electronic device with a processor and comprises a neural network module and a time sequence control module. Wherein, the neural network module carries out prediction calculation based on at least one of the received Power supply information (Power Sensing), Power storage information (Stored Energy Sensing) and Power failure information (Power Outage Sensing) at least one moment, and outputs at least one of a data bit width instruction (BitWidth), a Start instruction (System Start) or a Write strategy instruction (Write Configuration) or/and quality of service prediction information (Predicted QoS), the data transmission bit width or the data write strategy and the operation starting time are obtained by predicting the future electric energy input or power-off time, and to give the prediction to the processor to ensure that the processor's operation matches the expected energy it has acquired, and dynamically adjusting the retention time of the nonvolatile element to match the power condition according to the write strategy, and matches the Quality of Service (QoS) with the lowest (most basic) Quality of Service requested in advance.

The execution module 43 is configured to perform energy management on an operation according to at least one instruction of a data bit width instruction, a start instruction, or a write strategy instruction output by the neural network predictor, or/and the qos prediction information, for example, for the embodiments described in fig. 7, fig. 8, and fig. 14, which is not described herein again.

Please refer to fig. 16, which is a schematic diagram of an electronic device according to another embodiment of the present application. As shown, in an embodiment, the electronic device 4 further comprises a power supply device 40 for generating or storing electrical energy. In this embodiment, the power supply device 40 is, for example, a battery or a self-powered system, which includes an energy harvester, and obtains energy from human body movement, such as vibration energy from actions or behaviors of walking or limb swinging, jumping, pressing (such as pressure obtained by a small energy harvester implanted in a shoe during running), breathing, etc., and converts the vibration energy into electric energy, and in other cases, the energy may come from the natural environment, such as solar energy, etc. The electrical energy harvested by the power supply device 40 needs to be processed from AC to DC or DC to DC and then the harvested energy is temporarily stored in off-chip or even on-chip capacitors, which are used primarily to support data rather than to store energy. In one embodiment, the feature extraction module may be part of a power supply device.

In one embodiment, as shown in fig. 16, the electronic device 4 further comprises one or more sensing devices 45, and the one or more sensing devices 45 are configured to sense at least one of geographical location information, ambient light information, ambient magnetic field information, sound information, temperature information, humidity information, stress sensing information, acceleration information, ultraviolet information, blood glucose information, alcohol concentration information, pulse information, heart rate information, respiratory information, and motion amount information.

In embodiments, the sensors 45 may include various electronic, mechanical, electromechanical, optical, or other devices that provide information related to external conditions around the wearable device. In some embodiments, the sensor may provide a digital signal to the processing subsystem, for example, on a streaming basis or in response to polling by the processing subsystem, as desired. Any type of environmental sensor and combination of environmental sensors may be used; by way of example, accelerometers, magnetometers, gyroscopes and GPS receivers are shown.

Some environmental sensors may provide information about the location and/or motion of the wearable device. For example, an accelerometer may sense acceleration (relative to free fall) along one or more axes, e.g., using piezoelectric or other components in conjunction with associated electronics to generate a signal. A magnetometer may sense an ambient magnetic field (e.g., the earth's magnetic field) and generate a corresponding electrical signal that may be interpreted as a compass direction. Gyroscopic sensors may sense rotational motion in one or more directions, for example using one or more MEMS (micro-electromechanical systems) gyroscopes and associated control and sensing circuitry. A Global Positioning System (GPS) receiver may determine a location based on signals received from GPS satellites.

Other sensors may be included in addition to or in place of these examples. For example, the sound sensor may incorporate a microphone along with associated circuitry and/or program code to determine, for example, decibel levels of ambient sound, and may also include a temperature sensor, a proximity sensor, an ambient light sensor, a biometric sensor/physiological characteristic sensor such as a heartbeat, respiration, pulse, blood glucose, alcohol concentration detection sensor, and the like. In some embodiments, physiological or biometric sensors may be used to verify the identity of the wearer of the wearable device.

In one embodiment, as shown in FIG. 16, the electronic device further includes a storage device 44 for storing data output by the processor. In some examples, the storage device 44 is, for example, a Non-volatile memory (NVM), a Read-only memory (ROM), a Random Access Memory (RAM), an EEPROM, a CD-ROM or other magnetic storage device, a magnetic disk storage device or other magnetic storage device, a flash memory, or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer. Also, any connection is properly termed a computer-readable medium.

In one embodiment, as shown in fig. 16, the electronic device further includes a wireless communication module 46 for transmitting data output by the processor or receiving data wirelessly transmitted by an external device. The communication interface of the wireless communication module 46 is such as, but not limited to, a wireless interface, a bluetooth interface, a USB interface, a Wi-Fi interface, a TCP/IP interface, a network communication interface, or any conventional communication interface.

Referring to fig. 17, which is a schematic diagram of an embodiment of the energy management system of the present application, as shown in the drawing, the energy management system 5 includes a time-multiplexing-based neural network predictor 50 and an execution module 51.

The neural network predictor 50 based on time multiplexing is used for energy management of an electronic device with a processor, and comprises a neural network module and a time sequence control module.

The execution module 51 is configured to perform energy management on an operation according to at least one instruction of a data bit width instruction, a start instruction, or a write strategy instruction output by the neural network predictor, or/and service quality prediction information. Referring to fig. 18, which is a schematic diagram illustrating an architecture of another embodiment of the energy management system of the present application, as shown in the drawing, the executing module 51 includes: a start controller 510, an approximate bit width controller 511, and a retention time controller 512.

The approximate bit width controller 511 is configured to control the precision of an operation according to a data bit width instruction (Bitwidth) when receiving the data bit width instruction output by the neural network predictor 50; in this embodiment, the approximate bit width controller 511 has one or more nonvolatile shift units, such as a nonvolatile Shifter (NV Shifter), into which the approximate bit width controller 511 stores the received data bit width instruction. The processor is used for calculating and processing the sensing data or the interaction data acquired by the electronic equipment. In some examples, the processing of the sensed data, such as the wearable device, generates user data that may be transmitted by the wireless module or displayed by the display device by processing the collected heart rate data, blood pressure data, temperature data, blood oxygen saturation data, diet/nutrition information, medical reminders, health-related reminders or information, or other health-related data.

The boot controller 510 is used to boot the nonvolatile processor when receiving the boot instruction output by the neural network predictor 50; in one embodiment, the start-up controller 510 has one or more non-volatile shift cells, such as a non-volatile Shifter (NV Shifter), into which the start-up controller 510 stores received start-up instructions. In this embodiment, the Start Controller 510 is, for example, an NVP Start Trigger (NVP Start Trigger Controller) for controlling whether the nonvolatile processor starts to work.

The retention time controller 512 is configured to execute a Write operation according to at least one of Write current and Write time included in a Write strategy command (Write Configuration) when receiving the Write strategy command output by the neural network predictor. In this embodiment, the retention time controller 512 has one or more nonvolatile shift units, such as a nonvolatile Shifter (NV Shifter), into which the retention time controller 512 stores the received write strategy instruction. The retention time controller 512 performs a write operation according to the received write strategy command, in an example, data written to the memory of the electronic device by the write operation is, for example, calculation or processing data of the processor, or records a backup calculation state, etc.

The execution module 51 also receives quality of service prediction information (Predicted QoS) output by the neural network predictor to cause the processor to predict potential output quality of an electronic device running program based on dynamic bit width execution approximation and different approximation methods in dynamic backup data retention time, such that the results of the entire energy management system have quality of service control.

In the several embodiments provided in the present application, it should be understood that the disclosed system, apparatus and method may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the units is only one logical division, and other divisions may be realized in practice, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be in the form of an indirect coupling or communication connection through some interfaces, devices or units.

The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit.

In summary, the neural network predictor, the neural network chip, the nonvolatile processor, the energy management system and the applicable electronic device thereof based on time multiplexing perform predictive computation based on at least one of received power supply information, power storage information and power failure information of the electronic device at least one moment, and output at least one of a data bit width instruction, a start instruction or a write strategy instruction and service quality prediction information; and performing energy management on the operation of the processor according to the at least one instruction and the service quality prediction information.

Moreover, the data transmission bit width or the data write strategy and the operation starting time are obtained by predicting the future electric energy input or power-off time, so that the operation of the processor can be ensured to be matched with the obtained expected energy, the retention time of the nonvolatile elements is dynamically adjusted to be matched with the electric energy condition according to the write strategy, and the service quality is matched with the minimum service quality requested in advance.

Moreover, the neural network predictor based on time multiplexing is used for realizing various prediction calculations in different time periods through the hardware architecture of one neural network predictor, in other words, the neural network predictor based on time multiplexing realizes the prediction calculations of a plurality of small-scale neural networks in different time periods, thereby achieving the purpose of saving hardware cost and area.

The above embodiments are merely illustrative of the principles and utilities of the present application and are not intended to limit the application. Any person skilled in the art can modify or change the above-described embodiments without departing from the spirit and scope of the present application. Accordingly, it is intended that all equivalent modifications or changes which can be made by those skilled in the art without departing from the spirit and technical concepts disclosed in the present application shall be covered by the claims of the present application.

Claims

1. A time-multiplexed based neural network predictor, comprising:

the neural network module is used for performing prediction calculation on at least one of received power supply information, power storage information and power failure information at least one moment, and outputting a data bit width instruction obtained through prediction calculation on the basis of the power supply information and the power storage information, a starting instruction obtained through prediction calculation on the basis of the power supply information and the power storage information, a write strategy instruction obtained through prediction calculation on the basis of the power failure information, or service quality prediction information obtained through prediction calculation on the basis of the power supply information, the power storage information and the power failure information; the data bit width instruction is an instruction for controlling the output bit width of data; the starting instruction is used for starting the processor to work; the write strategy command is a command for controlling at least one of write current and write time to execute write operation; and

a timing control module for controlling the timing of the prediction calculation of the neural network module outputting the at least one instruction or/and the qos prediction information based on the received at least one information;

the neural network module performs the following prediction calculation in a prediction calculation process based on at least one of the received power supply information, the received power storage information and the received power failure information of the electronic equipment at least one moment:

obtaining future power-on time and power-on confidence of the electronic equipment based on the power supply information through prediction calculation;

performing predictive calculation on the future power-on time, the power-on confidence coefficient and the power storage information to obtain a data bit width instruction and a starting threshold value;

obtaining service quality prediction information based on the data bit width instruction and the power failure information;

calculating a future power-off time and a power-off confidence of the electronic device based on predicting the power-off information;

and performing prediction calculation on the future power-off time and the power-off confidence coefficient to obtain a write strategy instruction and a write strategy confidence coefficient.

2. The time-multiplexing-based neural network predictor of claim 1, wherein the timing control module is configured to control the neural network module to perform prediction calculation based on at least one of the received power supply information and power storage information in one power-on cycle and output at least one of a data bit width instruction and a start instruction or/and service quality prediction information.

3. The time-multiplexing-based neural network predictor of claim 1, wherein the timing control module controls the neural network module to perform the prediction calculation timing in one power-on cycle as follows:

enabling the neural network module to carry out prediction calculation based on the received power supply information to obtain the future power-on time and the power-on confidence coefficient of the electronic equipment;

enabling the neural network module to perform prediction calculation based on the received future power-on time, power-on confidence coefficient and power storage information to obtain a data bit width instruction and a starting instruction; and

and enabling the neural network module to perform prediction calculation based on the received data bit width instruction and the power failure information to obtain service quality prediction information.

4. The time-multiplexing based neural network predictor of claim 3, wherein the timing control module is further configured to control the neural network module to update the power-off information within one power-on cycle.

5. The time-multiplexing-based neural network predictor of claim 1, wherein the timing control module is configured to control the neural network module to perform prediction calculation and output a write strategy command based on the received power outage information in one power outage cycle.

6. The time-multiplexing-based neural network predictor of claim 5, wherein the timing control module is configured to control the neural network module to perform the prediction calculation at a power-off cycle with a timing of:

causing the neural network module to calculate a future outage time and an outage confidence of the electronic device based on predicting the outage information; and

and enabling the neural network module to perform prediction calculation on the future power-off time and the power-off confidence coefficient to obtain a write strategy instruction and a write strategy confidence coefficient.

7. The time-multiplexing-based neural network predictor of claim 6, wherein the timing control module is further configured to control the neural network module to update the power supply information within a power-off period.

8. The time-multiplexing based neural network predictor of claim 1, wherein the neural network module is a feed-forward neural network module comprising:

the neural network unit comprises a neuron register, a weight register in which a plurality of weights are stored, a plurality of selectors and a multiplication accumulation unit; and

and the single prediction state machine is used for controlling the time sequence of the neural network unit receiving the at least one type of information to perform single prediction calculation.

9. The time-multiplexing based neural network predictor of claim 8, wherein the weight register comprises a non-volatile storage unit for storing weights.

10. The time-multiplexing based neural network predictor of claim 8, wherein the one-shot prediction state machine comprises a non-volatile shift unit or a non-volatile storage unit for storing a timing control program of one-shot prediction calculations.

11. The time-multiplexing-based neural network predictor of claim 1, further comprising a nonvolatile shift unit or a nonvolatile storage unit for storing at least one of power supply information, power storage information and power failure information of the electronic device at one or more time points.

12. The time-multiplexing based neural network predictor of claim 1, wherein the timing control module comprises a non-volatile shift unit or a non-volatile storage unit for storing a timing control program.

13. A neural network chip comprising a time-multiplexing based neural network predictor according to any one of claims 1-12.

14. An electronic apparatus comprising a time-multiplexing based neural network predictor according to any one of claims 1-12.

15. An electronic device, comprising

The characteristic extraction module is used for extracting at least one of power supply information, power storage information and power failure information of the electronic equipment at least one moment;

the time-multiplexing based neural network predictor of any one of claims 1-12; and

and the execution module is used for carrying out energy management on the operation according to at least one instruction of a data bit width instruction, a starting instruction or a write strategy instruction output by the neural network predictor or/and service quality prediction information.

16. The electronic device of claim 15, wherein the feature extraction module comprises:

the detection unit is used for acquiring power supply information of the electronic equipment by acquiring a current value flowing through a detection element or voltage values at two ends of the detection element;

the energy storage unit is used for acquiring the electricity storage information of the electronic equipment by acquiring the voltage drop of an energy storage element; and/or the power failure sensing unit is used for acquiring the power failure information by acquiring the voltage drop at two ends of a power leakage element.

17. The electronic device of claim 15, further comprising power supply means for generating or storing electrical energy.

18. The electronic device of claim 15, further comprising one or more sensing devices for sensing at least one of geographic location information, ambient light information, ambient magnetic field information, sound information, temperature information, humidity information, stress-induced information, acceleration information, ultraviolet information, blood glucose information, alcohol concentration information, pulse information, heart rate information, respiration information, and motion amount information.

19. The electronic device of claim 15, further comprising a storage device for storing data output by the processor.

20. The electronic device of claim 15, further comprising a wireless communication module for transmitting data output by the processor or receiving data wirelessly transmitted by an external device.

21. The electronic device of claim 15, wherein the execution module is a non-volatile processor.

22. The electronic device of claim 15, wherein the electronic device is a wearable electronic device or a human implanted device.

23. A non-volatile processor, comprising:

the time-multiplexing based neural network predictor of any one of claims 1-12;

24. The non-volatile processor of claim 23, wherein the execution module comprises:

the approximate bit width controller is used for controlling the precision of operation according to the data bit width instruction when receiving the data bit width instruction output by the neural network predictor;

the starting controller is used for starting the work of the nonvolatile processor when receiving a starting instruction output by the neural network predictor;

and the retention time controller is used for executing write operation according to at least one information of write current and write time contained in the write strategy command when receiving the write strategy command output by the neural network predictor.

25. The non-volatile processor of claim 23, wherein the execution module comprises one or more non-volatile shift units to store at least one of the data bit width instructions, the boot instructions, or the write strategy instructions, or/and the qos prediction information.

26. An energy management system for use in an electronic device having a processor, comprising:

the time-multiplexing based neural network predictor of any one of claims 1-12;

and the execution module is used for carrying out energy management on the operation of the processor according to at least one instruction of a data bit width instruction, a starting instruction or a write strategy instruction output by the neural network predictor or/and service quality prediction information.

27. The energy management system of claim 26, wherein the execution module comprises:

the starting controller is used for starting the processor to work when receiving a starting instruction output by the neural network predictor;

28. The energy management system of claim 26, wherein the execution module comprises one or more non-volatile shift units to store at least one of the data bit width instructions, the boot instructions, or the write strategy instructions, or/and the qos prediction information.