CN110074776B

CN110074776B - Power consumption control method and system of artificial intelligence dynamic heart and lung monitoring equipment

Info

Publication number: CN110074776B
Application number: CN201910362918.3A
Authority: CN
Inventors: 马振宇
Original assignee: Guangdong Lezhikang Medical Technology Co ltd
Current assignee: Guangzhou Yunxin Information Technology Co ltd
Priority date: 2019-04-30
Filing date: 2019-04-30
Publication date: 2020-04-10
Anticipated expiration: 2039-04-30
Also published as: WO2020220661A1; CN110074776A

Abstract

The invention relates to a power consumption control method and a system of artificial intelligence dynamic heart-lung monitoring equipment, by the method and the system, computing resource requirements and completion time are taken as control limits, power consumption is taken as a control index, and computing resources of each layer are subjected to predictive computation and taken as a decision basis for system scheduling; the control decision calculation module can be continuously optimized through closed-loop control, the maximum energy efficiency of the dynamic heart-lung monitoring equipment system is realized, and therefore the artificial intelligent edge calculation equipment using a set of minimum hardware circuit resources meets various monitoring scenes and has the longest equipment standby time.

Description

Power consumption control method and system of artificial intelligence dynamic heart and lung monitoring equipment

Technical Field

The invention relates to the field of electrocardio-heart sound measurement, in particular to a power consumption control method of artificial intelligent dynamic heart-lung monitoring equipment.

Background

Firstly, wearable mobile artificial intelligence terminal equipment is still in the early development stage at present, and most of the terminal equipment collects data and then sends the data to a server background through various communication modes for artificial intelligence calculation. However, as the requirements of stable bandwidth of communication, data security, local personalized real-time processing and the like cannot be solved through cloud computing, a part of real-time and personalized functions are required to be embedded into an equipment terminal for artificial intelligence reasoning, and therefore edge computing becomes a basic technical application in the field of mobile internet of things.

At present, few chips and hardware circuits can be used for edge calculation, the main concern is how to improve calculation power, and the most important scheme for power consumption control depends on the design of an artificial intelligent chip, such as a type-selection low-power consumption MCU, an embedded low-power consumption FPGA and the like. Another way is to borrow the traditional power control way, such as entering sleep mode, turning off the peripheral hardware not used, etc.

In summary, no real power consumption optimization control scheme exists at present.

Second, at present, the electrocardio and heart sound equipment commonly used in hospitals is a 12-lead dynamic detector (holter), the holler function is single, and in the same time period, either electrocardiosignals or heart sound signals of a user are collected, so that the concurrence condition can not be judged in a centralized manner by comprehensively monitoring a plurality of signals of the user. The power consumption of Hotler is bigger, and the volume is also very big, generally operates under doctor's instruction when wearing, and the user live time is short under the full charge condition. For a user needing to continuously acquire electrocardio and heart sound signals (the condition of part of the heart is only 10 minutes from the detection of abnormality to the very bad condition), the electrocardio and heart sound data also need to be continuously acquired. Therefore, the current equipment cannot meet the medical continuous detection requirement of 3-7 days. In addition, the current Holter electrocardio monitoring and judging needs to transmit data to a background server for calculation, the calculation lag is obvious, and the requirements of cardiopulmonary real-time calculation and all-weather instant early warning cannot be met.

Disclosure of Invention

In view of this, the present invention provides a power consumption control method for an artificial intelligence dynamic cardiopulmonary monitor, which adopts a multi-mode control idea, calculates an optimal artificial intelligence hardware resource allocation strategy according to the requirement of artificial intelligence edge calculation, and implements the optimal artificial intelligence hardware resource allocation strategy on a device, thereby achieving the purpose of controlling the power consumption of the whole machine.

The specific technical scheme of the invention is as follows:

a power consumption control method of an artificial intelligent dynamic electrocardiogram detector,

receiving a multi-modal data input, the multi-modal data input comprising at least electrocardiographic signals;

establishing an AI inference model according to the multi-modal data input;

obtaining parameter requirements of the AI inference models of each application scene according to the AI inference models, wherein the parameter requirements at least comprise computing resource requirements and completion time requirements T0 of the application scenes;

performing multi-mode control decision calculation according to the parameters of the AI inference model to obtain modal control parameter configuration, and performing power consumption and time calculation based on the modal control parameter configuration to obtain simulated power consumption Pf and simulated time Tf;

performing multi-mode calculation unit calculation according to the modal control parameter configuration, and performing multi-mode data fusion of the multi-mode calculation unit; the multi-mode calculation unit at least comprises an electrocardio calculation unit and a cardiopulmonary sound calculation unit;

monitoring actual power consumption and actual time by calculating the multi-mode calculating unit to obtain actual power consumption Ps and actual time Tr;

and in the completion time requirement T0, the simulation time Tf and the actual time Tr are fed back to the multi-modal control decision calculation for continuous optimization by taking the lowest power as a control index, so that the optimal completion time and the optimal hardware resources are obtained.

Furthermore, after the local calculation result obtained after the multi-mode data fusion is output, the parameter requirements of the artificial intelligent reasoning model are adjusted according to different requirements, and meanwhile, closed-loop parameters are updated according to different application scenes.

Further, the multi-modal data input includes at least two of an electrocardiograph signal, a heart sound signal, a respiration sound signal, a thoracic impedance signal, and a blood oxygen signal.

Further, obtaining the parameter requirement of the AI inference model comprises:

and performing edge calculation and lightweight evaluation of a deep learning model according to the AI inference model.

Further, the edge calculation adopts a light-weighted GRU or LSTM model.

Further, the multi-modal control decision calculation includes:

acquiring the computing resource requirement and the completion time requirement in an application scene;

reading system hardware in the computing resource requirement, and carrying out decreasing grading on the resources of the system hardware;

calculating the time for completing the simulation calculation under the grading according to the grading sequence, if the time for completing the simulation calculation is lower than the completion time requirement T0, calculating the time for completing the simulation calculation under the resources of the system hardware of the next grade until the simulation time of the simulation calculation of a certain grade is higher than the completion time requirement T0;

and if the time for completing the simulation calculation under the last-stage classification is still lower than the completion time requirement, the multi-modal control decision calculation is output by adopting the resources of the system hardware of the last stage.

Further, according to the comparison between the simulated power consumption of the hardware resource subjected to the simulation calculation in the calculation resource requirements in the multi-modal control decision calculation and the corresponding actual power consumption and the comparison between the simulated time and the actual time, the parameters of the hardware resource subjected to the simulation calculation are optimized to obtain the parameters of the optimal hardware resource, and the optimal completion time is obtained according to the parameters of the optimal hardware resource.

Further, the system hardware comprises an AI data processing unit, and the AI data processing unit comprises an SOC chip and an FPGA chip;

the SOC chip comprises an MCU control unit and a computing unit, wherein the computing unit comprises a parallel RISC-V computing block with a plurality of cores and a hardware convolution computing engine (HWCE for short);

the FPGA chip comprises a plurality of parallel programmable engines;

the hierarchy includes at least all hardware resources, second level hardware resources, and minimum hardware resources decremented;

all hardware resources comprise all the AI data processing units, and under the simulation calculation of all the hardware resources, if the time for completing the simulation calculation of all the data is higher than or equal to the completion time requirement T0, the judgment that the model or the hardware parameter is wrong is made, and the system reports the error; if the time for completing all data simulation calculation is lower than the time requirement T0, switching to a second-level hardware resource for simulation calculation;

the second level hardware resource comprises the SOC chip; under simulated computation of the second level hardware resources:

if the time for completing the simulation calculation of all the data is higher than or equal to the requirement of the completion time, transmitting part of data (such as heart sound data) which needs a large amount of hardware resources and has low real-time requirement to the FPGA chip for calculation, gradually reducing the frequency of the FPGA chip, calculating the time T61 for the FPGA chip to complete the simulation calculation of the part of the data with the lowest power consumption, so that the time for the simulation calculation is less than or equal to the requirement of the completion time T0 (in the embodiment, the time for the SOC part is required to be explained to be less than the time of the FPGA), and calculating the sum of the power of the SOC chip and the FPGA chip at the moment as the simulation power consumption; if the time for completing all data simulation calculation is lower than the time requirement, switching to the least hardware resources for simulation calculation;

the minimum hardware resources comprise a hardware convolution calculation engine of the SOC chip; under the condition that the minimum hardware resources are subjected to simulation calculation with the maximum dominant frequency:

if the time for completing all data simulation calculation is higher than or equal to the completion time requirement, a parallel RISC-V calculation block with a plurality of cores and added into the SOC chip perform simulation calculation together with the hardware convolution calculation engine, find out the lowest dominant frequency for completing all data simulation calculation according to the frequency-voltage curve (inherent) of the calculation unit of the SOC chip and obtain the voltage corresponding to the lowest dominant frequency, so that the time T51 of the simulation calculation is less than or equal to the completion time requirement T0, and calculate the sum of the power consumption of the calculation unit of the SOC chip and the hardware convolution calculation engine as the simulation power consumption; if the time for completing the simulation calculation of all the data is lower than the completion time requirement, under the calculation of the pure hardware convolution calculation engine, the lowest dominant frequency for completing the simulation calculation of all the data is found out according to the frequency-voltage curve (inherent) of the hardware convolution calculation engine of the SOC chip, the voltage corresponding to the lowest dominant frequency is obtained, the time T41 of the simulation calculation is smaller than or equal to the completion time requirement, and the power consumption of the hardware convolution calculation engine of the SOC chip at the moment is calculated to be used as the simulation power consumption.

Further, an initial time margin Tm is set, and the Tm is added to the time Tx1 calculated by the simulation to obtain a simulation time Tf, i.e., the Tm is added to the times T61, T51 and T41 as simulation times T6, T5 and T4, wherein x =4 or 5 or 6.

Further, the continuous optimization steps of the multi-modal control decision calculation are as follows:

if the actual time Tr is less than or equal to the completion time requirement T0, determining whether the actual time Tr is less than the simulation time Tf:

if the actual time Tr is smaller than the simulation time Tf, determining whether the actual time is smaller than the simulation calculated time Tx 1:

if the actual time Tr is less than the time Tx1 of the simulation calculation, assigning the actual time Tr to the time Tx1 of the simulation calculation, and recalculating the simulation time Tf, and when Tr < Tx1min, the time Tx1= Tx1min of the simulation calculation, wherein Tx1min is the time of the simulation calculation with the least hardware resources;

if the time Tx1 is greater than or equal to the simulation calculation time, assigning the actual time to the simulation time, then subtracting the initial time margin Tm from the simulation time to obtain an intermediate time Tz, and increasing the frequency of corresponding hierarchical hardware resources by taking the intermediate time Tz as a control object so that the simulation calculation time approaches to the intermediate time Tz; then, the multi-mode calculation unit is calculated according to the enlarged parameters of the corresponding hierarchical hardware resources to obtain updated actual time Tr 1; adjusting the initial time margin Tm to obtain an updated time margin Tm 1; and giving the updated actual time Tr 1to the simulation time Tf, replacing the initial time margin Tm with the updated time margin Tm1, and repeating the calculation until the updated actual time falls into [ 1-k ] T0 and T0 ], wherein k is an error coefficient, and the range of k is 1-5%.

if the actual time Tr is less than or equal to the completion time requirement T0, taking the actual time as the optimal completion time, and taking the hardware and parameters corresponding to the actual power consumption as optimal hardware resources;

if the actual time is greater than the completion time request T0 and the actual time is greater than the simulation time, increasing the initial time margin Tm until the actual time is less than the simulation time;

if the actual time is greater than the completion time requirement T0 and the actual time is less than the simulation time, assigning the actual time to the simulation time, then subtracting the initial time margin Tm from the simulation time to obtain an intermediate time Tz, and increasing the frequency of the corresponding hierarchical hardware resources by using the intermediate time Tz as a control object, so that the time of the simulation calculation approaches the intermediate time Tz (approaches within a Tz time period that can be understood as ± 5%); then, the multi-mode calculation unit is calculated according to the enlarged parameters of the corresponding hierarchical hardware resources to obtain updated actual time Tr 1;

adjusting the initial time margin Tm to obtain an updated time margin Tm 1;

and giving the updated actual time Tr 1to the simulation time, replacing the initial time margin Tm with the updated time margin Tm1, and repeating the calculation of the stage until the updated actual time falls into [ 1-k ] T0 and T0 ], wherein k is an error coefficient, and the range of k is 1-5%.

An artificial intelligence dynamic electrocardio detection system comprises a multi-mode signal acquisition unit, an AI data processing unit, a power management unit, a wireless communication unit, a voice unit and a man-machine operation interface, wherein the AI data processing unit and the power management unit implement the method.

An artificial intelligence dynamic electrocardio monitoring system comprises an application layer, a system management layer and a data processing layer;

the application layer comprises a human-computer interaction unit;

the system management layer comprises a power management unit and a main control unit;

the data processing layer comprises a multi-mode data acquisition unit, a data preprocessing unit, an SOC low-power consumption real-time AI calculation unit, an FPGA big data non-real-time calculation unit, a calculation result fusion unit and a wireless communication unit;

the system management layer and the data processing layer implement the above-described method.

According to the technical scheme, the computing resource requirements and the completion time are taken as control limits, the power consumption is taken as a control index, and the computing resources of each layer are subjected to predictive computation and taken as a decision basis for system scheduling; the control decision calculation module can be continuously optimized through closed-loop control, the maximum energy efficiency of the dynamic heart-lung monitor system is realized, and therefore the artificial intelligent edge calculation equipment using a set of minimum hardware circuit resources meets various monitoring scenes and has the longest equipment standby time.

Drawings

FIG. 1 is a diagram of an artificial intelligence dynamic electrocardiographic detection system of the present invention.

Fig. 2 is a wearing diagram of the multi-modal signal acquisition unit of the present invention, which includes a control box integrated with an AI data processing unit at the lower right corner.

Fig. 3 is a schematic diagram of a portion of the modal signal acquisition of the present invention.

FIG. 4 is a software system framework diagram of the present invention.

Fig. 5 is an AI inference model diagram according to different application scenarios of the present invention.

FIG. 6 is a flowchart illustrating edge computation for the AI inference model according to the present invention.

Fig. 7 is an edge calculation model parameter obtained by performing a deep learning model lightweight evaluation on the processing results of fig. 5 and 6.

Fig. 8 is a schematic diagram of a chip architecture of the SOC chip.

Fig. 9 is a schematic diagram of a chip architecture of an FPGA chip.

FIG. 10 is a level diagram of computing resource and power consumption control of the wearable device of the present invention.

FIG. 11 is an example of power consumption evaluation of an SOC chip with different computing resources for different application scenario requirements.

Fig. 12 shows the operation speed and power consumption of the SOC chip corresponding to different computing resources.

FIG. 13 illustrates the effect of different dominant frequencies and voltages on power consumption for the same computing resource.

Fig. 14 is a flowchart of a power consumption control method according to the present invention.

FIG. 15 is a flow chart of a multi-modal control decision calculation of the present invention.

FIG. 16 is a time-based power consumption control feedback diagram of the present invention.

Detailed Description

In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below, and it is obvious that the described embodiments are a part of the embodiments of the present invention, but not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

The terminology used in the embodiments of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used in the examples of the present invention and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms as well, and "a" and "an" generally include at least two, but do not exclude at least one, unless the context clearly dictates otherwise.

It should be understood that the term "and/or" as used herein is merely one type of association that describes an associated object, meaning that three relationships may exist, e.g., a and/or B may mean: a exists alone, A and B exist simultaneously, and B exists alone. In addition, the character "/" herein generally indicates that the former and latter related objects are in an "or" relationship.

It should be understood that although the terms first, second, third, etc. may be used to describe … … in embodiments of the present invention, these … … should not be limited to these terms. These terms are used only to distinguish … …. For example, the first … … can also be referred to as the second … … and similarly the second … … can also be referred to as the first … … without departing from the scope of embodiments of the present invention.

The words "if", as used herein, may be interpreted as "at … …" or "at … …" or "in response to a determination" or "in response to a detection", depending on the context. Similarly, the phrases "if determined" or "if detected (a stated condition or event)" may be interpreted as "when determined" or "in response to a determination" or "when detected (a stated condition or event)" or "in response to a detection (a stated condition or event)", depending on the context.

It is also noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a good or system that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such good or system. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a commodity or system that includes the element.

In addition, the sequence of steps in each method embodiment described below is only an example and is not strictly limited.

As shown in fig. 1, the hardware system of the present invention mainly includes a multi-mode signal acquisition unit, an AI data processing unit, a power management unit, a battery, a man-machine operation interface, a wireless communication unit, and a voice unit.

a) Multi-mode signal acquisition module

The multi-mode signal acquisition module integrates twelve-lead electrocardiosignal acquisition and thoracic impedance signal acquisition and can acquire heart and lung sound signals and blood oxygen signals.

The front end of the twelve-lead electrocardiosignal and thoracic impedance signal acquisition circuit is a flexible sensor electrode, the rear end of the twelve-lead electrocardiosignal and thoracic impedance signal acquisition circuit adopts an Analog-to-Digital Converter (ADC) chip with low power consumption and special for ECG acquisition, the chip integrates a filtering module, and the twelve-lead electrocardiosignal and thoracic impedance signal acquisition circuit has the characteristics of high precision, low power consumption, high integration level and the like and simultaneously has thoracic impedance signal acquisition.

Heart and lung sounds are processed by a special pickup cavity and an amplifying and filtering circuit by adopting an MEMS (Micro-Electro-Mechanical System) sensor technology, sound signals of lung breathing of a patient are collected in a high fidelity mode, the sound signals are converted into digital signals through ADC (analog to digital converter), data are analyzed and processed by a data module, and detection, diagnosis, early warning and intervention on lung abnormality are realized through artificial intelligence algorithm reasoning.

The blood oxygen signal acquisition adopts photoelectric sensor, the blood oxygen concentration information of direct acquisition human skin, can gather the rhythm of the heart signal simultaneously, and the signal passes through low pass filter and analog-to-digital conversion module, becomes digital signal and gives data processing module analysis.

As shown in fig. 2, it is a preferred device of the multi-modal signal acquisition unit of the invention, specifically a wearable mobile electrocardiograph monitoring device. The equipment can simultaneously acquire 12-lead electrocardiosignals, heart sound signals, breathing sound signals and thoracic impedance signals. Of course, the photo sensor may be integrated to acquire the blood oxygen signal. The device is small in size, light in weight, portable, comfortable to wear, low in power consumption, capable of achieving local artificial intelligence calculation function and capable of achieving real-time monitoring and early warning. An AI data processing unit is integrated in the control box in the lower left corner of fig. 2. The device is hardware for collecting data, and is not the core protected by the application, and the applicant can refer to the device in other applications.

The data processing of the multi-modal signal acquisition unit is shown in fig. 3. After multimodal data acquisition, multimodal biometric identification is performed. The multi-mode biological recognition means that two or more biological recognition characteristics are integrated or fused, the unique advantages of the multiple biological recognition technology are utilized, and the data fusion technology is combined, so that the authentication and recognition process is more accurate and safer. The multi-mode biological recognition method is mainly different from the traditional single biological recognition method in that the multi-mode biological recognition technology can collect different biological characteristics through independent or multiple collection methods which are combined into one collector, and carries out recognition and authentication through analyzing and judging characteristic values of the multiple biological recognition methods. Thus, multimodal biometric identification techniques have unique advantages over single biometric identification: first, extremely high reliability and safety. The multi-mode biological recognition generally integrates or fuses two or more biological recognition modes, and is safer and more reliable compared with a single biological recognition technology, and the accuracy of an authentication result is ensured by multiple authentication processes; and secondly, the system is customized individually to meet different requirements of users. The user can select different biological identification technologies according to the application environment, so that the inherent defect of a single biological identification technology can be effectively overcome, the dependence degree of biological identification on the environment is greatly reduced, and different business requirements of the user are met more confidently;

b) data processing and master control unit

The data processing unit adopts a structure of combining a high-performance FPGA (Field-Programmable gate array) and a low-power SOC (System-on-a-Chip) Chip in the latest process at present, and expands a DRAM (dynamic random access memory); the FPGA is internally provided with tens of thousands of logic units and multipliers and designed into an artificial intelligence data processing unit and a hardware accelerator which are calculated in parallel by selecting a proper framework, the highest single-chip processing efficiency can reach more than 1TOPS/W, the computing capability is strong, the heart and lung signal processing and artificial intelligence algorithm reasoning can be rapidly carried out, and the abnormity early warning and the intervention on the heart and lung functions can be carried out in real time. In addition, an ultra-low power consumption SOC chip with a neural network computing function is used as a main controller to form a main control unit, signals are preprocessed, SOC power consumption is low, standby power consumption can reach below 1mW, electrocardio data can be collected and processed in real time and cached, IO management of a system can be achieved, and an FPGA is matched to operate a cardiopulmonary analysis artificial intelligence algorithm, so that the power consumption of the whole machine is lower.

c) Human-computer interaction unit

The man-machine operation interface adopts a small-size TFT (thin Film transistor) touch screen, and can check electrocardiogram data and set operation parameters in real time.

The voice module integrates a microphone and a loudspeaker, can remind a patient and carry out two-way conversation by voice in real time so as to carry out active intervention when abnormal early warning of cardiopulmonary signals occurs.

d) Wireless communication unit

The Bluetooth module is integrated, meanwhile, the WIFI, NB-IOT and 3G/4G communication functions can be expanded, and the optimal communication mode can be set and selected through the configuration of a software system.

e) Battery and power management unit

And a large-capacity lithium battery is adopted, so that the terminal can continuously work for more than 48 hours. The power management module manages charging and discharging of the battery, accurately detects the residual electric quantity and ensures that the instrument works safely and stably according to the preset function. The power management unit only needs to adopt a common power chip.

The software system of the whole wearable device is shown in fig. 4, and comprises an application layer, a system management layer and a data processing layer. The functional module mainly comprises a main control unit and an operation unit, and the hardware is adjacent to the kick unit and comprises an SOC control unit, an SOC calculation unit and an FPGA chip.

a) Main control unit

The main control unit runs on an SOC control chip and performs multi-mode task control and calculation data scheduling based on a real-time lightweight embedded operating system (such as a FREERTS) to realize the optimal control of the power consumption and the parallel operation of the system.

b) Arithmetic unit

The operation unit has two parts, one part runs in a low-power SOC neural network module and mainly performs 12-lead ECG calculation with higher real-time requirement; the other part runs in an FPGA high-performance artificial intelligence operation module, and mainly calculates heart and lung sounds and other signals which have low real-time requirements but large data volume. By means of the distributed computing scheduling, the real-time performance of multi-mode data can be guaranteed, and meanwhile power consumption is the lowest.

As shown in fig. 5, after receiving signals at least including electrocardiographic data, respiratory sounds and heart sound data, establishing an AI inference model, for example, for an application scenario of heart failure, 3-6 lead electrocardiographic signals among heart sound signals, thoracic impedance signals and 12 leads are required; for example, for the application scenario of atrial fibrillation, 3-lead electrocardiogram data is needed; for the application scene of sudden death, 12-lead electrocardiosignals and heart sound signals are needed; for example, for myocardial infarction or coronary heart disease, 12 leads of electrocardiosignals are needed. For different application scenarios, the model can be selected in a data center with multiple modal inputs in a targeted manner, so that an accurate model in the application scenario is obtained.

The flow of the artificial intelligence edge calculation model of the whole equipment is shown in the attached figure 5, and the edge calculation model running on the equipment is shown in the attached figure 6:

this system can run any artificial intelligence model that meets the needs, because the device is mainly directed to cardiopulmonary signals, and therefore this embodiment uses a lightweight GRU or LSTM model (both models perform similarly, both are variants of RNN).

Fig. 7 shows the demand of the computational resource of the system after the weight reduction of the computational model of the heart sound monitoring part of the present invention, which is the input benchmark for the subsequent optimization of the computational resource. In the multi-modal data input of the device, heart sounds are the part with the largest calculation amount, but the requirement on real-time performance is not high, and a chip with strong calculation capacity is required for calculation. The multi-lead electrocardiosignal has the characteristics of high real-time requirement and continuous calculation, but the waveform is not complex, the sampling frequency is low, and the multi-lead electrocardiosignal is relatively suitable for multi-path calculation resources with low power consumption and general calculation capability to carry out calculation.

The invention is based on the Greenwave artificial intelligence SOC chip and the Xilinx FPGA chip. FIG. 8 is a schematic diagram of a Greenwave GAP chip architecture diagram. Fig. 9 is a schematic diagram of an artificial intelligence FPGA (based on Ku115 chip) architecture of deep technology under Xilinx.

As shown in the architecture of fig. 8, the SOC chip has an MCU control unit, which is called Fabric controller (FC for short). The computing unit is also called cluster, comprises a parallel RISC-V computing unit with 8 cores and can be used for accelerating artificial intelligence computing; and a HardWare convolution calculation engine, known as HardWare ConvolationEngine (HWCE). The chip has the characteristics of ultra-low power consumption control capability, and the artificial intelligence calculation acceleration module can be used for common calculation, which is the control key point of the embodiment. With respect to the Greenwave-based artificial intelligence SOC chip, reference may be made to "2018 IEEE 29^thInternational Conference on Application-specificSystems，Architectures and Processors（ASAP），10-12 JULY 2018，DOI：10.1109/ASAP.2018.8445101”。

As shown in the architecture of fig. 9, the FPGA chip performs targeted architecture optimization for artificial intelligence calculation, particularly LSTM algorithm, and improves calculation bandwidth and operation efficiency effectively by a special sparse LSTM module. Meanwhile, similar to the architecture of the SOC, the FPGA designs a plurality of parallel HW/SW programmable engines (PE for short), and each PE computing unit is composed of thousands of MAC dsp cores (4096 constituent MAC communications working at 500MHz in this embodiment, and theoretical peak computing power is 4 Tflops). The FPGA is characterized by strong computing power but larger power consumption. Therefore, the power consumption can be controlled without starting the power supply. The FPGA in the embodiment is mainly used for calculating the heart sound signals according to the requirements of an application model. With respect to the architecture of an FPGA, reference may be made to http:// www.deephi.com/technology.

Fig. 10 shows a computing resource and power consumption control hierarchy of the wearable device of the present embodiment. Because the input of the present device is a multimodal source of data input, it is not necessary for applications of cardiopulmonary monitoring to start all of the computing resources at a time to perform the calculations. For example, atrial fibrillation, requires only 12-lead electrocardiographic waveforms and does not require heart sound detection; while heart failure requires heart sound detection, it is sufficient to have 6-lead electrocardiographic waveforms. Therefore, the use of computing resources can be allocated according to the requirements of the model of the application end, and the purpose of optimizing the power consumption of the system is achieved.

The computing resources of the system include the chip level SOC and FPGA. The next level is the computation modules inside the two chips, which can be controlled individually. Meanwhile, the calculation main frequency and voltage of the chip are also important factors influencing power consumption, so the method is also a control object of the method.

FIG. 11 illustrates an example of power consumption assessment of different computing resources for different application model requirements. This table illustrates that different applications may have different computing resources and different power consumptions. It is to be noted here that the calculation speed and the power consumption themselves do not correspond exactly. We can see that the speed is improved by 10.9 times after the CIFAR-10 invests all the computing resources, but the power consumption is reduced by only 4.9 times. So sometimes in applications where the speed requirement is not so high, not all computing resources need to be invested, but rather better energy efficiency can be achieved.

Fig. 12 shows the operation speed and power consumption corresponding to different computing resources. The table is based on the same application model, different computing resources are invested, and the generated energy efficiency ratios are different. It is particularly evident that the first and second rows, with the same 8-core cluster, have a reduction in the dominant frequency from 175Mhz to 15.4Mhz, which is an 11.4-fold reduction, but a reduction in power consumption from 70mW to 3.7mW, which is an 18.9-fold reduction. It is verified again that the lowest power consumption can be achieved by reducing the dominant frequency, if the speed is met.

FIG. 13 illustrates the effect on power consumption of the same computing resource using different dominant frequencies and voltages. This table also illustrates that reducing the operating frequency and voltage has a positive effect on the reduction of power consumption.

In conjunction with the above description of FIGS. 5-13, the present invention provides a method for multi-modal power consumption control for an artificial intelligence edge computing device. The basic idea is to adopt control modes of different scheduling strategies according to the operation requirements or states of the system, namely to schedule the most appropriate resources under different operation conditions according to the actual operation state of the system, so that the highest energy efficiency after the combination of the timeliness and the power consumption of the system calculation is realized.

Specifically, as shown in fig. 14, the method may be a closed-loop control method, and first obtain the computing resource requirement and the completion time requirement of the application model, and then obtain the optimal computing resource scheduling through multi-modal control decision calculation. And after entering the system, the system actually runs, and simultaneously, the actual power consumption and the calculation result of the system are evaluated through monitoring, fed back to the control decision calculation module and continuously optimized.

The method comprises the steps of firstly, receiving multi-mode data input, wherein the multi-mode data input can comprise at least two of electrocardiosignals, heart sound signals, breathing sound signals, thoracic impedance signals and blood oxygen signals.

And step two, establishing an AI inference model according to the multi-mode data input. The application scenarios and objects to which the AI inference model applies have been elaborated in the description with respect to fig. 5.

And step three, obtaining parameter requirements of the AI inference models of each application scene according to the AI inference models, wherein the parameter requirements at least comprise the computing resource requirements and the completion time requirements T0 of the application scenes. The invention takes power consumption as a control object, and achieves the lowest power consumption under the condition of meeting the time requirement by controlling the time requirement.

And fourthly, performing multi-mode control decision calculation according to the parameters of the AI inference model to obtain modal control parameter configuration, and performing power consumption and time calculation based on the modal control parameter configuration to obtain simulated power consumption and simulated time. The part is the key control step of the invention, and the calculation is continuously optimized on the basis of the requirement of the completion time through an empirical value and a preset time margin.

Fifthly, calculating a multi-mode calculating unit according to the modal control parameter configuration, and performing multi-mode data fusion of the multi-mode calculating unit; the multi-mode calculation unit at least comprises an electrocardio calculation unit and a cardiopulmonary sound calculation unit;

sixthly, monitoring actual power consumption and actual time of the calculation of the multi-mode calculation unit to obtain the actual power consumption and the actual time;

and step seven. And in the completion time requirement T0, the simulation time and the actual time are fed back to the multi-modal control decision calculation for continuous optimization by taking the lowest power as a control index, so as to obtain the optimal completion time and the optimal hardware resources.

In the feedback in fig. 14, there is actually time feedback (not shown in the feedback in fig. 14, actually present), power consumption is a target, and time is a measure.

And step eight, after local calculation results obtained after multi-mode data fusion are output, adjusting the parameter requirements of the artificial intelligent reasoning model according to different requirements, and updating closed-loop parameters according to different application scenes.

Obtaining the parameter requirements of the AI inference model comprises: and performing edge calculation and lightweight evaluation of a deep learning model according to the AI inference model. And the edge calculation adopts a lightweight GRU or LSTM model.

As shown in fig. 15, the multi-modal control decision calculation includes:

And optimizing the parameters of the hardware resources subjected to simulation calculation according to the comparison between the simulation power consumption of the hardware resources subjected to simulation calculation in the calculation resource requirements in the multi-modal control decision calculation and the corresponding actual power consumption and the comparison between the simulation time and the actual time to obtain the parameters of the optimal hardware resources, and obtaining the optimal completion time according to the parameters of the optimal hardware resources.

One embodiment for the above multi-modal control decision calculation is as follows:

the system hardware includes an AI data processing unit including an SOC chip shown in fig. 8 and an FPGA chip shown in fig. 9.

if the time for completing the simulation calculation of all the data is higher than or equal to the completion time requirement, transmitting part of data (such as heart sound data) which needs a large amount of hardware resources and has low real-time requirement to the FPGA chip for calculation, gradually reducing the frequency of the FPGA chip, calculating the time T61 for completing the simulation calculation of the part of data by the FPGA chip with the lowest power consumption, enabling the time for the simulation calculation to be less than or equal to the completion time requirement T0, and calculating the sum of the power of the SOC chip and the power of the FPGA chip at the moment as the simulation power consumption; and if the time for completing all data simulation calculation is lower than the time requirement, switching to the least hardware resources for simulation calculation.

Since the SOC performs real-time calculation, in practice, the SOC must have completed real-time calculation in the SOC chip within the time the FPGA completes calculation.

Similarly, when computing with multiple cores, we default to the hardware convolution compute engine having completed the computation.

In the calculation process, according to the characteristics of the system, the invention creatively sets an initial time margin Tm which is added on the basis of the T61, T51 and T41 to be used as simulation times T6, T5 and T4. The simulation time is the time of hardware resource calculation simulation calculation in the multi-mode power consumption control algorithm process plus the initial time margin. For the initial time margin, the present invention further:

as shown in FIG. 16, in one embodiment, the continuous optimization step of the multi-modal control decision calculation is as follows:

if the simulation calculation time Tx1 is greater than or equal to the simulation calculation time Tx1, the actual time is assigned to the simulation time, then the simulation time is subtracted by the initial time margin Tm to obtain an intermediate time Tz, and the intermediate time Tz is taken as a control object to increase the frequency of corresponding hierarchical hardware resources so that the simulation calculation time approaches the intermediate time Tz (approaches within a Tz time period which can be understood as +/-5%); then, the multi-mode calculation unit is calculated according to the enlarged parameters of the corresponding hierarchical hardware resources to obtain updated actual time Tr 1; adjusting the initial time margin Tm to obtain an updated time margin Tm1 (in order to obtain an optimal solution as soon as possible, ratio adjustment is generally adopted, such as Tm1/Tm = (Tr 1-T0)/(Tr-T0)); and giving the updated actual time Tr 1to the simulation time Tf, replacing the initial time margin Tm with the updated time margin Tm1, and repeating the calculation until the updated actual time falls into [ 1-k ] T0 and T0 ], wherein k is an error coefficient, and the range of k is 1-5%.

If the actual time Tr is greater than or equal to the simulation time Tf, assigning the actual time to the simulation-calculated time Tx1, and recalculating the simulation time Tf;

if the actual time Tr is greater than the completion time request T0, the actual time is assigned to the simulated calculated time Tx1 and the simulated time Tf is recalculated.

In another embodiment, the continuous optimization step of the multi-modal control decision calculation is as follows:

if the actual time Tr is greater than the completion time request T0 and the actual time is greater than the simulation time, increasing the initial time margin Tm until the actual time is less than the simulation time;

if the actual time Tr is greater than the completion time requirement T0 and the actual time is less than the simulation time, assigning the actual time to the simulation time, then subtracting the initial time margin Tm from the simulation time to obtain an intermediate time Tz, and increasing the frequency of the corresponding hierarchical hardware resources by using the intermediate time Tz as a control object, so that the time of the simulation calculation approaches the intermediate time Tz (approaches within a Tz time period that can be understood as ± 5%); then, the multi-mode calculation unit is calculated according to the enlarged parameters of the corresponding hierarchical hardware resources to obtain updated actual time Tr 1;

adjusting the initial time margin Tm to obtain an updated time margin Tm1,

For example, the completion time requires T0=100ms, and under the simulation calculation of the second level hardware resource, the calculated T51 is 100ms, and the initial time margin set by us can be generally set to be larger, because it is the first calculation, we allow enough time for actual operation, and therefore, we set the margin Tm to be 50 ms. At this time, the simulation time was T51+ Tm =150 ms. (since the power consumption savings and time savings of the hardware are not linear, as can be seen in FIGS. 11 and 12, no direct adjustments can be made, requiring multiple approximations)

Under the parameter of the main frequency of T51, if the time for the second-level hardware resource to actually calculate is less than 100ms, the actual operation state is the optimal state in this case because the above-mentioned T51 is already obtained under the lowest power consumption (frequency and voltage).

Under the parameter of obtaining the dominant frequency of T51, if the time for the second-level hardware resource to actually calculate is greater than 100ms, such as 130ms, since 130ms is greater than 100ms, optimization is required, and the 130 minus the initial setting margin of 50ms is used to obtain the intermediate time Tz =80ms, obviously, below 100ms of T51, to complete in a faster time, the frequency must be increased (as shown in fig. 13, the voltage varies with the frequency, so we only need to substantially adjust the frequency), so that the time for the hardware running at the frequency to complete the calculation approaches to 80ms, and since wireless approximation requires multiple attempts, we generally set an allowable range, such as ± 5%. The frequency is determined and then the actual calculations are performed at that frequency to obtain an updated actual time, e.g., whether the actual time is 105 or above the completion time requirement T0.

In this case, the initial time margin Tm is adjusted to obtain an updated time margin Tm1, and in order to obtain an optimal solution as soon as possible, a ratio adjustment is generally used, such as Tm1/Tm = (Tr 1-T0)/(Tr-T0).

And giving the updated actual time Tg to the simulation time, replacing the initial time margin Tm with the updated time margin Tg, and repeating the calculation of the stage until the updated actual time is within (1-k) × T0, T0 ].

The hardware system of the invention is as follows:

The software system of the invention is as follows:

the application layer comprises a human-computer interaction unit;

According to the technical scheme, the computing resource requirements and the completion time are taken as control limits, the power consumption is taken as a control index, and the computing resources of each layer are subjected to predictive computation and taken as a decision basis for system scheduling; the control decision calculation module can be continuously optimized through closed-loop control, the maximum energy efficiency of the dynamic heart-lung monitor system is realized, and therefore the purpose that a set of artificial intelligent edge calculation equipment with the minimum volume is used, various monitoring scenes are met, and the standby time of the equipment is longest is achieved.

Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.

Claims

1. A method of power consumption control for an artificial intelligence dynamic cardiopulmonary monitoring device, the method comprising:

establishing an AI inference model according to the multi-modal data input;

monitoring actual power consumption and actual time of calculation of the multi-mode calculation unit to obtain actual power consumption Pr and actual time Tr;

2. The power consumption control method according to claim 1, wherein after the local calculation result obtained after the multi-modal data fusion is output, the parameter requirements of the artificial intelligence reasoning model are adjusted according to different requirements, and the closed-loop parameters are updated according to different application scenarios.

3. The power consumption control method of claim 1, wherein deriving the parameter requirements of the AI inference model comprises:

4. The power consumption control method of claim 1, wherein the multi-modal control decision calculation comprises:

and if the time for completing the simulation calculation under the last-stage classification is still lower than the requirement of the completion time, the multi-modal control decision calculation is output by adopting the resources of the system hardware of the last stage.

5. The power consumption control method according to claim 4, wherein the parameters of the hardware resources subjected to the simulation calculation are optimized according to a comparison between the simulated power consumption of the hardware resources subjected to the simulation calculation and the corresponding actual power consumption and a comparison between the simulated time and the actual time in the calculation resource requirements in the multi-modal control decision calculation, so as to obtain the parameters of the optimal hardware resources, and the optimal completion time is obtained according to the parameters of the optimal hardware resources.

6. The power consumption control method according to claim 5, wherein the system hardware includes an AI data processing unit including an SOC chip and an FPGA chip;

the SOC chip comprises an MCU control unit and a computing unit, wherein the computing unit comprises a parallel RISC-V computing block with a plurality of cores and a hardware convolution computing engine;

the FPGA chip comprises a plurality of parallel programmable engines;

if the time for completing the simulation calculation of all the data is higher than or equal to the requirement of the completion time, transmitting part of the data which needs a large amount of hardware resources and has low real-time requirement to the FPGA chip for calculation, gradually reducing the frequency of the FPGA chip, calculating the time T61 for completing the simulation calculation of the part of the data by the FPGA chip with the lowest power consumption, enabling the time for the simulation calculation to be less than or equal to the requirement of the completion time T0, and calculating the sum of the power of the SOC chip and the power of the FPGA chip at the moment to serve as the simulation power consumption; if the time for completing all data simulation calculation is lower than the time requirement, switching to the least hardware resources for simulation calculation;

if the time for completing the simulation calculation of all the data is higher than or equal to the completion time requirement, a parallel RISC-V calculation block with a plurality of cores and the hardware convolution calculation engine which are added into the SOC chip carry out simulation calculation together, the lowest main frequency for completing the simulation calculation of all the data is found out according to the frequency-voltage curve of the calculation unit of the SOC chip, the voltage corresponding to the lowest main frequency is obtained, the time T51 of the simulation calculation is smaller than or equal to the completion time requirement T0, and the sum of the power consumption of the calculation unit of the SOC chip and the power consumption of the hardware convolution calculation engine is calculated to be used as the simulation power consumption; if the time for completing the simulation calculation of all the data is lower than the completion time requirement, finding out the lowest dominant frequency for completing the simulation calculation of all the data and obtaining the voltage corresponding to the lowest dominant frequency according to the frequency-voltage curve of the hardware convolution calculation engine of the SOC chip under the calculation of the pure hardware convolution calculation engine, so that the time T41 of the simulation calculation is less than or equal to the completion time requirement, and calculating the power consumption of the hardware convolution calculation engine of the SOC chip at the moment as the simulation power consumption.

7. The power consumption control method of claim 6, wherein an initial time margin Tm is set, and the Tm is added to the time Tx1 of the simulation calculation to obtain a simulation time Tf, i.e., the Tm is added to the time T61, T51 and T41 as simulation times T6, T5 and T4, where x =4 or 5 or 6.

8. The power consumption control method of claim 7, wherein the step of continuously optimizing the multi-modal control decision calculation is as follows:

9. The power consumption control method of claim 7, wherein the step of continuously optimizing the multi-modal control decision calculation is as follows:

if the actual time is less than or equal to the completion time requirement T0, taking the actual time as the optimal completion time, and taking the hardware and parameters corresponding to the actual power consumption as optimal hardware resources;

if the actual time is greater than the completion time requirement T0 and the actual time is less than the simulation time, assigning the actual time to the simulation time, then subtracting the initial time margin Tm from the simulation time to obtain an intermediate time Tz, and increasing the frequency of corresponding hierarchical hardware resources by taking the intermediate time Tz as a control object so that the time of simulation calculation approaches to the intermediate time Tz; then, the multi-mode calculation unit is calculated according to the enlarged parameters of the corresponding hierarchical hardware resources to obtain updated actual time Tr 1;

adjusting the initial time margin Tm to obtain an updated time margin Tm 1;

10. An artificial intelligence dynamic cardiopulmonary monitoring system, the system comprising a multi-modal signal acquisition unit, an AI data processing unit, a power management unit, a wireless communication unit, a voice unit and a human-machine operation interface, the AI data processing unit and the power management unit implementing the power consumption control method of any one of claims 1-9.