CN111192257A

CN111192257A - Method, system and equipment for determining equipment state

Info

Publication number: CN111192257A
Application number: CN202010000656.9A
Authority: CN
Inventors: 郭双全; 许伟; 汪振江
Original assignee: Shanghai Electric Group Corp
Current assignee: Shanghai Electric Group Corp
Priority date: 2020-01-02
Filing date: 2020-01-02
Publication date: 2020-05-22
Anticipated expiration: 2040-01-02
Also published as: CN111192257B

Abstract

The invention discloses a method, a system and equipment for determining the state of equipment, which can be applied to data analysis of industrial equipment and improve the accuracy and reliability of state evaluation of the industrial equipment. The method comprises the following steps: converting sensor data of the equipment into standard-time spectrogram data, wherein the standard-time spectrogram data is time-frequency spectrogram data meeting a preset standard specification; inputting the standard-time spectrogram data into a first convolutional neural network model through an input channel of the first convolutional neural network model to obtain first model data; dividing the standard-time spectrogram data into a plurality of data sequences according to a set direction, and inputting the plurality of data sequences into a second convolutional neural network model through a plurality of input channels of the second convolutional neural network model corresponding to the set direction to obtain second model data; and determining the state of the equipment according to the first model data and the second model data.

Description

Method, system and equipment for determining equipment state

Technical Field

The invention relates to the technical field of industrial intelligence, in particular to a method, a system and equipment for determining equipment state.

Background

In the field of industrial equipment, for example, large-scale equipment such as large-scale machine tools, wind turbines, steam turbines, industrial motors and the like are often provided with an online monitoring system for ensuring stable and reliable operation, a plurality of monitoring sensors have more measuring points, the sampling frequency of key parameters such as vibration, electricity, pressure and the like is high, the original data volume is large, and the traditional data analysis method taking signal processing characteristic extraction as means has the difficulties of high complexity, limited precision and the like.

At present, a signal feature extraction method for high-frequency sampling data includes a time domain analysis method, a frequency domain analysis method and a time-frequency analysis method (short-time fourier transform, wavelet transform, empirical mode decomposition and the like). The time domain analysis mainly takes time domain statistical index calculation and correlation analysis as main parts, the attention to frequency domain characteristics is low, the frequency domain analysis is based on Fourier transform, and the time domain analysis is more suitable for equipment stable signal analysis and cannot give consideration to the time domain characteristics; a time-frequency matrix (or a time-frequency spectrum) generated by processing through the time-frequency analysis method gives consideration to characteristics of a time domain and a frequency domain, but a typical characteristic extraction needs to be carried out according to manual professional experience in a traditional equipment state evaluation or fault judgment method such as cluster analysis, a decision tree, a Gaussian mixture model and the like, so that the equipment state evaluation method has certain subjectivity and incompleteness, and the equipment state evaluation result is uncertain.

Disclosure of Invention

The invention provides a method, a system and equipment for determining equipment state, which can be applied to data analysis of industrial equipment, and can extract the characteristics of time-frequency spectrograms from different levels by utilizing a plurality of convolutional neural network models aiming at time-frequency spectrogram data obtained after the sensing data of the industrial equipment is processed, thereby comprehensively improving the state evaluation effect of the industrial equipment and improving the accuracy and reliability of the state evaluation of the industrial equipment.

In a first aspect, the present invention provides a method of determining a device state, the method comprising:

converting sensor data of the equipment into standard-time spectrogram data, wherein the standard-time spectrogram data is time-frequency spectrogram data meeting a preset standard specification;

inputting the standard-time spectrogram data into a first convolutional neural network model through an input channel of the first convolutional neural network model to obtain first model data;

dividing the standard-time spectrogram data into a plurality of data sequences according to a set direction, and inputting the plurality of data sequences into a second convolutional neural network model through a plurality of input channels of the second convolutional neural network model corresponding to the set direction to obtain second model data;

and determining the state of the equipment according to the first model data and the second model data.

As a possible implementation manner, dividing the time-frequency spectrogram data into a plurality of data sequences according to a set direction includes:

dividing the time-frequency spectrogram data into a plurality of time data sequences corresponding to frequencies in a mode of forming data sequences along the direction of a time axis; and/or dividing the time-frequency spectrogram data into frequency data sequences corresponding to a plurality of moments in a mode of forming the data sequences along the direction of a frequency axis.

As a possible implementation, the time-frequency spectrogram data is divided into a plurality of time data sequences corresponding to frequencies in a manner of forming a data sequence along a time axis direction, and the time-frequency spectrogram data is divided into a plurality of frequency data sequences corresponding to moments in a manner of forming a data sequence along a frequency axis direction, wherein the second convolutional neural network model comprises a time axis convolutional neural network model corresponding to the time axis direction and a frequency axis convolutional neural network model corresponding to the frequency axis direction;

inputting the plurality of data sequences into the second convolutional neural network model through a plurality of input channels of the second convolutional neural network model corresponding to the set direction to obtain second model data, including:

inputting the time data sequences into the time axis convolution neural network model through a plurality of input channels of the time axis convolution neural network model to obtain second time model data;

inputting the plurality of frequency data sequences into the frequency axis convolution neural network model through a plurality of input channels of the frequency axis convolution neural network model to obtain second frequency model data;

and determining the second time model data and the second frequency model data as second model data.

As a possible implementation manner, the number of input channels of the second convolutional neural network model is the same as the number of data sequences obtained by dividing the time-frequency spectrogram data according to a set direction.

As a possible implementation, determining the state of the device according to the first model data and the second model data includes:

inputting the first model data and the second model data into a fusion model, and determining the state of the equipment, wherein the fusion model comprises: a formula model or a machine learning model.

As a possible embodiment, converting sensor data of a device into standard-time spectrogram data includes:

performing signal processing on sensor data of equipment to obtain time-frequency spectrogram data;

establishing an interpolation function for a spectrum value corresponding to the data sequence of the time-frequency spectrogram data according to a preset resolution to obtain the time-frequency spectrogram data with the preset resolution;

and determining the intercepted time range and frequency range of the obtained time-frequency spectrogram data with the preset resolution according to a preset rule, and intercepting the time-frequency spectrogram data with the preset resolution according to the time range and the frequency range to obtain standard-time spectrogram data.

As a possible implementation manner, determining the intercepted time range according to a preset rule includes:

determining the intercepted time range according to the moment when the energy value of the frequency data sequence in the time-frequency spectrogram data with the preset resolution is larger than the corresponding preset threshold; or

Determining the intercepted time range by taking the moment corresponding to the maximum energy value of the frequency data sequence in the time-frequency spectrogram data with the preset resolution as a center; or

And determining the intercepted time range according to the time period that the sum of the energy values of the frequency data sequences in the set time period in the time-frequency spectrogram data with the preset resolution is greater than the time period corresponding to the preset value.

As a possible implementation, the method further comprises:

and converting the standard-time spectrogram data into time-frequency spectrogram data with preset dimensions by a normalization or standardization algorithm.

As a possible way of implementing the method,

the first convolution neural network model is obtained by training a standard-time spectrogram data sample and a corresponding first model data sample;

the second convolutional neural network model is obtained by training a data sequence sample in the standard-time spectrogram data sample and a corresponding second model data sample.

As a possible implementation manner, if the fusion model is a machine learning model, the fusion model is obtained by training using the first model data sample, the second model data sample, and the corresponding device state sample.

In a second aspect, the present invention provides a system for determining the status of a device, the system comprising: data conversion module, first convolution neural network module, second convolution neural network module, definite equipment state module, wherein:

the data conversion module is used for converting sensor data of the equipment into standard-time spectrogram data, and the standard-time spectrogram data is time-frequency spectrogram data meeting a preset standard specification;

the first convolution neural network module is used for inputting the standard time spectrogram data into the first convolution neural network model through an input channel of the first convolution neural network model to obtain first model data;

the second convolutional neural network module is used for dividing the standard-time spectrogram data into a plurality of data sequences according to a set direction, and inputting the plurality of data sequences into a second convolutional neural network model through a plurality of input channels of the second convolutional neural network model corresponding to the set direction to obtain second model data;

and the device state determining module is used for determining the state of the device according to the first model data and the second model data.

As a possible implementation, the second convolutional neural network module is specifically configured to:

the second convolutional neural network module is specifically configured to:

As a possible implementation manner, the module for determining a device status is specifically configured to:

As a possible implementation manner, the data conversion module is specifically configured to:

As a possible implementation, the system further includes a normalization processing module specifically configured to:

As a possible implementation manner, the first convolutional neural network model is obtained by training a standard-time spectrogram data sample and a corresponding first model data sample;

In a third aspect, the present invention provides an apparatus for determining a state of an apparatus, the apparatus comprising: a processor and a memory, wherein the memory stores program code that, when executed by the processor, causes the processor to perform the steps of:

As a possible implementation, the processor is specifically configured to:

the processor is specifically configured to:

As a possible implementation, the processor is specifically configured to:

As a possible implementation, the processor is specifically further configured to:

As a possible implementation, the processor is specifically configured to:

As a possible way of implementing the method,

In a fourth aspect, the present invention provides a computer storage medium having stored thereon a computer program which, when executed by a processor, performs the steps of the method of the first aspect.

The method, the system and the equipment for determining the equipment state have the following beneficial effects that:

the method can be applied to data analysis of industrial equipment, and for time-frequency spectrogram data obtained after the sensing data of the industrial equipment is processed, the two-dimensional time-frequency spectrogram data is subjected to feature extraction through the first convolutional neural network model, the one-dimensional data sequence in the time spectrogram data is subjected to feature extraction through the second convolutional neural network model, the time spectrograms are subjected to feature extraction from different layers through the plurality of convolutional neural network models, the state evaluation effect of the industrial equipment is comprehensively improved, and the accuracy and the reliability of state evaluation of the industrial equipment are improved.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without inventive exercise.

Fig. 1 is a flowchart of a method for determining a device status according to an embodiment of the present invention;

fig. 2 is a schematic diagram of time-frequency spectrogram data provided in the embodiment of the present invention;

FIG. 3 is a schematic structural diagram of a first convolutional neural network model according to an embodiment of the present invention;

FIG. 4 is a schematic diagram of a partition of time-frequency spectrogram data according to an embodiment of the present invention;

fig. 5 is another schematic diagram for dividing the time-frequency spectrogram data according to the embodiment of the present invention;

FIG. 6 is a diagram illustrating a structure of a second convolutional neural network model according to an embodiment of the present invention;

FIG. 7 is a diagram illustrating the structure of another second convolutional neural network model according to an embodiment of the present invention;

fig. 8 is a schematic diagram of a standard-time spectrogram data according to an embodiment of the present invention;

fig. 9 is a specific flowchart for performing normalization processing on the time-frequency spectrogram data according to the embodiment of the present invention;

fig. 10 is a flowchart illustrating an embodiment of determining a device status according to the present invention;

fig. 11 is a schematic diagram of a system for determining a device status according to an embodiment of the present invention;

fig. 12 is a schematic diagram of a device for determining a device status according to an embodiment of the present invention.

Detailed Description

In order to make the objects, technical solutions and advantages of the present invention clearer, the present invention will be described in further detail with reference to the accompanying drawings, and it is apparent that the described embodiments are only a part of the embodiments of the present invention, not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

The term "and/or" in the embodiments of the present invention describes an association relationship of associated objects, and indicates that three relationships may exist, for example, a and/or B may indicate: a exists alone, A and B exist simultaneously, and B exists alone. The character "/" generally indicates that the former and latter associated objects are in an "or" relationship.

The application scenario described in the embodiment of the present invention is for more clearly illustrating the technical solution of the embodiment of the present invention, and does not form a limitation on the technical solution provided in the embodiment of the present invention, and it can be known by a person skilled in the art that with the occurrence of a new application scenario, the technical solution provided in the embodiment of the present invention is also applicable to similar technical problems. In the description of the present invention, the term "plurality" means two or more unless otherwise specified.

The embodiment of the invention can be applied to data analysis of industrial equipment, mainly aims at analyzing time-frequency spectrogram data obtained after the sensing data of the industrial equipment is processed, and evaluates the state of the industrial equipment, and the application scene of the embodiment of the invention is briefly explained as follows:

in the field of industrial equipment, for example, large-scale equipment such as large-scale machine tools, wind turbines, steam turbines, industrial motors and the like are often equipped with an online monitoring system for ensuring stable and reliable operation, because monitoring points of monitoring sensors are more, sampling frequency of key parameters such as vibration, electricity and pressure is higher, the obtained data volume of the sensors is larger, and a traditional data analysis method which is carried out by taking signal feature extraction of the data of the sensors as a means has the difficulties of high complexity, limited precision and the like.

At present, methods for extracting signal characteristics of sensor data of equipment obtained by high-frequency sampling include time domain analysis, frequency domain analysis and time-frequency analysis (such as short-time fourier transform, wavelet transform, empirical mode decomposition and the like), wherein, the time domain analysis is mainly based on time domain statistical index calculation and correlation analysis, the attention to the frequency domain characteristics is not high, the frequency domain analysis is based on Fourier transform, is more suitable for the stable signal analysis of equipment and can not give consideration to the time domain characteristics of signals, a time frequency matrix (or a time frequency spectrum) generated by the time frequency analysis method has the characteristics of both time domain and frequency domain, however, the traditional state evaluation or fault judgment methods of the equipment, such as cluster analysis, decision trees, Gaussian mixture models and the like, must extract typical features according to manual professional experience, have certain subjectivity and incompleteness, and have uncertainty in evaluation results.

To solve the above technical problem, an embodiment of the present invention provides a method for determining a device status, after time-frequency spectrogram data (or time-frequency matrix) obtained by performing time-frequency analysis processing on sensor data of the equipment is converted into standard-time spectrogram data, the time spectrogram data is input into a plurality of convolutional neural network models for feature extraction, so that subjectivity and incompleteness caused by manual feature extraction are avoided, a two-dimensional convolutional neural network model (namely a first convolutional neural network model) aiming at the standard-time spectrogram data is adopted based on two dimensions of time and frequency of the standard-time spectrogram data, and feature extracting the standard-time spectrogram data against a one-dimensional convolutional neural network model (i.e. a second convolutional neural network model) of a one-dimensional data sequence in the standard-time spectrogram data, and determining final state data of the equipment according to the obtained first state data and the second state data.

As shown in fig. 1, a specific implementation flow of a method for determining a device status provided in an embodiment of the present invention is as follows:

step 100, converting sensor data of equipment into standard-time spectrogram data, wherein the standard-time spectrogram data is time-frequency spectrogram data meeting a preset standard specification;

it should be noted that, in this embodiment, the sensor data of the device may be one data or multiple sensor data, and if the sensor data is multiple sensor data, the method for converting the sensor data of the device into the standard-time spectrogram data includes: sensor data of the device is converted into a set of standard-time spectrogram data corresponding to the plurality of sensor data.

It should be noted that the devices in the embodiments of the present invention include, but are not limited to: a wind driven generator, a coal mill, a machine tool, a generator set and a crane; in this embodiment, the type of the industrial device is not limited too much, and the method in this embodiment may be applied to an industrial device having sensor data to determine the device status.

Optionally, the method for obtaining the time-frequency spectrogram data by performing signal processing on the sensor data of the device includes, but is not limited to, the following methods: short-time Fourier transform, wavelet transform, empirical mode decomposition. The present example does not unduly limit how the time-frequency spectrogram data is obtained from the sensor data.

Optionally, the sensor signal (sensor data) of the device is subjected to signal processing by a time-frequency analysis method, for example, short-time fourier transform, wavelet transform, empirical mode decomposition, and the like are performed on the sensor signal, the sensor data of the device is converted into time-frequency spectrogram data, and the energy density or intensity of the sensor signal at different times and frequencies can be obtained according to the time-frequency spectrogram data, wherein the time-frequency spectrogram data analyzes the sensor signal by using the combination function of time and frequency, and the amplitude of the sensor signal at each time and frequency can be given.

It should be noted that the time-frequency spectrogram data is a two-dimensional matrix, and each element in the matrix may be referred to as a spectral value; alternatively, the time-frequency spectrogram data can be represented by fig. 2, wherein the horizontal axis represents time, the vertical axis represents frequency, and the spectral values represent the amplitude of the sensor signal at the corresponding time and frequency.

In this embodiment, the standard-time spectrogram data is time-frequency spectrogram data meeting a preset standard specification, and it is easy to understand that the resolution and the size of the standard-time spectrogram data in this embodiment are time-frequency spectrogram data meeting the preset standard specification, and the resolution and the size of the time-frequency spectrogram data obtained according to the sensor data of the device may be standardized to obtain time-frequency spectrogram data meeting the preset standard specification.

Step 101, inputting the time-frequency spectrogram data into a first convolutional neural network model through an input channel of the first convolutional neural network model to obtain first model data;

it should be noted that, in this embodiment, one time-frequency spectrogram data may be input to the first convolution neural network model to obtain first model data, or a set formed by a plurality of time-frequency spectrogram data may be input to the first convolution neural network model to obtain a corresponding first model data set.

It should be noted that, in this embodiment, the first convolution neural network model may be obtained by training a standard-time spectrogram data sample and a corresponding first model data sample, where the standard-time spectrogram data sample is obtained by performing normalization processing on time-frequency spectrogram data in a historical time period, the first model data sample is a data sample corresponding to the time-frequency spectrogram data after the normalization processing, and the first model data sample may be status data of equipment or other data, and may be obtained according to a user requirement.

Optionally, the standard-time spectrogram data sample and the corresponding first model data sample are obtained as follows:

acquiring time-frequency spectrogram data of equipment in a set time period, and carrying out standardization processing on the time-frequency spectrogram data to obtain a standard-time spectrogram data sample; acquiring first model data of equipment in a set time period to obtain a first model data sample; and matching the standard-time spectrogram data in the set time period with the first model data in the set time period, namely that the standard-time spectrogram data in the set time period corresponds to a unique model data to obtain a standard-time spectrogram data sample and a corresponding first model data sample.

Optionally, the standard-time spectrogram data in the set time period is matched with the first model data in the set time period through a unique identification code.

Optionally, storing the standard-time spectrogram data in the set time period in a time-frequency spectrogram training storage area; and storing the first model data in the set time period in a state data training storage area.

The first convolutional neural network model in this implementation is a two-dimensional convolutional neural network model, and includes an input layer, hidden layers, and output layers, where the input layer includes an input channel for receiving two-dimensional data, that is, receiving time-frequency spectrogram data (or time-frequency matrix), the number of hidden layers is greater than 1, the first hidden layer behind the input layer is a two-dimensional convolutional layer, where the convolutional core is a two-dimensional convolutional core, and other hidden layers include convolutional layers, pooling layers, full-link layers, and the like, and the size of the first model data output by the output layer may be a custom data size, or a state data size of a system-default device, or a preset data size according to a use requirement, and the size of the first model data is not limited too much in this embodiment.

The structure of the first convolutional neural network model in this embodiment is shown in fig. 3, and includes an input layer 300, a hidden layer 301, and an output layer 302, where the number of channels of the input layer is 1, and the size of a channel may be determined according to the size of the time-frequency spectrogram data (or time-frequency matrix), and it is easy to understand that the size of the channel may be equal to the length N of the time data sequence multiplied by the length M of the frequency data sequence;

the first hidden layer after the input layer is a two-dimensional convolutional layer, the number of channels (i.e. the number of convolutional kernels) of the hidden layer is N1_ c1, and the size of the convolutional kernels is W1_ c1 × H1_ c1, wherein the number and the size of the convolutional kernels of the hidden layer can be set as required, and optionally can be determined according to the size of the output first state data and the number of the hidden layers in the first convolutional neural network model;

the number of hidden layers is greater than 1, and the hidden layers 2 to N except the first hidden layer can comprise: the implementation does not limit the number and specific types of other hidden layers;

the size of the data output by the output layer may be consistent with the size of the first model data, which may be the state data of the device, including but not limited to any one or any number of the following:

gearbox bearing performance status data; motor health status data; gear operating state data; vibration state data; bearing operating condition data.

The state data includes, but is not limited to, a normal or fault state, a power generation or non-power generation state, a rated or overrun state, and other operating condition states, and the type of the state is not limited too much in this embodiment.

Alternatively, the state data of the device may be a discrete value or a continuous value, for example, the state data of the device may be represented by a binary number, for example, the gear box bearing performance state data may be set to three states of 00 (for indicating excellent performance), 01 (for indicating good performance) and 10 (for indicating poor performance), the motor health state data may be set to three states of 00 (for indicating health), 01 (for indicating sub-health) and 10 (for indicating unhealthy condition), and if the state data of the device includes a plurality of state data, the number of bits of the binary number of the state data of the device may be determined according to the number of types of the included state data.

Step 102, dividing the standard-time spectrogram data into a plurality of data sequences according to a set direction, and inputting the plurality of data sequences into a second convolutional neural network model through a plurality of input channels of the second convolutional neural network model corresponding to the set direction to obtain second model data;

optionally, in this embodiment, the time-frequency spectrogram data may be divided into a set number of data sequences, and the number of the divided data sequences is not limited too much in this embodiment.

It should be noted that, in this embodiment, one standard-time spectrogram data may be divided into a plurality of data sequences, or each standard-time spectrogram data in a standard-time spectrogram data set composed of a plurality of standard-time spectrogram data may be divided into a plurality of data sequences to obtain a data sequence set, and the data sequence set is input to the second convolutional neural network model to obtain a corresponding second model data set.

In implementation, the time-frequency spectrogram data is divided into a plurality of data sequences according to a set direction, and for convenience of description, the following three implementation modes are provided in this embodiment, and are described below with reference to fig. 4:

the method comprises the following steps of 1, dividing time-frequency spectrogram data into a plurality of time data sequences corresponding to frequencies in a mode of forming data sequences along a time axis direction;

optionally, in this embodiment, the time-frequency spectrogram data may be divided into a set number of time data sequences, where each time data sequence corresponds to one frequency; the time-frequency spectrogram data can also be divided into time data sequences with the same number as the number of frequencies according to the number of frequencies in the time-frequency spectrogram data (i.e. the length of the frequency sequences), wherein each time data sequence corresponds to one frequency.

As shown in fig. 4, the time axis direction is from left to right (or from right to left), and the time-frequency spectrogram is divided in the left-to-right direction, so as to obtain m time data sequences, wherein the frequency f1 corresponds to the time data sequence x1, the frequency f2 corresponds to the time data sequence x2, the frequency f3 corresponds to the time data sequences x3 and … …, and the frequency fm corresponds to the time data sequence xm; the time data sequence x1, the time data sequence x2, the time data sequences x3 and … …, and the time data sequence xm all include n data.

Mode 2, dividing the time-frequency spectrogram data into a plurality of frequency data sequences corresponding to moments in a mode of forming a data sequence along the direction of a frequency axis;

optionally, in this embodiment, the time-frequency spectrogram data may be divided into a set number of frequency data sequences, where each frequency data sequence corresponds to one time; the time-frequency spectrogram data can also be divided into frequency data sequences with the same number as the number of the moments according to the number of the moments in the time-frequency spectrogram data (i.e., the length of the time sequence), wherein each frequency data sequence corresponds to one moment.

As shown in fig. 5, the frequency axis direction is from bottom to top (or from top to bottom), and the time-frequency spectrogram is divided in the direction from bottom to top, so as to obtain n frequency data sequences, where time t1 corresponds to the frequency data sequence y1, time t2 corresponds to the frequency data sequence y2, time t3 corresponds to the frequency data sequences y3 and … …, and time tn corresponds to the frequency data sequence yn; the frequency data sequence y1, the frequency data sequence y2, the frequency data sequences y3 and … …, and the frequency data sequence yn all include m data.

And 3, dividing the time-frequency spectrogram data into a plurality of time data sequences corresponding to frequencies in a mode of forming a data sequence along the time axis direction, and dividing the time-frequency spectrogram data into a plurality of frequency data sequences corresponding to moments in a mode of forming a data sequence along the frequency axis direction.

With reference to fig. 4 and 5, the time-frequency spectrogram data is divided in the time axis direction and the frequency axis direction respectively to obtain m time data sequences and n frequency data sequences, where the time data sequences include n data, and the frequency data sequences include m data.

The following describes the method for obtaining the second model data in this embodiment with reference to the above three embodiments:

inputting the plurality of time data sequences into a second convolutional neural network model through a plurality of input channels of the second convolutional neural network model corresponding to the time axis direction to obtain second model data;

wherein the number of input channels is not less than the number of time data sequences.

The method 2 comprises the steps of dividing the time-frequency spectrogram data into a plurality of frequency data sequences corresponding to moments in a mode of forming a data sequence along the direction of a frequency axis;

inputting the plurality of frequency data sequences into a second convolutional neural network model through a plurality of input channels of the second convolutional neural network model corresponding to the frequency axis direction to obtain second model data;

wherein the number of input channels is not less than the number of frequency data sequences.

A method 3, dividing the time-frequency spectrogram data into a plurality of time data sequences corresponding to frequencies in a mode of forming data sequences along the time axis direction; dividing the time-frequency spectrogram data into a plurality of frequency data sequences corresponding to moments in a mode of forming a data sequence along the direction of a frequency axis;

in this method, the second convolutional neural network model includes a time axis convolutional neural network model corresponding to the time axis direction and a frequency axis convolutional neural network model corresponding to the frequency axis direction;

inputting the plurality of time data sequences into the time axis convolution neural network model through a plurality of input channels of the time axis convolution neural network model corresponding to the time axis direction to obtain second time model data;

inputting the plurality of frequency data sequences into the frequency axis convolution neural network model through a plurality of input channels of the frequency axis convolution neural network model corresponding to the frequency axis direction to obtain second frequency model data;

wherein the number of input channels of the time axis convolutional neural network model is not less than the number of the time data sequences; the number of input channels of the frequency axis convolution neural network model is not less than the number of the frequency data sequences.

As an optional implementation manner, the number of input channels of the second convolutional neural network model is the same as the number of data sequences obtained by dividing the time-frequency spectrogram data according to a set direction.

In implementation, if the time-frequency spectrogram data is divided into a plurality of time data sequences corresponding to frequencies in a manner of forming the data sequences along the time axis direction; the number of input channels of the second convolutional neural network model is the same as the number of the time data sequences;

if the time-frequency spectrogram data is divided into a plurality of frequency data sequences corresponding to moments in a mode of forming the data sequences along the direction of a frequency axis; the number of input channels of the second convolutional neural network model is the same as the number of the frequency data sequences;

if the time-frequency spectrogram data is divided into a plurality of time data sequences corresponding to frequencies in a mode of forming a data sequence along the direction of a time axis, and the time-frequency spectrogram data is divided into a plurality of frequency data sequences corresponding to moments in a mode of forming a data sequence along the direction of a frequency axis; the number of input channels of the time axis convolutional neural network model is equal to the number of the time data sequences; the number of input channels of the frequency axis convolution neural network model is equal to the number of the frequency data sequences.

It should be noted that, in this embodiment, the second convolutional neural network model is obtained by training a data sequence sample in a standard-time spectrogram data sample and a corresponding second model data sample, where the standard-time spectrogram data sample is obtained by performing normalization processing on time-frequency spectrogram data in a historical time period, the data sequence sample is obtained by dividing the standard-time spectrogram data according to a set direction, the second model data sample is obtained by a data sample corresponding to the data sequence sample in the normalized time-frequency spectrogram data sample, the second model data sample may be status data of equipment, or other data, and the corresponding second model data sample may be obtained according to a user requirement.

The size and/or type of the first model data sample and the second model data sample may be the same.

Optionally, the data sequence sample and the corresponding second model data sample in the standard-time spectrogram data sample are obtained as follows:

acquiring time-frequency spectrogram data of equipment in a set time period, carrying out standardization processing on the time-frequency spectrogram data to obtain a standard-time spectrogram data sample, and obtaining a data sequence sample in the standard-time spectrogram data sample through the three modes; acquiring second model data of the equipment in a set time period to obtain a second model data sample; and matching the data sequence sample in the standard-time spectrogram data sample in the set time period with the second model data in the set time period, namely, the data sequence samples in the standard-time spectrogram data sample in the set time period correspond to a unique model data, so as to obtain the data sequence sample in the standard-time spectrogram data sample and the corresponding second model data sample.

Optionally, the data sequence in the standard-time spectrogram data in the set time period is matched with the second model data in the set time period through the unique identification code.

The second convolutional neural network model in this embodiment is a one-dimensional convolutional neural network model, and includes an input layer, a hidden layer, and an output layer, where the input layer includes multiple input channels, and is configured to receive multiple one-dimensional data sequences, the number of the hidden layers is greater than 1, a first hidden layer behind the input layer is a one-dimensional convolutional layer, where the convolutional core is a one-dimensional convolutional core, and other hidden layers include convolutional layers, pooling layers, and full connection layers, and the second model data output by the output layer may be state data of a device or other data, and the size of the second model data may be a custom model data size or a system-default state data size of the device, or a data size preset according to a use requirement, and the size of the second model data is not limited too much in this embodiment.

It should be noted that, the second convolutional neural network model in this embodiment is a model corresponding to a set direction, and if the time-frequency spectrogram data is divided according to two set directions in this embodiment, such as a frequency axis direction and a time axis direction, in this embodiment, two second convolutional neural network models are required, including a frequency axis convolutional neural network model and a time axis convolutional neural network model, where the frequency axis convolutional neural network model corresponds to a plurality of data sequences obtained by dividing the time-frequency spectrogram data according to the frequency axis direction, and the time axis convolutional neural network model corresponds to a plurality of data sequences obtained by dividing the time-frequency spectrogram data according to the time axis direction, and therefore, the second convolutional neural network model in this embodiment may be one or multiple, and is specifically determined by the number of the set directions.

The second convolutional neural network model in this embodiment has a structure as shown in fig. 6 and 7, where the number of input channels of the input layer is the number of input sequences, and the size of the input channels may be determined according to the size of a data sequence divided by the standard-time spectrogram data.

As shown in fig. 6, the second convolutional neural network model includes an input layer 600, an implied layer 601, and an output layer 602, and divides the standard-time spectrogram data into a plurality of time data sequences in a time axis direction, where the number of input channels of the input layer of the second convolutional neural network model is M, and the size of the input channels may be equal to the length N of the time data sequences divided by the standard-time spectrogram data;

the first hidden layer after the input layer is a one-dimensional convolutional layer, the number of channels (i.e. the number of convolutional kernels) of the hidden layer is N2_ c1, and the size of the convolutional kernels is W2_ c1, wherein the number and the size of the convolutional kernels of the hidden layer can be set as required, and optionally can be determined according to the size of the output first state data and the number of the hidden layers in the first convolutional neural network model;

As shown in fig. 7, the second convolutional neural network model includes an input layer 700, an implied layer 701, and an output layer 702, and divides the standard-time spectrogram data into a plurality of frequency data sequences in the frequency axis direction, the number of input channels of the input layer of the second convolutional neural network model is N, and the size of the input channels may be equal to the length M of the frequency data sequences divided by the standard-time spectrogram data;

the first hidden layer after the input layer is a one-dimensional convolutional layer, the number of channels (i.e. the number of convolutional kernels) of the hidden layer is N3_ c1, and the size of the convolutional kernels is H3_ c1, wherein the number and the size of the convolutional kernels of the hidden layer can be set as required, and optionally can be determined according to the size of the output first model data and the number of hidden layers in the first convolutional neural network model;

The first model data and the second model data in this embodiment may be the same or different, and this embodiment is not limited to this.

It should be noted that the first convolutional neural network model and the second convolutional neural network model may be trained separately or simultaneously, and are not limited herein.

And 103, determining the state of the equipment according to the first model data and the second model data.

The status of the device in this implementation includes, but is not limited to, any one or any number of the following:

Optionally, if the first model data and the second model data are the same, the state of the device may be the first model data or the second model data; if the first model data and the second model data are different, the state of the equipment may be determined according to a set rule, for example, the first model data and the second model data may be weighted and summed according to weights corresponding to the first model data and the second model data, and the state of the equipment may be determined according to the summed value.

As an optional implementation manner, determining the state of the device according to the first model data and the second model data includes:

It should be noted that, in this embodiment, the first model data and the second model data may be input into the fusion model to determine the state of the device;

the first model data set and the second model data set may also be input into a fusion model to determine a set of states of the device.

If the second model data is obtained by the method 3, the second model data includes: second time model data and second frequency model data, the state of the device is then determined by:

determining the state of the equipment according to the first model data, the second time model data and the second frequency model data;

optionally, the first model data, the second time model data, and the second frequency model data are input into a fusion model, and the state of the device is determined, and if the fusion model is a machine learning model, the fusion model may be obtained by training using the first model data, the second time model data, the second frequency model data, and corresponding device state samples.

If the fusion model is a machine learning model, optionally, the fusion model includes, but is not limited to, any of the following:

a convolutional neural network model; a decision tree model; and (4) random forest models.

In implementation, the training of the first convolutional neural network model, the second convolutional neural network model and the fusion model may be synchronous iterative training, or the training of the first convolutional neural network model and the second convolutional neural network model may be performed first, and after the training is completed, the output data of the first convolutional neural network model and the second convolutional neural network model and corresponding device state samples are used as training samples to train the fusion model.

If the first convolutional neural network model, the second convolutional neural network model, and the fusion model are trained simultaneously, the first convolutional neural network model, the second convolutional neural network model, and the fusion model can be regarded as a comprehensive model, and the comprehensive model can be trained by using a standard-time spectrogram data sample, a data sequence sample in the standard-time spectrogram data sample, and a corresponding device state sample.

The embodiment also provides a method for performing normalization processing on time spectrogram data, and the specific implementation manner is as follows:

and determining the intercepted time range and frequency range of the obtained time-frequency spectrogram data with the preset resolution according to a preset rule or a preset size, and intercepting the time-frequency spectrogram data with the preset resolution according to the time range and the frequency range to obtain standard-time spectrogram data.

Optionally, determining the intercepted time range according to a preset rule includes:

determining the intercepted time range and frequency range according to the corresponding time and frequency when the energy value corresponding to the frequency data sequence in the time-frequency spectrogram data with the preset resolution meets the preset condition; or

Determining the intercepted time range and frequency range according to the corresponding time and frequency when the root mean square value corresponding to the frequency data sequence in the time-frequency spectrogram data with the preset resolution meets the preset condition; or

And determining the intercepted time range and frequency range according to the corresponding time and frequency when the average value corresponding to the frequency data sequence in the time-frequency spectrogram data with the preset resolution meets the preset condition.

It should be noted that the preset resolution and the preset size refer to resolution and size of a time-frequency spectrogram, where the preset resolution refers to that time intervals and frequency intervals of the time-frequency spectrogram are preset, the preset size refers to that time ranges and frequency ranges of the time-frequency spectrogram are preset, and the abscissa and ordinate axes of the standard-time spectrogram data obtained according to the preset resolution and the preset size are shown in fig. 8, where the time axis range is T₁～T_NFrequency axis range of F₁～F_MTime interval of T₂-T₁Frequency interval of F₂-F₁。

In an implementation, the data sequence of the time-frequency spectrogram data includes two types: a frequency data sequence and a time data sequence, wherein the data sequence corresponding to the time axis direction in the time-frequency spectrogram data is the time data sequence, and the data sequence corresponding to the frequency axis direction in the time-frequency spectrogram data is the frequency data sequence;

determining time intervals and frequency intervals of a time-frequency spectrogram according to a preset resolution, and if the time intervals of the time-frequency spectrogram data are different from the preset resolution, establishing an interpolation function for spectrum values corresponding to each time data sequence in the time-frequency spectrogram data to generate time-frequency spectrogram data with the same time intervals as the preset time intervals of the time-frequency spectrogram;

if the frequency interval of the time-frequency spectrogram data is different from the preset resolution, an interpolation function can be established for the spectrum values corresponding to each frequency data sequence in the time-frequency spectrogram data, and the time-frequency spectrogram data with the same frequency interval as the preset time-frequency spectrogram is generated;

if the time interval and the frequency interval of the time-frequency spectrogram data are different from the preset resolution, an interpolation function can be established for each time data sequence and the spectrum value corresponding to each frequency data sequence in the time-frequency spectrogram data, and the time-frequency spectrogram data with the same time interval and frequency interval as the preset time-frequency spectrogram is generated.

It is easy to understand that the energy value in the time-frequency spectrogram data is an energy value corresponding to the frequency data sequence at a time, and the energy value can be calculated by using a spectrum value corresponding to the frequency data sequence at the time.

Optionally, the intercepting time range is determined according to a preset rule, and the method includes any one of the following modes:

mode 1, determining an intercepted time range according to a moment corresponding to a preset threshold value when an energy value of a frequency data sequence in the time-frequency spectrogram data with the preset resolution is larger than the preset threshold value;

and determining the intercepted time range according to the minimum time and the maximum time corresponding to the energy value being greater than the preset threshold value.

Mode 2, determining the intercepted time range by taking the moment corresponding to the maximum energy value of the frequency data sequence in the time-frequency spectrogram data with the preset resolution as a center;

and determining the intercepted time range according to the preset length by taking the moment corresponding to the maximum energy value as a center.

And 3, determining the intercepted time range according to the time period in which the sum of the energy values of the frequency data sequences in the set time period in the time-frequency spectrogram data with the preset resolution is greater than the time period corresponding to the preset value.

Optionally, the intercepted frequency range is determined according to a preset rule, and includes any one of the following:

determining an intercepted frequency range according to the frequency corresponding to the preset condition that the spectrum value corresponding to the time data sequence in the time-frequency spectrogram data with the preset resolution meets the preset condition; or

And determining the intercepted frequency range according to the preset frequency range.

The determination of the truncated time range and the truncated frequency range in this embodiment is only an example, and in implementation, the manner of determining the truncated time range and the truncated frequency range may be defined according to requirements.

It should be noted that the sensor data of the device in this embodiment may be one type of sensor data, or may be multiple types of sensor data, and in implementation, the multiple types of sensor data may be converted into standard-time spectrogram data (or a set of standard-time spectrogram data) corresponding to the same type of sensor data;

if the sensor data of the device includes multiple types of sensor data, the present embodiment may also determine the state of the device through multiple first convolutional neural network models and multiple second convolutional neural network models for the sensor data of the multiple types of devices at the same time; after each type of sensor data is converted into standard time spectrogram data, the standard time spectrogram data is input into a first convolutional neural network model and a second convolutional neural network model corresponding to the type of sensor data, and the state of the equipment is determined.

In implementation, if the resolution (including the time interval and the frequency interval) and the size (including the time range and the frequency range) of the time-frequency spectrogram data obtained by performing signal processing on the sensor data of the device are different from the preset resolution and size, the time-frequency spectrogram data is normalized, as shown in fig. 9, and a specific normalization processing flow is as follows:

step 900, according to a preset resolution, establishing an interpolation function for spectrum values corresponding to each frequency data sequence in the time-frequency spectrogram data;

step 901, establishing an interpolation function for spectrum values corresponding to each time data sequence in the time-frequency spectrogram data according to a preset resolution;

step 902, generating first time spectrogram data having the same resolution as a preset resolution;

step 903, determining a time range and a frequency range for intercepting the first time spectrogram data according to a preset rule;

step 904, determining second time-frequency spectrogram data according to the time range and the frequency range;

and 905, intercepting the second time-frequency spectrogram data according to a preset size to obtain standard-time spectrogram data.

As an optional implementation manner, after the standard-time spectrogram data is obtained, the standard-time spectrogram data may be further converted into time-frequency spectrogram data of a preset dimension through a normalization or normalization algorithm, for example, a spectral value (an element value) in the standard-time spectrogram data is normalized or normalized, and if the spectral value in the original standard-time spectrogram data is a value from 0.01 to 0.09, a spectral value between 0 and 1 may be obtained after the normalization or normalization.

In the embodiment, the time-frequency spectrogram data obtained from the sensor data of the device is subjected to standardization processing, and the standard-time spectrogram data is converted into the time-frequency spectrogram data with the preset dimension, so that a standardized time-frequency spectrogram data set with uniform resolution and size can be finally obtained, the method is convenient to be used for training and using the first convolution neural network model, the second convolution neural network model and the fusion model, the training effect of the model can be greatly improved, and the device state determined by using the model has higher accuracy and reliability.

It should be noted that, in the embodiment of the present invention, different storage areas are set for storing different data, which may be stored in real time or stored at set time intervals in real time, and the storage time is not limited too much in this embodiment.

Optionally, the time-frequency spectrogram data samples are stored through a time-frequency spectrogram training storage area, and the first model data samples corresponding to the time-frequency spectrogram data samples are stored through a state data training storage area;

and/or storing a data sequence sample in the time-frequency spectrogram data sample through a time-frequency spectrogram training storage area, and storing a second model data sample corresponding to the data sequence sample through a state data training storage area.

The processor can read data in the time-frequency spectrogram training storage area and the state data training storage area in the storage area to train the first convolutional neural network model and/or the second convolutional neural network model.

Optionally, the time-frequency spectrogram data converted through the sensor data is stored through the time-frequency spectrogram evaluation storage area, and the first model data obtained through the first convolutional neural network model is stored through the state data evaluation storage area;

and/or storing a plurality of data sequences obtained by dividing the time-frequency spectrogram data through a time-frequency spectrogram evaluation storage area, and storing second model data obtained through a second convolutional neural network model through a state data evaluation storage area;

and/or, the storage area is evaluated by status data to store the status of the device.

If the state of the device is determined by the fusion model from the first model data and the second model data, the state of the device output by the fusion model may be stored in the state data evaluation storage area.

Optionally, the first convolutional neural network model and the second convolutional neural network model are stored through a model storage area, or the first convolutional neural network model, the second convolutional neural network model and the fusion model are stored through a model storage area; the first convolutional neural network model, the second convolutional neural network model and the fusion model stored in the model storage area are trained models.

In implementation, on one hand, a comprehensive model in a model storage area may be an integrated model of a first convolutional neural network model, a second convolutional neural network model, and a fusion model, and spectrogram data (or a time-frequency spectrogram data set) and a corresponding device state (or a device state set) are input to train the comprehensive model, and on the other hand, after the training is completed, the comprehensive model in the model storage area may be called, and time-frequency spectrogram data (or a time-frequency spectrogram data set) which needs to be subjected to device state evaluation may be input to the comprehensive model, so as to obtain a corresponding device state.

As shown in fig. 10, the following describes in detail a method for determining a device status according to an embodiment of the present invention with reference to an implementation process of performing normalization processing on time spectrogram data in the embodiment, where the implementation process includes:

step 1000, performing signal processing on sensor data of equipment to obtain time-frequency spectrogram data;

step 1001, establishing an interpolation function for a spectrum value corresponding to the data sequence of the time-frequency spectrogram data according to a preset resolution to obtain the time-frequency spectrogram data with the preset resolution;

step 1002, determining an intercepted time range and a frequency range of the obtained time-frequency spectrogram data with a preset resolution according to a preset rule, and intercepting the time-frequency spectrogram data with the preset resolution according to the time range and the frequency range to obtain standard time spectrogram data;

step 1003, converting the standard-time spectrogram data into time-frequency spectrogram data with preset dimensions through a normalization or standardization algorithm;

step 1004, inputting the time-frequency spectrogram data into a first convolutional neural network model through an input channel of the first convolutional neural network model to obtain first model data;

step 1005, dividing the time-frequency spectrogram data into a plurality of time data sequences corresponding to frequencies in a mode of forming data sequences along the time axis direction;

step 1006, inputting the plurality of time data sequences into the time axis convolutional neural network model through a plurality of input channels of the time axis convolutional neural network model corresponding to the time axis direction to obtain second time model data;

step 1007, dividing the time-frequency spectrogram data into a plurality of frequency data sequences corresponding to moments in a mode of forming data sequences along the direction of a frequency axis;

step 1008, inputting the plurality of frequency data sequences into the frequency axis convolution neural network model through a plurality of input channels of the frequency axis convolution neural network model corresponding to the frequency axis direction to obtain second frequency model data;

step 1009, determining the second time model data and the second frequency model data as second model data;

step 1010, inputting the first model data and the second model data into a fusion model, and determining the state of the equipment.

Wherein, the

steps

1004, 1005 and 1007 may be executed simultaneously.

Based on the same inventive concept, the embodiment of the present invention further provides a system for determining a device status, and since the system is the system in the method in the embodiment of the present invention, and the principle of the system for solving the problem is similar to that of the method, the implementation of the system may refer to the implementation of the method, and repeated details are not repeated.

As shown in fig. 11, the system includes: a data conversion module 1100, a first convolutional neural network module 1101, a second convolutional neural network module 1102, and a device state determination module 1103, wherein:

a data conversion module 1100, configured to convert sensor data of a device into standard-time spectrogram data, where the standard-time spectrogram data is time-frequency spectrogram data meeting a preset standard specification;

a first convolutional neural network module 1101, configured to input the standard-time spectrogram data to a first convolutional neural network model through an input channel of the first convolutional neural network model, so as to obtain first model data;

the second convolutional neural network module 1102 is configured to divide the standard-time spectrogram data into a plurality of data sequences according to a set direction, and input the plurality of data sequences to a second convolutional neural network model through a plurality of input channels of the second convolutional neural network model corresponding to the set direction, so as to obtain second model data;

a device state determining module 1103, configured to determine a state of the device according to the first model data and the second model data.

As a possible implementation manner, the second convolutional neural network module 1102 is specifically configured to:

the second convolutional neural network module 1102 is specifically configured to:

As a possible implementation manner, the device state determining module 1103 is specifically configured to:

As a possible implementation manner, the data conversion module 1100 is specifically configured to:

Based on the same inventive concept, the embodiment of the present invention further provides a device for determining a device state, and since the device is a device in the method in the embodiment of the present invention, and a principle of the device for solving the problem is similar to that of the method, the implementation of the device may refer to the implementation of the method, and repeated details are omitted.

As shown in fig. 12, the apparatus includes: a processor 1200 and a memory 1201, wherein the memory 1201 stores program code that, when executed by the processor 1200, causes the processor 1200 to perform the steps of:

As a possible implementation, the processor 1200 is specifically configured to:

the processor 1200 is specifically configured to:

As a possible implementation, the processor 1200 is specifically configured to:

As a possible implementation, the processor 1200 is specifically further configured to:

As a possible implementation, the processor 1200 is specifically configured to:

As a possible way of implementing the method,

Based on the same inventive concept, an embodiment of the present invention further provides a computer storage medium, on which a computer program is stored, which when executed by a processor implements the following steps:

As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, optical storage, and the like) having computer-usable program code embodied therein.

The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

It will be apparent to those skilled in the art that various changes and modifications may be made in the present invention without departing from the spirit and scope of the invention. Thus, if such modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include such modifications and variations.

Claims

1. A method of determining a state of a device, the method comprising:

2. The method of claim 1, wherein dividing the time-frequency spectrogram data into a plurality of data sequences in a set direction comprises:

3. The method of claim 2,

dividing the time-frequency spectrogram data into a plurality of time data sequences corresponding to frequencies in a mode of forming a data sequence along a time axis direction, and dividing the time-frequency spectrogram data into a plurality of frequency data sequences corresponding to moments in a mode of forming a data sequence along a frequency axis direction, wherein the second convolutional neural network model comprises a time axis convolutional neural network model corresponding to the time axis direction and a frequency axis convolutional neural network model corresponding to the frequency axis direction;

4. The method according to any one of claims 1 to 3, wherein the number of input channels of the second convolutional neural network model is the same as the number of data sequences obtained by dividing the time-frequency spectrogram data in a set direction.

5. The method of claim 1, wherein determining the state of the device based on the first model data and the second model data comprises:

6. The method of claim 1, wherein converting sensor data of a device to standard-time spectrogram data comprises:

7. The method of claim 6, further comprising:

8. The method of claim 1,

9. The method of claim 5, wherein if the fusion model is a machine learning model, the fusion model is trained using the first model data sample, the second model data sample, and the corresponding device state sample.

10. A system for determining a state of a device, the system comprising: data conversion module, first convolution neural network module, second convolution neural network module, definite equipment state module, wherein:

11. The system of claim 10, wherein the second convolutional neural network module is specifically configured to:

12. The system of claim 11, wherein the time-frequency spectrogram data is divided into a plurality of frequency-corresponding time data sequences by composing data sequences along a time axis direction, and the time-frequency spectrogram data is divided into a plurality of time-corresponding frequency data sequences by composing data sequences along a frequency axis direction, the second convolutional neural network model comprising a time axis convolutional neural network model corresponding to the time axis direction and a frequency axis convolutional neural network model corresponding to the frequency axis direction;

the second convolutional neural network module is specifically configured to:

13. The system according to any one of claims 10 to 12, wherein the number of input channels of the second convolutional neural network model is the same as the number of data sequences obtained by dividing the time-frequency spectrogram data in a set direction.

14. The system of claim 10, wherein the determine device status module is specifically configured to:

15. The system of claim 10, wherein the data conversion module is specifically configured to:

16. The system according to claim 15, wherein the system further comprises a normalization processing module specifically configured to:

17. The system of claim 10,

18. The system of claim 14, wherein if the fusion model is a machine learning model, the fusion model is trained using the first model data sample, the second model data sample, and the corresponding device state sample.

19. An apparatus for state evaluation, the apparatus comprising: a processor and a memory, wherein the memory stores program code that, when executed by the processor, causes the processor to perform the steps of:

20. The device of claim 19, wherein the processor is specifically configured to:

21. The apparatus of claim 20, wherein said time-frequency spectrogram data is divided into a plurality of frequency-corresponding time data sequences by composing data sequences along a time axis direction, and wherein said time-frequency spectrogram data is divided into a plurality of time-corresponding frequency data sequences by composing data sequences along a frequency axis direction, said second convolutional neural network model comprising a time axis convolutional neural network model corresponding to said time axis direction and a frequency axis convolutional neural network model corresponding to said frequency axis direction;

the processor is specifically configured to:

22. The apparatus according to any one of claims 19 to 21, wherein the number of input channels of the second convolutional neural network model is the same as the number of data sequences obtained by dividing the time-frequency spectrogram data in a set direction.

23. The device of claim 19, wherein the processor is specifically configured to:

24. The device of claim 19, wherein the processor is further configured to:

25. The device of claim 24, wherein the processor is further configured to:

26. The apparatus of claim 19,

27. The device of claim 23, wherein if the fusion model is a machine learning model, the fusion model is trained using the first model data samples, the second model data samples, and the corresponding device state samples.

28. A computer storage medium having a computer program stored thereon, the program, when executed by a processor, implementing the steps of the method according to any one of claims 1 to 9.