CN111192257B

CN111192257B - Method, system and equipment for determining equipment state

Info

Publication number: CN111192257B
Application number: CN202010000656.9A
Authority: CN
Inventors: 郭双全; 许伟; 汪振江
Original assignee: Shanghai Electric Group Corp
Current assignee: Shanghai Electric Group Corp
Priority date: 2020-01-02
Filing date: 2020-01-02
Publication date: 2023-05-23
Anticipated expiration: 2040-01-02
Also published as: CN111192257A

Abstract

The invention discloses a method, a system and equipment for determining equipment states, which can be applied to data analysis of industrial equipment and improve the accuracy and reliability of state evaluation of the industrial equipment. The method comprises the following steps: converting sensor data of the device into standard time-frequency spectrogram data, wherein the standard time-frequency spectrogram data is time-frequency spectrogram data conforming to a preset standard specification; inputting the standard time-frequency spectrogram data into a first convolutional neural network model through an input channel of the first convolutional neural network model to obtain first model data; dividing the standard time-frequency spectrogram data into a plurality of data sequences according to a set direction, and inputting the data sequences into a second convolutional neural network model through a plurality of input channels of the second convolutional neural network model corresponding to the set direction to obtain second model data; and determining the state of the equipment according to the first model data and the second model data.

Description

Method, system and equipment for determining equipment state

Technical Field

The present invention relates to the field of industrial intelligence technologies, and in particular, to a method, a system, and an apparatus for determining a device state.

Background

In the field of industrial equipment, for example, large-scale equipment such as large-scale machine tools, wind turbines, steam turbines, industrial motors and the like are often equipped with an online monitoring system for guaranteeing stable and reliable operation, monitoring sensor measuring points are more, sampling frequency of key parameters such as vibration, electricity, pressure and the like is high, original data volume is large, and the traditional data analysis method using signal processing characteristic extraction as a means has the difficulties of high complexity, limited precision and the like.

The current signal characteristic extraction method aiming at the high-frequency sampling data comprises a time domain analysis method, a frequency domain analysis method, and a time-frequency analysis method (short-time Fourier transform, wavelet transform, empirical mode decomposition and the like). The time domain analysis mainly takes time domain statistics index calculation and correlation analysis as main materials, the attention to the frequency domain characteristics is not high, the frequency domain analysis is based on Fourier transformation, and the time domain analysis is more suitable for equipment stable signal analysis and cannot consider the time domain characteristics; the time-frequency matrix (or time frequency spectrum) generated by the time-frequency analysis method is processed to give consideration to the characteristics of the time domain and the frequency domain, but the traditional equipment state evaluation or fault judgment method such as cluster analysis, decision trees, gaussian mixture models and the like must be used for extracting typical characteristics according to manual professional experience, and has certain subjectivity and imperfection, and the equipment state evaluation result has uncertainty.

Disclosure of Invention

The invention provides a method, a system and equipment for determining equipment states, which can be applied to data analysis of industrial equipment, and aims at time spectrogram data obtained after the processing of sensing data of the industrial equipment, a plurality of convolutional neural network models are utilized to extract features of time spectrograms from different layers, so that the state evaluation effect of the industrial equipment is comprehensively improved, and the accuracy and the reliability of state evaluation of the industrial equipment are improved.

In a first aspect, the present invention provides a method of determining a status of a device, the method comprising:

converting sensor data of the device into standard time-frequency spectrogram data, wherein the standard time-frequency spectrogram data is time-frequency spectrogram data conforming to a preset standard specification;

inputting the standard time-frequency spectrogram data into a first convolutional neural network model through an input channel of the first convolutional neural network model to obtain first model data;

dividing the standard time-frequency spectrogram data into a plurality of data sequences according to a set direction, and inputting the data sequences into a second convolutional neural network model through a plurality of input channels of the second convolutional neural network model corresponding to the set direction to obtain second model data;

And determining the state of the equipment according to the first model data and the second model data.

As one possible implementation manner, dividing the time-frequency spectrogram data into a plurality of data sequences according to a set direction includes:

dividing the time-frequency spectrogram data into time data sequences corresponding to a plurality of frequencies in a mode of forming the data sequences along the direction of a time axis; and/or dividing the time-frequency spectrogram data into frequency data sequences corresponding to a plurality of moments in a mode of forming the data sequences along the direction of the frequency axis.

As one possible implementation manner, the time-frequency spectrogram data is divided into a plurality of time data sequences corresponding to frequencies in a manner of composing the data sequences along a time axis direction, and the time-frequency spectrogram data is divided into a plurality of frequency data sequences corresponding to moments in a manner of composing the data sequences along a frequency axis direction, wherein the second convolution neural network model comprises a time axis convolution neural network model corresponding to the time axis direction and a frequency axis convolution neural network model corresponding to the frequency axis direction;

inputting the plurality of data sequences into the second convolutional neural network model through a plurality of input channels of the second convolutional neural network model corresponding to the set direction to obtain second model data, wherein the second model data comprises:

Inputting the plurality of time data sequences into the time axis convolutional neural network model through a plurality of input channels of the time axis convolutional neural network model to obtain second time model data;

inputting the plurality of frequency data sequences into the frequency axis convolutional neural network model through a plurality of input channels of the frequency axis convolutional neural network model to obtain second frequency model data;

and determining the second time model data and the second frequency model data as second model data.

As a possible implementation manner, the number of input channels of the second convolutional neural network model is the same as the number of data sequences obtained by dividing the time-spectrum data according to a set direction.

As a possible implementation manner, determining the state of the device according to the first model data and the second model data includes:

inputting the first model data and the second model data into a fusion model, and determining the state of equipment, wherein the fusion model comprises: a formula model or a machine learning model.

As one possible implementation, converting sensor data of a device into standard time-frequency spectrogram data includes:

Signal processing is carried out on the sensor data of the equipment to obtain time-frequency spectrogram data;

establishing an interpolation function for a spectrum value corresponding to a data sequence of the time-frequency spectrogram data according to a preset resolution to obtain the time-frequency spectrogram data with the preset resolution;

and determining a time range and a frequency range of interception according to a preset rule for the obtained time-frequency spectrogram data with the preset resolution, and intercepting the time-frequency spectrogram data with the preset resolution according to the time range and the frequency range to obtain standard time-frequency spectrogram data.

As a possible implementation manner, determining the intercepting time range according to a preset rule includes:

determining a truncated time range according to the moment when the energy value of the frequency data sequence in the time spectrogram data with the preset resolution is larger than a preset threshold value; or (b)

Determining a truncated time range by taking a moment corresponding to a maximum energy value of a frequency data sequence in the time-frequency spectrogram data with the preset resolution as a center; or (b)

And determining the intercepted time range according to the time period corresponding to the time period, in which the sum of the energy values of the frequency data sequences in the set time period in the time spectrogram data of the preset resolution is larger than the preset value.

As a possible implementation manner, the method further comprises:

And converting the standard time-frequency spectrogram data into preset-dimension time-frequency spectrogram data through a normalization or standardization algorithm.

As a possible implementation manner, the device comprises,

the first convolutional neural network model is obtained by training a standard time-frequency spectrogram data sample and a corresponding first model data sample;

the second convolutional neural network model is obtained by training a data sequence sample in the standard time spectrum data samples and a corresponding second model data sample.

As a possible implementation manner, if the fusion model is a machine learning model, the fusion model is obtained by training with a first model data sample, a second model data sample and a corresponding device state sample.

In a second aspect, the present invention provides a system for determining the status of a device, the system comprising: the device comprises a data conversion module, a first convolutional neural network module, a second convolutional neural network module and a device state determining module, wherein:

the data conversion module is used for converting the sensor data of the equipment into standard time-frequency spectrogram data, wherein the standard time-frequency spectrogram data is time-frequency spectrogram data which accords with a preset standard specification;

The first convolutional neural network module is used for inputting the standard time-frequency spectrogram data into the first convolutional neural network model through an input channel of the first convolutional neural network model to obtain first model data;

the second convolutional neural network module is used for dividing the standard time-frequency spectrogram data into a plurality of data sequences according to a set direction, and inputting the data sequences into a second convolutional neural network model through a plurality of input channels of the second convolutional neural network model corresponding to the set direction to obtain second model data;

and the equipment state determining module is used for determining the state of the equipment according to the first model data and the second model data.

As a possible implementation manner, the second convolutional neural network module is specifically configured to:

The second convolutional neural network module is specifically configured to:

As a possible implementation manner, the device state determining module is specifically configured to:

As a possible implementation manner, the data conversion module is specifically configured to:

As a possible implementation manner, the system further comprises a standardized processing module, specifically configured to:

As a possible implementation manner, the first convolutional neural network model is obtained by training using a standard time-spectrum spectrogram data sample and a corresponding first model data sample;

In a third aspect, the present invention provides an apparatus for determining the status of an apparatus, the apparatus comprising: a processor and a memory, wherein the memory stores program code that, when executed by the processor, causes the processor to perform the steps of:

As a possible implementation manner, the processor is specifically configured to:

The processor is specifically configured to:

As a possible implementation, the processor is specifically further configured to:

As a possible implementation manner, the device comprises,

In a fourth aspect, the present invention provides a computer storage medium having stored thereon a computer program which when executed by a processor performs the steps of the method of the first aspect described above.

The method, the system and the equipment for determining the equipment state have the following beneficial effects:

the method can be applied to data analysis of industrial equipment, and aims at the time-frequency spectrogram data obtained after the processing of the sensing data of the industrial equipment, the characteristic extraction is carried out on the two-dimensional time-frequency spectrogram data through a first convolution neural network model, the characteristic extraction is carried out on one-dimensional data sequences in the time-frequency spectrogram data through a second convolution neural network model, the characteristic extraction is carried out on the time-frequency spectrogram from different layers through a plurality of convolution neural network models, the state evaluation effect of the industrial equipment is comprehensively improved, and the accuracy and the reliability of the state evaluation of the industrial equipment are improved.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings that are needed in the description of the embodiments will be briefly described below, it will be apparent that the drawings in the following description are only some embodiments of the present invention, and that other drawings can be obtained according to these drawings without inventive effort for a person skilled in the art.

FIG. 1 is a flowchart of a method for determining a device status according to an embodiment of the present invention;

FIG. 2 is a schematic diagram of a time-frequency spectrum diagram according to an embodiment of the present invention;

fig. 3 is a schematic structural diagram of a first convolutional neural network model according to an embodiment of the present invention;

FIG. 4 is a schematic diagram of dividing time-frequency spectrogram data according to an embodiment of the present invention;

FIG. 5 is a schematic diagram of another embodiment of dividing the time-frequency spectrogram data;

FIG. 6 is a schematic diagram of a second convolutional neural network model according to an embodiment of the present invention;

FIG. 7 is a schematic diagram of another second convolutional neural network model provided in an embodiment of the present invention;

FIG. 8 is a diagram of a standard time-frequency spectrum diagram according to an embodiment of the present invention;

FIG. 9 is a flowchart of a normalization process for the time-frequency spectrogram data according to an embodiment of the present invention;

FIG. 10 is a flowchart of an embodiment of determining a device status according to an embodiment of the present invention;

FIG. 11 is a schematic diagram of a system for determining a device status according to an embodiment of the present invention;

fig. 12 is a schematic diagram of an apparatus for determining an apparatus status according to an embodiment of the present invention.

Detailed Description

In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention will be described in further detail below with reference to the accompanying drawings, and it is apparent that the described embodiments are only some embodiments of the present invention, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.

In the embodiment of the invention, the term "and/or" describes the association relation of the association objects, which means that three relations can exist, for example, a and/or B can be expressed as follows: a exists alone, A and B exist together, and B exists alone. The character "/" generally indicates that the context-dependent object is an "or" relationship.

The application scenario described in the embodiment of the present invention is for more clearly describing the technical solution of the embodiment of the present invention, and does not constitute a limitation on the technical solution provided by the embodiment of the present invention, and as a person of ordinary skill in the art can know that the technical solution provided by the embodiment of the present invention is applicable to similar technical problems as the new application scenario appears. In the description of the present invention, unless otherwise indicated, the meaning of "a plurality" is two or more.

The embodiment of the invention can be applied to data analysis of industrial equipment, and is mainly used for analyzing the time-frequency spectrogram data obtained after the processing of the sensing data of the industrial equipment, evaluating the state of the industrial equipment, and the following is a simple description of the application scene of the embodiment of the invention:

in the field of industrial equipment, for example, large-scale equipment such as a large-scale machine tool, a wind turbine, a steam turbine, an industrial motor and the like are often equipped with an online monitoring system for guaranteeing stable and reliable operation, and because the monitoring points of the monitoring sensor are more, the sampling frequency of key parameters such as vibration, electricity, pressure and the like is higher, the obtained data volume of the sensor is larger, and the traditional data analysis method using the signal characteristic extraction of the data of the sensor as a means has the difficulties of high complexity, limited precision and the like.

The existing signal feature extraction method of the sensor data of the equipment obtained by high-frequency sampling comprises time domain analysis, frequency domain analysis and time-frequency analysis (such as short-time Fourier transform, wavelet transform, empirical mode decomposition and the like), wherein the time domain analysis mainly comprises time domain statistics index calculation and correlation analysis, the attention of frequency domain features is not high, the frequency domain analysis is based on Fourier transform, the frequency domain analysis is relatively suitable for stable signal analysis of the equipment, the time domain features of the signals cannot be considered, the time-frequency matrix (or time spectrum) generated by the time-frequency analysis method is processed, and the time domain features and the frequency domain features are considered, but the state evaluation or fault judgment method of the traditional equipment such as cluster analysis, decision tree, gaussian mixture model and the like must be used for extracting typical features according to manual professional experience, and has a certain subjectivity and incompleteness, and uncertainty exists in an evaluation result.

In order to solve the above technical problems, the embodiments of the present invention provide a method for determining a state of an apparatus, where after time-frequency analysis is performed on sensor data of the apparatus, time-frequency spectrogram data (or a time-frequency matrix) obtained after the time-frequency analysis is converted into standard time-frequency spectrogram data, the standard time-frequency spectrogram data is input into a plurality of convolutional neural network models to perform feature extraction on the time-frequency spectrogram data, so as to avoid subjectivity and imperfection caused by manual feature extraction, and based on two dimensions of time and frequency of the standard time-frequency spectrogram data, a two-dimensional convolutional neural network model (i.e., a first convolutional neural network model) for the standard time-frequency spectrogram data and a one-dimensional convolutional neural network model (i.e., a second convolutional neural network model) for a one-dimensional data sequence in the standard time-frequency spectrogram data are adopted to perform feature extraction on the standard time-frequency spectrogram data, and determine final state data of the apparatus according to the obtained first state data and the second state data.

As shown in fig. 1, the embodiment of the invention provides a specific implementation flow of a method for determining a device state, which is as follows:

step 100, converting sensor data of equipment into standard time-frequency spectrogram data, wherein the standard time-frequency spectrogram data is time-frequency spectrogram data conforming to a preset standard specification;

it should be noted that, in this embodiment, the sensor data of the device may be one data or may be a plurality of sensor data, and if the sensor data is a plurality of sensor data, the method for converting the sensor data of the device into standard time-frequency spectrogram data includes: sensor data of the device is converted into a standard time-frequency spectrogram data set corresponding to the plurality of sensor data.

It should be noted that, the apparatus in the embodiment of the present invention includes, but is not limited to: wind driven generator, coal mill, machine tool, generator set and crane; the type of the industrial equipment is not limited in this embodiment, and the method in this embodiment can be applied to the industrial equipment having sensor data to determine the state of the equipment.

Optionally, the method for obtaining the time-frequency spectrogram data by performing signal processing on the sensor data of the device includes, but is not limited to, the following ways: short-time fourier transform, wavelet transform, empirical mode decomposition. The present example does not overly define how the time-spectrum data is derived from the sensor data.

Optionally, signal processing is performed on the sensor signal (sensor data) of the device by a time-frequency analysis method, such as performing short-time fourier transform, wavelet transform, empirical mode decomposition and the like on the sensor signal, the sensor data of the device is converted into time-frequency spectrogram data, and energy densities or intensities of the sensor signal at different times and frequencies can be obtained according to the time-frequency spectrogram data, wherein the time-frequency spectrogram data utilizes the joint function of time and frequency to analyze the sensor signal, and the amplitude of the sensor signal at each time and frequency can be given.

It should be noted that, the time-frequency spectrogram data is a two-dimensional matrix, and each element in the matrix may be referred to as a spectrum value; alternatively, the time-frequency spectrogram data may be represented by fig. 2, wherein the horizontal axis represents time, the vertical axis represents frequency, and the spectral values represent the magnitudes of the sensor signals at the corresponding time and frequency.

The standard time-frequency spectrogram data in this embodiment is time-frequency spectrogram data meeting a preset standard specification, and it is easy to understand that the resolution and the size of the standard time-frequency spectrogram data in this embodiment are time-frequency spectrogram data meeting the preset standard specification, and the resolution and the size in the time-frequency spectrogram data obtained according to the sensor data of the device can be standardized to obtain the time-frequency spectrogram data meeting the preset standard specification.

Step 101, inputting the time-frequency spectrogram data into a first convolutional neural network model through an input channel of the first convolutional neural network model to obtain first model data;

in this embodiment, one piece of time-frequency spectrogram data may be input to the first convolutional neural network model to obtain first model data, or a set composed of a plurality of pieces of time-frequency spectrogram data may be input to the first convolutional neural network model to obtain a corresponding first model data set.

It should be noted that, the first convolutional neural network model in this embodiment may be obtained by training using a standard time-frequency spectrogram data sample and a corresponding first model data sample, where the standard time-frequency spectrogram data sample is obtained by performing normalization processing according to time-frequency spectrogram data in a history period, the first model data sample is a data sample corresponding to the normalized time-frequency spectrogram data, and the first model data sample may be state data of a device or other data, and may obtain the corresponding first model data sample according to a user requirement.

Optionally, the standard time-spectrum data sample and the corresponding first model data sample are obtained by:

Acquiring time-frequency spectrogram data of equipment in a set time period, and carrying out standardized processing on the time-frequency spectrogram data to obtain standard time-frequency spectrogram data samples; acquiring first model data of equipment in a set time period to obtain a first model data sample; and matching the standard time spectrogram data in the set time period with the first model data in the set time period, namely, the standard time spectrogram data in the set time period corresponds to only one model data, and obtaining a standard time spectrogram data sample and a corresponding first model data sample.

Optionally, the standard time-frequency spectrogram data in the set time period and the first model data in the set time period are matched through a unique identification code.

Optionally, storing the standard time-frequency spectrogram data in the set time period in a time-frequency spectrogram training storage area; and storing the first model data in the set time period in a state data training storage area.

The first convolutional neural network model in the embodiment is a two-dimensional convolutional neural network model and comprises an input layer, an hidden layer and an output layer, wherein the input layer comprises an input channel for receiving two-dimensional data, namely receiving time-frequency spectrogram data (or a time-frequency matrix), the number of the hidden layers is larger than 1, the first hidden layer behind the input layer is the two-dimensional convolutional layer, the convolutional kernel is the two-dimensional convolutional kernel, other hidden layers comprise the convolutional layer, a pooling layer, a full-connection layer and the like, the size of the first model data output by the output layer can be the self-defined data size, the state data size of equipment defaulting to the system, and the preset data size can be also obtained according to the use requirement.

The structure of the first convolutional neural network model in this embodiment is shown in fig. 3, and includes an input layer 300, an hidden layer 301, and an output layer 302, where the number of channels of the input layer is 1, and the size of a channel can be determined according to the size of the time-frequency spectrogram data (or time-frequency matrix), and it is easy to understand that the size of the channel can be equal to the length N of the time data sequence multiplied by the length M of the frequency data sequence;

the first hidden layer after the input layer is a two-dimensional convolution layer, the channel number of the hidden layer (i.e. the number of convolution kernels) is N1 c1, the size of the convolution kernels is w1_c1×h1_c1, wherein the number and the size of the convolution kernels of the hidden layer can be set according to the need, and optionally, can be determined according to the size of the output first state data and the number of the hidden layers in the first convolution neural network model;

the number of hidden layers is greater than 1, and other hidden layers 2 to N except the first hidden layer may include: the number of other hidden layers and the types specifically included are not excessively limited by the implementation;

the data size output by the output layer may be consistent with a first model data size, which may be state data of the device, including, but not limited to, any one or more of the following:

Gearbox bearing performance status data; motor health status data; gear running state data; vibration state data; bearing operating state data.

The state data includes, but is not limited to, a normal or fault state, a power generation or non-power generation state, a rated or overrun state, and other working condition states, and the type of the state is not limited too much in this embodiment.

Alternatively, the state data of the device may be discrete values or continuous values, for example, the state data of the device may be represented by binary numbers, for example, the gearbox bearing performance state data may be set to three states of 00 (for representing excellent performance), 01 (for representing poor performance) and 10 (for representing poor performance), the motor health state data may be set to three states of 00 (for representing healthy), 01 (for representing sub-healthy) and 10 (for representing unhealthy), and if the state data of the device includes a plurality of state data, the number of binary numbers of the state data of the device may be determined according to the number of types of the included state data.

102, dividing the standard time-frequency spectrogram data into a plurality of data sequences according to a set direction, and inputting the data sequences into a second convolutional neural network model through a plurality of input channels of the second convolutional neural network model corresponding to the set direction to obtain second model data;

Optionally, in this embodiment, the time-frequency spectrogram data may be divided into a set number of data sequences, and in this embodiment, the number of the divided data sequences is not limited too much.

In this embodiment, one standard time-frequency spectrogram data may be divided into a plurality of data sequences, or each standard time-frequency spectrogram data in a standard time-frequency spectrogram data set formed by a plurality of standard time-frequency spectrogram data may be divided into a plurality of data sequences, so as to obtain a data sequence set, and the data sequence set is input into a second convolutional neural network model to obtain a corresponding second model data set.

In implementation, the time-frequency spectrogram data is divided into a plurality of data sequences according to a set direction, and this embodiment provides the following three embodiments, which are described below with reference to fig. 4 for convenience of description:

the method 1 comprises the steps of dividing the time-frequency spectrogram data into time data sequences corresponding to a plurality of frequencies in a mode of forming the data sequences along the direction of a time axis;

optionally, in this embodiment, the time-frequency spectrogram data may be divided into a set number of time data sequences, where each time data sequence corresponds to a frequency; the time-frequency spectrogram data may also be divided into a number of time-data sequences equal to the number of frequencies according to the number of frequencies (i.e. the length of the frequency sequences) in the time-frequency spectrogram data, wherein each time-data sequence corresponds to a frequency.

As shown in fig. 4, the time axis direction is a left-to-right (or right-to-left) direction, and the time-frequency spectrogram is divided according to the left-to-right direction, so that m time data sequences can be obtained, wherein the frequency f1 corresponds to the time data sequence x1, the frequency f2 corresponds to the time data sequence x2, the frequency f3 corresponds to the time data sequences x3 and … …, and the frequency fm corresponds to the time data sequence xm; the time data sequence x1, the time data sequence x2, the time data sequences x3 and … … and the time data sequence xm all comprise n data.

Mode 2, dividing the time-frequency spectrogram data into frequency data sequences corresponding to a plurality of moments in a mode of forming data sequences along the direction of a frequency axis;

optionally, in this embodiment, the time-frequency spectrogram data may be divided into a set number of frequency data sequences, where each frequency data sequence corresponds to a time instant; the time-frequency spectrogram data may also be divided into a number of frequency data sequences equal to the number of time instants according to the number of time instants (i.e. the length of the time sequence) in the time-frequency spectrogram data, wherein each frequency data sequence corresponds to a time instant.

As shown in fig. 5, the frequency axis direction is a direction from bottom to top (or from top to bottom), and the time spectrogram is divided in the direction from bottom to top, so that n frequency data sequences can be obtained, where time t1 corresponds to the frequency data sequence y1, time t2 corresponds to the frequency data sequence y2, time t3 corresponds to the frequency data sequences y3 and … …, and time tn corresponds to the frequency data sequence yn; the frequency data sequence y1, the frequency data sequence y2, the frequency data sequences y3 and … … and the frequency data sequence yn all comprise m data.

And 3, dividing the time-frequency spectrogram data into time data sequences corresponding to a plurality of frequencies in a mode of forming the data sequences along the time axis direction, and dividing the time-frequency spectrogram data into frequency data sequences corresponding to a plurality of moments in a mode of forming the data sequences along the frequency axis direction.

In combination with fig. 4 and fig. 5, the time-frequency spectrogram data are divided in a time axis direction and a frequency axis direction respectively, so as to obtain m time data sequences and n frequency data sequences, wherein the time data sequences comprise n data, and the frequency data sequences comprise m data.

The method of obtaining the second model data in this example is described below in conjunction with the above three embodiments:

the method comprises the steps of 1, dividing the time-frequency spectrogram data into time data sequences corresponding to a plurality of frequencies in a mode of forming the data sequences along the direction of a time axis;

inputting the time data sequences into the second convolutional neural network model through a plurality of input channels of the second convolutional neural network model corresponding to the time axis direction to obtain second model data;

wherein the number of input channels is not less than the number of time data series.

Dividing the time-frequency spectrogram data into frequency data sequences corresponding to a plurality of moments in a mode of forming data sequences along the direction of a frequency axis;

inputting the plurality of frequency data sequences into the second convolutional neural network model through a plurality of input channels of the second convolutional neural network model corresponding to the frequency axis direction to obtain second model data;

wherein the number of input channels is not less than the number of frequency data sequences.

Dividing the time-frequency spectrogram data into time data sequences corresponding to a plurality of frequencies in a mode of forming a data sequence along the direction of a time axis; dividing the time-frequency spectrogram data into frequency data sequences corresponding to a plurality of moments in a mode of forming data sequences along the direction of a frequency axis;

in the method, the second convolution neural network model comprises a time axis convolution neural network model corresponding to the time axis direction and a frequency axis convolution neural network model corresponding to the frequency axis direction;

inputting the plurality of time data sequences into the time axis convolutional neural network model through a plurality of input channels of the time axis convolutional neural network model corresponding to the time axis direction, so as to obtain second time model data;

Inputting the plurality of frequency data sequences into the frequency axis convolutional neural network model through a plurality of input channels of the frequency axis convolutional neural network model corresponding to the frequency axis direction, so as to obtain second frequency model data;

the number of input channels of the time axis convolutional neural network model is not less than the number of the time data sequences; the number of input channels of the frequency axis convolutional neural network model is not less than the number of frequency data sequences.

As an alternative embodiment, the number of input channels of the second convolutional neural network model is the same as the number of data sequences obtained by dividing the time-spectrum data in a set direction.

In the implementation, if the time-frequency spectrogram data are divided into time data sequences corresponding to a plurality of frequencies in a mode of forming the data sequences along the direction of a time axis; the number of input channels of the second convolutional neural network model is the same as the number of time data sequences;

dividing the time-frequency spectrogram data into frequency data sequences corresponding to a plurality of moments in a mode of forming the data sequences along the direction of a frequency axis; the number of input channels of the second convolutional neural network model is the same as the number of frequency data sequences;

Dividing the time-frequency spectrogram data into time data sequences corresponding to a plurality of frequencies in a mode of forming the data sequences along the direction of a time axis, and dividing the time-frequency spectrogram data into frequency data sequences corresponding to a plurality of moments in a mode of forming the data sequences along the direction of the frequency axis; the number of input channels of the time-axis convolutional neural network model is equal to the number of time data sequences; the number of input channels of the frequency axis convolutional neural network model is equal to the number of frequency data sequences.

It should be noted that, the second convolutional neural network model in this embodiment is obtained by training using a data sequence sample in a standard time-frequency spectrogram data sample and a corresponding second model data sample, where the standard time-frequency spectrogram data sample is obtained by performing standardization processing according to time-frequency spectrogram data in a history period, the data sequence sample is obtained by dividing the standard time-frequency spectrogram data according to a set direction, the second model data sample is a data sample corresponding to a data sequence sample in the standardized time-frequency spectrogram data sample, and the second model data sample may be state data of a device or other data, or may obtain a corresponding second model data sample according to a user requirement.

The first model data sample and the second model data sample may be the same size and/or type.

Optionally, the data sequence samples in the standard time-spectrum data samples and the corresponding second model data samples are obtained by:

obtaining time-frequency spectrogram data of equipment in a set time period, carrying out standardization processing on the time-frequency spectrogram data to obtain standard time-frequency spectrogram data samples, and obtaining data sequence samples in the standard time-frequency spectrogram data samples in the three modes; acquiring second model data of the equipment within a set time period to obtain a second model data sample; and matching the data sequence samples in the standard time spectrum data samples in the set time period with the second model data in the set time period, namely, the data sequence samples in the standard time spectrum data samples in the set time period correspond to only one model data, and obtaining the data sequence samples in the standard time spectrum data samples and the corresponding second model data samples.

Optionally, the data sequence in the standard time-frequency spectrogram data in the set time period and the second model data in the set time period are matched through a unique identification code.

The second convolutional neural network model in the embodiment is a one-dimensional convolutional neural network model and comprises an input layer, hidden layers and an output layer, wherein the input layer comprises a plurality of input channels and is used for receiving a plurality of one-dimensional data sequences, the number of the hidden layers is larger than 1, the first hidden layer after the input layer is a one-dimensional convolutional layer, the convolutional kernel is a one-dimensional convolutional kernel, other hidden layers comprise a convolutional layer, a pooling layer, a full-connection layer and the like, the second model data output by the output layer can be state data of equipment, other data can be output by the output layer, the size of the second model data can be custom model data, the state data size of equipment defaulted by a system can be output by the second model data, and the size of the second model data can be preset according to the use requirement.

It should be noted that, in this embodiment, if the second convolutional neural network model is a model corresponding to the set direction, and if the time spectrum data is divided according to two set directions, such as the frequency axis direction and the time axis direction in this embodiment, two second convolutional neural network models are required in this embodiment, including a frequency axis convolutional neural network model and a time axis convolutional neural network model, where the frequency axis convolutional neural network model corresponds to a plurality of data sequences obtained by dividing the time spectrum data according to the frequency axis direction, and the time axis convolutional neural network model corresponds to a plurality of data sequences obtained by dividing the time spectrum data according to the time axis direction, so the second convolutional neural network model in this embodiment may be one or a plurality, and is specifically determined by the number of set directions.

The structure of the second convolutional neural network model in this embodiment is shown in fig. 6 and 7, where the number of input channels of the input layer is the number of input sequences, and the size of the input channels may be determined according to the size of the data sequence divided by the standard time-frequency spectrogram data.

As shown in fig. 6, the second convolutional neural network model includes an input layer 600, an hidden layer 601, and an output layer 602, the standard time-frequency spectrogram data is divided into a plurality of time data sequences in a time axis direction, the number of input channels of the input layer of the second convolutional neural network model is M, and the size of the input channels may be equal to the length N of the time data sequences into which the standard time-frequency spectrogram data is divided;

the first hidden layer behind the input layer is a one-dimensional convolution layer, the channel number of the hidden layer (namely, the number of convolution kernels) is N2_c1, the size of the convolution kernels is W2_c1, wherein the number and the size of the convolution kernels of the hidden layer can be set according to the requirement, and optionally, the number and the size of the hidden layer can be determined according to the size of the output first state data and the number of the hidden layers in the first convolution neural network model;

As shown in fig. 7, the second convolutional neural network model includes an input layer 700, an implicit layer 701, and an output layer 702, the standard time-frequency spectrogram data is divided into a plurality of frequency data sequences according to the frequency axis direction, the number of input channels of the input layer of the second convolutional neural network model is N, and the size of the input channels may be equal to the length M of the frequency data sequences divided by the standard time-frequency spectrogram data;

the first hidden layer behind the input layer is a one-dimensional convolution layer, the channel number of the hidden layer (namely, the number of convolution kernels) is N3-c 1, the size of the convolution kernels is H3-c 1, wherein the number and the size of the convolution kernels of the hidden layer can be set according to the needs, and optionally, the number and the size of the hidden layer can be determined according to the size of the output first model data and the number of the hidden layers in the first convolution neural network model;

The first model data and the second model data in this embodiment may be the same or different, and this embodiment is not limited thereto.

It should be noted that, the first convolutional neural network model and the second convolutional neural network model may be trained separately or simultaneously, which is not limited herein.

And step 103, determining the state of the equipment according to the first model data and the second model data.

The status of the device in this implementation includes, but is not limited to, any one or more of the following:

Optionally, if the first model data and the second model data are the same, the state of the device may be the first model data or the second model data; if the first model data and the second model data are different, the state of the device may be determined according to a set rule, for example, the first model data and the second model data may be weighted and summed according to weights corresponding to the first model data and the second model data, and the state of the device may be determined according to the sum value.

As an alternative embodiment, determining the state of the device according to the first model data and the second model data includes:

It should be noted that, in this embodiment, the first model data and the second model data may be input into a fusion model, to determine a state of the device;

The first model data set and the second model data set may be input into a fusion model to determine a state set of the device.

If the second model data is obtained by adopting the method 3, the second model data includes: second time model data and second frequency model data, the status of the device is determined by:

determining a state of the device according to the first model data, the second time model data and the second frequency model data;

optionally, the first model data, the second time model data and the second frequency model data are input into a fusion model, the state of the equipment is determined, and if the fusion model is a machine learning model, the fusion model can be obtained by training the first model data, the second time model data, the second frequency model data and corresponding equipment state samples.

If the fusion model is a machine learning model, optionally, the fusion model includes, but is not limited to, any of the following:

a convolutional neural network model; a decision tree model; random forest models.

In implementation, the training of the first convolutional neural network model, the second convolutional neural network model and the fusion model may be synchronous iteration training, or may be training the first convolutional neural network model and the second convolutional neural network model first, and after the training is completed, training the fusion model by using output data of the first convolutional neural network model and the second convolutional neural network model and corresponding equipment state samples as training samples.

If the first convolutional neural network model, the second convolutional neural network model and the fusion model are trained simultaneously, the first convolutional neural network model, the second convolutional neural network model and the fusion model can be regarded as one comprehensive model, and then the comprehensive model can be trained by using a standard time spectrogram data sample, a data sequence sample in the standard time spectrogram data sample and a corresponding equipment state sample.

The embodiment also provides a method for carrying out standardized processing on the time-frequency spectrogram data, and the specific implementation mode is as follows:

and determining a time range and a frequency range of interception according to a preset rule or a preset size for the obtained time-frequency spectrogram data with preset resolution, and intercepting the time-frequency spectrogram data with the preset resolution according to the time range and the frequency range to obtain standard time-frequency spectrogram data.

Optionally, determining the intercepted time range according to a preset rule includes:

Determining a intercepted time range and a intercepted frequency range according to the corresponding time and frequency when the energy value corresponding to the frequency data sequence in the time-frequency spectrogram data with the preset resolution meets the preset condition; or (b)

Determining a intercepted time range and a intercepted frequency range according to the corresponding moment and frequency when the root mean square value corresponding to the frequency data sequence in the time spectrogram data with the preset resolution meets the preset condition; or (b)

And determining the intercepted time range and frequency range according to the time and frequency corresponding to the average value corresponding to the frequency data sequence in the time-frequency spectrogram data of the preset resolution meeting the preset condition.

It should be noted that the preset resolution and the preset size are the resolution and the size of the time-frequency spectrogram, wherein the preset resolution refers to the time interval and the frequency interval of the time-frequency spectrogram are preset, the preset size refers to the time range and the frequency range of the time-frequency spectrogram are preset, the abscissa axis and the ordinate axis of the standard time-frequency spectrogram data obtained according to the preset resolution and the preset size are shown in fig. 8, and the time axis range is T ₁ ～T _N The frequency axis range is F ₁ ～F _M The time interval is T ₂ -T ₁ Frequency interval of F ₂ -F ₁ 。

In practice, the data sequence of the time-frequency spectrogram data includes two types: a frequency data sequence and a time data sequence, for example, a data sequence corresponding to a time axis direction in the time spectrogram data is a time data sequence, and a data sequence corresponding to a frequency axis direction in the time spectrogram data is a frequency data sequence;

According to the preset resolution, the time interval and the frequency interval of the time-frequency spectrogram can be determined, if the time interval of the time-frequency spectrogram data is different from the preset resolution, an interpolation function can be established for the spectrum value corresponding to each time data sequence in the time-frequency spectrogram data, and the time-frequency spectrogram data identical to the time interval of the preset time-frequency spectrogram is generated;

if the frequency interval of the time-frequency spectrogram data is different from the preset resolution, an interpolation function can be established for the spectrum value corresponding to each frequency data sequence in the time-frequency spectrogram data, so that the time-frequency spectrogram data with the same frequency interval as the preset time-frequency spectrogram is generated;

if the time interval and the frequency interval of the time-frequency spectrogram data are different from the preset resolution, an interpolation function can be established for each time data sequence and the spectrum value corresponding to each frequency data sequence in the time-frequency spectrogram data, so that the time-frequency spectrogram data identical to the time interval and the frequency interval of the preset time-frequency spectrogram are generated.

It is easy to understand that the energy value in the time-frequency spectrogram data is an energy value corresponding to the frequency data sequence at one time, and the energy value can be obtained by calculating the spectrum value corresponding to the frequency data sequence at the time.

Optionally, determining the intercepted time range according to a preset rule includes any one of the following modes:

mode 1, determining a truncated time range according to a moment when an energy value of a frequency data sequence in the time spectrogram data of the preset resolution is larger than a preset threshold value;

and determining the intercepted time range according to the minimum time and the maximum time corresponding to the energy value larger than the preset threshold.

Mode 2, determining a truncated time range by taking a moment corresponding to a maximum energy value of a frequency data sequence in the time spectrum diagram data of the preset resolution as a center;

and taking the moment corresponding to the maximum energy value as a center, and determining the intercepting time range according to the preset length.

And 3, determining the intercepted time range according to the fact that the sum of energy values of the frequency data sequences in the set time period in the time spectrogram data of the preset resolution is larger than the time period corresponding to the preset value.

Optionally, determining the intercepted frequency range according to a preset rule includes any one of the following:

determining a intercepted frequency range according to the frequency corresponding to the preset condition satisfied by the spectrum value corresponding to the time data sequence in the time-frequency spectrogram data with the preset resolution; or (b)

And determining the intercepted frequency range according to the preset frequency range.

In this embodiment, the determination of the truncated time range and the truncated frequency range is merely an example, and in implementation, the manner of determining the truncated time range and the truncated frequency range may be defined according to the requirements, which is not limited too much in this embodiment.

It should be noted that, in the embodiment, the sensor data of the device may be one type of sensor data, or may be multiple types of sensor data, and in implementation, the multiple types of sensor data may be converted into standard time-frequency spectrogram data (or a standard time-frequency spectrogram data set) corresponding to the same type of sensor data;

if the sensor data of the device includes multiple types of sensor data, the embodiment may also determine the state of the device by using multiple first convolutional neural network models and multiple second convolutional neural network models to the sensor data of the multiple types of devices at the same time; after each type of sensor data is converted into standard time-frequency spectrogram data, the standard time-frequency spectrogram data is input into a first convolution neural network model and a second convolution neural network model corresponding to the type of sensor data, and the state of equipment is determined.

In implementation, if the resolution (including time interval and frequency interval) and the size (including time range and frequency range) of the time-frequency spectrogram data obtained by performing signal processing according to the sensor data of the device are different from the preset resolution and size, the time-frequency spectrogram data is normalized, as shown in fig. 9, and a specific normalization process flow is as follows:

step 900, establishing an interpolation function for spectrum values corresponding to each frequency data sequence in the time-frequency spectrogram data according to a preset resolution;

step 901, establishing an interpolation function for spectrum values corresponding to each time data sequence in the time-frequency spectrogram data according to a preset resolution;

step 902, generating first time-frequency spectrogram data with the same resolution as the preset resolution;

step 903, determining a time range and a frequency range for intercepting the first time spectrogram data according to a preset rule;

step 904, determining second time-frequency spectrogram data according to the time range and the frequency range;

step 905, intercepting the second time-frequency spectrogram data according to a preset size to obtain standard time-frequency spectrogram data.

As an alternative implementation manner, after the standard time-frequency spectrogram data is obtained, the standard time-frequency spectrogram data can be converted into time-frequency spectrogram data with a preset dimension through a normalization or standardization algorithm, for example, the normalization or standardization processing is performed on the spectrum value (element value) in the standard time-frequency spectrogram data, and if the spectrum value in the original standard time-frequency spectrogram data is a value from 0.01 to 0.09, the normalization or standardization processing is performed to obtain a spectrum value from 0 to 1.

In this embodiment, the standardized time-frequency spectrogram data obtained by the sensor data of the device is subjected to standardized processing, and the standard time-frequency spectrogram data is converted into the time-frequency spectrogram data with a preset dimension, so that a standardized time-frequency spectrogram data set with uniform resolution and size can be finally obtained, the standardized time-frequency spectrogram data set is convenient to use for training and using the first convolutional neural network model, the second convolutional neural network model and the fusion model, the training effect of the model can be greatly improved, and the device state determined by using the model has higher accuracy and reliability.

It should be noted that, in the embodiment of the present invention, different storage areas are provided for storing different data, which may be real-time storage or storage of setting a period at a real-time interval, and the embodiment does not limit the storage time too much.

Optionally, a time-frequency spectrogram data sample is stored through a time-frequency spectrogram training storage area, and a first model data sample corresponding to the time-frequency spectrogram data sample is stored through a state data training storage area;

and/or storing data sequence samples in the time-frequency spectrogram data samples through a time-frequency spectrogram training storage area, and storing second model data samples corresponding to the data sequence samples through a state data training storage area.

The processor may read the data in the time-frequency spectrogram training memory area, the state data training memory area, train the first convolutional neural network model, and/or train the second convolutional neural network model in the memory area.

Optionally, the time-frequency spectrogram data converted by the sensor data is stored by a time-frequency spectrogram evaluation storage area, and the first model data obtained by the first convolutional neural network model is stored by a state data evaluation storage area;

and/or storing a plurality of data sequences dividing the time-frequency spectrogram data through a time-frequency spectrogram evaluation storage area, and storing second model data obtained through a second convolutional neural network model through a state data evaluation storage area;

and/or storing the state of the device by evaluating the storage area with the state data.

If the state of the device is determined from the first model data and the second model data by the fusion model, the state of the device output by the fusion model may be stored in the state data evaluation storage area.

Optionally, the first convolutional neural network model and the second convolutional neural network model are stored through a model storage area, or the first convolutional neural network model, the second convolutional neural network model and the fusion model are stored through a model storage area; the first convolutional neural network model, the second convolutional neural network model and the fusion model stored in the model storage area are all trained models.

In implementation, on one hand, the integrated model in the model storage area may be a combined model of the first convolutional neural network model, the second convolutional neural network model and the fusion model, the time-frequency spectrogram data (or the time-frequency spectrogram data set) and the state of the corresponding device (or the state set of the device) are input to train the integrated model, on the other hand, after training is completed, the integrated model in the model storage area may be called, and the time-frequency spectrogram data (or the time-frequency spectrogram data set) needing to be subjected to device state evaluation is input to the integrated model to obtain the state of the corresponding device.

As shown in fig. 10, in the following, a method for determining a device state according to an embodiment of the present invention is described in detail with reference to an implementation procedure of normalizing time spectrum data in this embodiment, where a specific implementation procedure is as follows:

step 1000, performing signal processing on sensor data of equipment to obtain time-frequency spectrogram data;

step 1001, establishing an interpolation function for a spectrum value corresponding to a data sequence of the time-frequency spectrogram data according to a preset resolution, so as to obtain the time-frequency spectrogram data of the preset resolution;

Step 1002, determining a time range and a frequency range of interception according to a preset rule for the obtained time-frequency spectrogram data with preset resolution, and intercepting the time-frequency spectrogram data with preset resolution according to the time range and the frequency range to obtain standard time-frequency spectrogram data;

step 1003, converting the standard time-frequency spectrogram data into time-frequency spectrogram data with a preset dimension through a normalization or standardization algorithm;

step 1004, inputting the time-frequency spectrogram data to a first convolutional neural network model through an input channel of the first convolutional neural network model to obtain first model data;

step 1005, dividing the time-frequency spectrogram data into time data sequences corresponding to a plurality of frequencies in a mode of forming a data sequence along the direction of a time axis;

step 1006, inputting the plurality of time data sequences to the time axis convolutional neural network model through a plurality of input channels of the time axis convolutional neural network model corresponding to the time axis direction, so as to obtain second time model data;

step 1007, dividing the time-frequency spectrogram data into frequency data sequences corresponding to a plurality of moments in a mode of forming data sequences along the direction of a frequency axis;

Step 1008, inputting the plurality of frequency data sequences to the frequency axis convolutional neural network model through a plurality of input channels of the frequency axis convolutional neural network model corresponding to the frequency axis direction, so as to obtain second frequency model data;

step 1009, determining the second time model data and the second frequency model data as second model data;

and step 1010, inputting the first model data and the second model data into a fusion model to determine the state of the equipment.

The

steps

1004, 1005, 1007 may be performed simultaneously.

Based on the same inventive concept, the embodiment of the present invention further provides a system for determining a device state, and since the system is a system in the method in the embodiment of the present invention and the principle of the system for solving the problem is similar to that of the method, the implementation of the system may refer to the implementation of the method, and the repetition is omitted.

As shown in fig. 11, the system includes: a data conversion module 1100, a first convolutional neural network module 1101, a second convolutional neural network module 1102, a determine device state module 1103, wherein:

the data conversion module 1100 is configured to convert sensor data of the device into standard time-frequency spectrogram data, where the standard time-frequency spectrogram data is time-frequency spectrogram data that accords with a preset standard specification;

The first convolutional neural network module 1101 is configured to input the standard time-frequency spectrogram data to the first convolutional neural network model through an input channel of the first convolutional neural network model, so as to obtain first model data;

the second convolutional neural network module 1102 is configured to divide the standard time-frequency spectrogram data into a plurality of data sequences according to a set direction, and input the plurality of data sequences to a second convolutional neural network model through a plurality of input channels of the second convolutional neural network model corresponding to the set direction, so as to obtain second model data;

the device state determining module 1103 is configured to determine a state of the device according to the first model data and the second model data.

As a possible implementation manner, the second convolutional neural network module 1102 is specifically configured to:

The second convolutional neural network module 1102 is specifically configured to:

As a possible implementation manner, the device state determining module 1103 is specifically configured to:

As a possible implementation manner, the data conversion module 1100 is specifically configured to:

Based on the same inventive concept, the embodiment of the present invention further provides an apparatus for determining an apparatus state, and since the apparatus is the apparatus in the method in the embodiment of the present invention and the principle of the apparatus for solving the problem is similar to that of the method, the implementation of the apparatus may refer to the implementation of the method, and the repetition is omitted.

As shown in fig. 12, the apparatus includes: a processor 1200 and a memory 1201, wherein the memory 1201 stores program code that, when executed by the processor 1200, causes the processor 1200 to perform the steps of:

As a possible implementation, the processor 1200 is specifically configured to:

The processor 1200 is specifically configured to:

As a possible implementation, the processor 1200 is specifically configured to:

As a possible implementation, the processor 1200 is specifically further configured to:

As a possible implementation, the processor 1200 is specifically configured to:

As a possible implementation manner, the device comprises,

Based on the same inventive concept, the embodiments of the present invention also provide a computer storage medium having stored thereon a computer program which, when executed by a processor, performs the steps of:

It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, magnetic disk storage, optical storage, and the like) having computer-usable program code embodied therein.

The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

It will be apparent to those skilled in the art that various modifications and variations can be made to the present invention without departing from the spirit or scope of the invention. Thus, it is intended that the present invention also include such modifications and alterations insofar as they come within the scope of the appended claims or the equivalents thereof.

Claims

1. A method of determining a status of a device, the method comprising:

dividing the standard time-frequency spectrogram data into a plurality of data sequences according to a set direction, and inputting the data sequences into a second convolutional neural network model through a plurality of input channels of the second convolutional neural network model corresponding to the set direction to obtain second model data; dividing the time-frequency spectrogram data into a plurality of data sequences according to a set direction, wherein the method comprises the following steps: dividing the time-frequency spectrogram data into time data sequences corresponding to a plurality of frequencies in a mode of forming the data sequences along the direction of a time axis; and/or dividing the time-frequency spectrogram data into frequency data sequences corresponding to a plurality of moments in a mode of forming the data sequences along the direction of the frequency axis; if the time-frequency spectrogram data are divided into time data sequences corresponding to a plurality of frequencies in a mode of forming a data sequence along a time axis direction, and the time-frequency spectrogram data are divided into frequency data sequences corresponding to a plurality of moments in a mode of forming a data sequence along a frequency axis direction, the second convolution neural network model comprises a time axis convolution neural network model corresponding to the time axis direction and a frequency axis convolution neural network model corresponding to the frequency axis direction, the plurality of time data sequences are input into the time axis convolution neural network model through a plurality of input channels of the time axis convolution neural network model, and second time model data are obtained; inputting the plurality of frequency data sequences into the frequency axis convolutional neural network model through a plurality of input channels of the frequency axis convolutional neural network model to obtain second frequency model data; determining the second time model data and the second frequency model data as second model data;

2. The method of claim 1, wherein the number of input channels of the second convolutional neural network model is the same as the number of data sequences obtained by dividing the time-spectrum data in a set direction.

3. The method of claim 1, wherein determining the status of the device based on the first model data and the second model data comprises:

4. The method of claim 1, wherein converting the sensor data of the device to standard time-frequency spectrogram data comprises:

5. The method of claim 4, further comprising:

6. The method of claim 1, wherein the step of determining the position of the substrate comprises,

7. A method according to claim 3, wherein if the fusion model is a machine learning model, the fusion model is trained using a first model data sample, a second model data sample, and a corresponding device state sample.

8. A system for determining a status of a device, the system comprising: the device comprises a data conversion module, a first convolutional neural network module, a second convolutional neural network module and a device state determining module, wherein:

the second convolutional neural network module is used for dividing the standard time-frequency spectrogram data into a plurality of data sequences according to a set direction, and inputting the data sequences into a second convolutional neural network model through a plurality of input channels of the second convolutional neural network model corresponding to the set direction to obtain second model data; the method is particularly used for: dividing the time-frequency spectrogram data into time data sequences corresponding to a plurality of frequencies in a mode of forming the data sequences along the direction of a time axis; and/or dividing the time-frequency spectrogram data into frequency data sequences corresponding to a plurality of moments in a mode of forming the data sequences along the direction of the frequency axis; if the time-frequency spectrogram data is divided into time data sequences corresponding to a plurality of frequencies in a manner of forming a data sequence along a time axis direction, and the time-frequency spectrogram data is divided into frequency data sequences corresponding to a plurality of moments in a manner of forming a data sequence along a frequency axis direction, the second convolutional neural network model comprises a time axis convolutional neural network model corresponding to the time axis direction and a frequency axis convolutional neural network model corresponding to the frequency axis direction, the second convolutional neural network module is specifically configured to: inputting the plurality of time data sequences into the time axis convolutional neural network model through a plurality of input channels of the time axis convolutional neural network model to obtain second time model data; inputting the plurality of frequency data sequences into the frequency axis convolutional neural network model through a plurality of input channels of the frequency axis convolutional neural network model to obtain second frequency model data; determining the second time model data and the second frequency model data as second model data;

9. The system of claim 8, wherein the number of input channels of the second convolutional neural network model is the same as the number of data sequences that are obtained by dividing the time-spectrum data in a set direction.

10. The system of claim 8, wherein the determining device status module is specifically configured to:

11. The system of claim 8, wherein the data conversion module is specifically configured to:

12. The system according to claim 11, characterized in that it further comprises a standardized processing module, in particular for:

13. The system of claim 8, wherein the system further comprises a controller configured to control the controller,

14. The system of claim 10, wherein if the fusion model is a machine learning model, the fusion model is trained using a first model data sample, a second model data sample, and a corresponding device state sample.

15. An apparatus for state assessment, the apparatus comprising: a processor and a memory, wherein the memory stores program code that, when executed by the processor, causes the processor to perform the steps of:

dividing the standard time-frequency spectrogram data into a plurality of data sequences according to a set direction, and inputting the data sequences into a second convolutional neural network model through a plurality of input channels of the second convolutional neural network model corresponding to the set direction to obtain second model data; dividing the time-frequency spectrogram data into time data sequences corresponding to a plurality of frequencies in a mode of forming the data sequences along the direction of a time axis; and/or dividing the time-frequency spectrogram data into frequency data sequences corresponding to a plurality of moments in a mode of forming the data sequences along the direction of the frequency axis; if the time-frequency spectrogram data are divided into time data sequences corresponding to a plurality of frequencies in a mode of forming a data sequence along a time axis direction, and the time-frequency spectrogram data are divided into frequency data sequences corresponding to a plurality of moments in a mode of forming a data sequence along a frequency axis direction, the second convolution neural network model comprises a time axis convolution neural network model corresponding to the time axis direction and a frequency axis convolution neural network model corresponding to the frequency axis direction, the plurality of time data sequences are input into the time axis convolution neural network model through a plurality of input channels of the time axis convolution neural network model, and second time model data are obtained; inputting the plurality of frequency data sequences into the frequency axis convolutional neural network model through a plurality of input channels of the frequency axis convolutional neural network model to obtain second frequency model data; determining the second time model data and the second frequency model data as second model data;

16. The apparatus of claim 15, wherein the number of input channels of the second convolutional neural network model is the same as the number of data sequences obtained by dividing the time-spectrum data in a set direction.

17. The apparatus of claim 15, wherein the processor is specifically configured to:

18. The apparatus of claim 15, wherein the processor is further specifically configured to:

19. The apparatus of claim 18, wherein the processor is further specifically configured to:

20. The apparatus of claim 15, wherein the device comprises a plurality of sensors,

21. The apparatus of claim 17, wherein if the fusion model is a machine learning model, the fusion model is trained using a first model data sample, a second model data sample, and a corresponding apparatus state sample.

22. A computer storage medium having stored thereon a computer program, which when executed by a processor performs the steps of the method according to any of claims 1-7.