WO2023123930A1

WO2023123930A1 - Image processing method, system, device and readable storage medium

Info

Publication number: WO2023123930A1
Application number: PCT/CN2022/101151
Authority: WO
Inventors: 尹文枫; 董刚; 赵雅倩; 梁玲燕; 晁银银
Original assignee: 浪潮电子信息产业股份有限公司
Priority date: 2021-12-30
Filing date: 2022-06-24
Publication date: 2023-07-06
Also published as: CN114444657A

Abstract

Disclosed is an image processing method. The method comprises: carrying out grouping on a weight matrix of a preset long short-term memory network model on the basis of inherent structured sparsity, obtaining corresponding weight groups; calculating the respective Pearson correlation coefficients of each weight group and the other weight groups, taking the Pearson correlation coefficient to serve as a sampling probability of a weight group being sampled, and on the basis of a preset compression ratio, by means of a sampling probability, randomly selecting a corresponding weight group to carry out compression, obtaining a compressed long short-term memory network model; performing image processing on the input image using the compressed long short-term memory network model.

Description

Image processing method, system, device and readable storage medium

Cross References to Related Applications

This application claims the priority of the Chinese patent application with the application number 202111666557.5 and the application name "An image processing method, system, device, and readable storage medium" submitted to the China Patent Office on December 30, 2021, and its entire content Incorporated in this application by reference.

technical field

The present application relates to the field of image processing, in particular to an image processing method, system, device and readable storage medium.

Background technique

The over-parameterized dividend has greatly improved the accuracy of the neural network, making deep learning widely used in machine vision fields such as multi-target tracking and image segmentation, and has developed a demand for deployment on various embedded devices or mobile platforms, but these The computing and storage resources of the platform have an upper limit, which cannot support the storage and operation of neural networks with huge parameters. Neural network compression technology can effectively reduce the amount of parameters and the amount of calculation during inference, and solve the deployment problem of deep learning in resource-constrained environments. The pruning algorithm for Convolutional Neural Networks (CNN) has been widely used in industry, showing a variety of development trends, such as structured and unstructured pruning algorithms, inference-time pruning or training-based pruning, static pruning or dynamic pruning. Unlike CNN pruning methods, pruning methods for Recurrent Neural Networks (RNNs) have not been fully studied.

The inventor realized that, in view of the difference in calculation logic between RNN and CNN, most CNN pruning methods cannot be directly applied to RNN, which increases the difficulty of researching RNN pruning algorithms.

Contents of the invention

In order to solve the above technical problems, the present application provides an image processing method, the method comprising:

Group the weight matrix of the preset long-short-term memory network model according to the inherent structural sparsity to obtain the corresponding weight group;

Calculate the Pearson correlation coefficient of each weight group and other weight groups separately, use the Pearson correlation coefficient as the sampling probability of the weight group being sampled, and randomly select the corresponding weight group through the sampling probability according to the preset compression rate performing compression to obtain a compressed long-short-term memory network model; and

Image processing is performed on the input image by using the compressed long-short-term memory network model.

In one embodiment, before using the compressed long-short-term memory network model to perform image processing on the input image, it also includes:

Calculate the output feature map of the weight matrix of the preset long short-term memory network model according to the weight group;

Calculate the compressed output feature map of the weight matrix of the compressed long-short-term memory network model according to the weight group;

determining the optimal weight matrix of the compressed long-short-term memory network model according to the output feature map and the compressed output feature map by the least squares method; and

The parameters of the compressed long-short-term memory network model are optimized by using the optimal weight matrix.

In one embodiment, the output feature map of the weight matrix of the preset long-short-term memory network model is calculated according to the weight group, including:

Calculate the input value of the activation function of the forget gate, input gate and output gate of the l-th layer of the preset long-short-term memory network model at time step t, and the output value at the previous time step t-1

And determine the corresponding output weight matrix

and the input weight matrix

and

According to the formula

and

Calculate the output feature map of the weight matrix of the preset long short-term memory network model;

Among them, W ^hl is the output weight matrix of the first layer of the preset long short-term memory network model,

are the output weights of the forget gate, input gate, update gate and output gate of the preset long-short-term memory network model respectively, W ^xl is the input weight matrix of the _l-th layer of the preset long-short-term memory network model,

are the input weights of the forget gate, input gate, update gate and output gate of the preset long-short-term memory network model respectively, and FM ^hl is the output feature map of the output weight matrix of the _l-th layer of the preset long-short-term memory network model, y _fh , y _ih , y _uh , y _oh are the output feature maps of the output weight matrix of the forget gate, input gate, update gate and output gate of the preset long short-term memory network model, respectively,

is the output value of the preset long-short-term memory network model at the last time step t-1, FM ^xl is the output feature map of the input weight matrix of the l-th layer of the preset long-short-term memory network model, y _fx , y _ix , y _ux , y _ox are the output feature maps of the input weight matrix of the forget gate, input gate, update gate and output gate of the preset long short-term memory network model, respectively,

is the input value of layer l of the preset long-short-term memory network model at time step t.

In one embodiment, the compressed output feature map of the weight matrix of the compressed long-short-term memory network model is calculated according to the weight group, including:

Obtain the output value of the l-th layer of the preset long-short-term memory network model at the previous time step t-1

and will output the value

as input to the compressed LSTM network model; and

Will

As the compressed output feature map of the output weight matrix of the compressed long short-term memory network model, and

The compressed output feature map as the input weight matrix of the compressed long short-term memory network model;

in,

is the output weight matrix after W ^hl compression,

for

The transpose matrix of

is the input weight matrix after W ^xl compression,

for

The transpose matrix of

is the output value of layer l of the preset long-short-term memory network model at the last time step t-1, and T is the total number of time steps of the input data x.

In one embodiment, the optimal weight matrix of the compressed long-short-term memory network model is determined according to the output feature map and the compressed output feature map by the least squares method, including:

Record the number of the compressed weight group in the preset long-short-term memory network model as a set θ;

According to the set θ, extract the local data of the output feature map of the output weight matrix of the preset long short-term memory network model at the previous time step t-1

and the output feature map local data of the input weight matrix

According to the formula

Determine the optimal output weight matrix of layer l of the compressed long-short-term memory network model

and

According to the formula

Determine the optimal input weight matrix of layer l of the compressed long short-term memory network model

in,

is the F norm,

is the output feature map local data of the output weight matrix,

is the output feature map local data of the input weight matrix.

In one embodiment, also includes:

According to the set θ, extract the weight of the corresponding row of the input weight matrix W ^x(l+1) of the l+1 layer of the preset long-short-term memory network model

and

According to the formula

Determine the optimal input weight matrix of the l+1 layer of the compressed long short-term memory network model

Among them, FM ^x(l+1) is the output feature map of the input weight matrix of the l+1th layer of the preset long-short-term memory network model, W ^x(l+1) is the first-th layer of the preset long-short-term memory network model The input weight matrix of the l+1 layer,

is the compressed input weight matrix of W ^x(l+1) ,

for

The transpose matrix of

is the input value of layer l+1 of the preset long-short-term memory network model at time step t.

In one embodiment, after optimizing the parameters of the compressed long-short-term memory network model using the optimal weight matrix, it also includes:

In response to parameter optimization of all layers of the compressed long-short-term memory network model, retraining the compressed long-short-term memory network model for a preset number of times.

In order to solve the above technical problems, the present application also provides an image processing system, which includes:

The grouping module is used to group the weight matrix of the preset long short-term memory network model according to the inherent structured sparsity to obtain the corresponding weight group;

The compression module is used to separately calculate the Pearson correlation coefficient of each weight group and other weight groups, and use the Pearson correlation coefficient as the sampling probability of the weight group being sampled, and randomly select the sampling probability according to the preset compression rate The corresponding weight group is compressed to obtain the compressed long-short-term memory network model; and

The image processing module is used for performing image processing on the input image by using the compressed long-short-term memory network model.

In order to solve the above technical problems, the present application also provides an image processing device, which includes:

memory for storing computer readable instructions; and

One or more processors, configured to implement the steps of the image processing method provided in any of the foregoing embodiments when executing the computer-readable instructions.

In order to solve the above-mentioned technical problems, the present application also provides a readable storage medium, on which computer-readable instructions are stored, and when the computer-readable instructions are executed by a processor, the implementation as provided in any of the above-mentioned embodiments can be realized. The steps of the image processing method.

The details of one or more embodiments of the application are set forth in the accompanying drawings and the description below. Other features and advantages of the application will be apparent from the description, drawings, and claims.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only It is an embodiment of the present application, and those skilled in the art can also obtain other drawings according to the provided drawings without creative work.

FIG. 1 is a flowchart of an image processing method provided by one or more embodiments of the present application;

FIG. 2 is a flowchart of another image processing method provided by one or more embodiments of the present application;

FIG. 3 is a structural diagram of an image processing system provided by one or more embodiments of the present application;

Fig. 4 is a structural diagram of an image processing device provided by one or more embodiments of the present application.

Detailed ways

The core of the present application is to provide an image processing method, system, device and readable storage medium for realizing the compression of the cyclic neural network.

In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments It is a part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the scope of protection of this application.

Existing RNN pruning algorithms include amplitude-based weight pruning, or structured pruning based on Least absolute shrinkage and selection operator (LASSO) regression. The latter uses LASSO regression regularization to achieve structured pruning of different granularities, such as block structure weight pruning based on group LASSO regularization, weight pruning based on Intrinsic Structured Sparsity (ISS) or based on Neuron pruning with LASSO regularization. Among them, ISS-based pruning constructs a specific weight grouping. It is assumed that a certain layer of long short-term memory network model (Long Short-Term Memory, LSTM) has K hidden states. When the kth hidden state (hidden state ) is identified as a useless state and needs to be pruned, the kth cell state (cell state) and the kth output gate that generate the kth hidden state can be deleted together. That is, the ISS group consisting of the column associated with the kth unit state in the weight matrix of the current time step, the column corresponding to the kth hidden state of the current time step, and the row corresponding to the kth hidden state of the previous time step Weights can be removed. However, this method does not compress and prune the rows associated with the input in the weight matrix, and cannot prune the neurons of the LSTM. For this defect, neuron pruning based on LASSO regularization introduces two gating variables for the input and hidden states and constructs corresponding LASSO regularization constraints, and learns the value of the gating variable as the mask of the compressed weight matrix during training. Code, and then achieve neuron-level pruning. However, the structured pruning methods based on LASSO regression all belong to pruning during training, rely on the training process to learn the compression mask, and cannot achieve flexible compression according to a given compression rate; therefore, this application provides an image processing method for Solve the above problems.

Please refer to FIG. 1 , which is a flowchart of an image processing method provided by an embodiment of the present application.

It specifically includes the following steps:

S101: Group the weight matrixes of the preset long-short-term memory network model according to the inherent structured sparsity to obtain corresponding weight groups.

In this step, the purpose of grouping the weight matrix is to randomly select the corresponding weight group through the sampling probability according to the preset compression rate for compression, so as to achieve pruning during reasoning of the long-short-term memory network model, and then can not rely on The training directly obtains the new weights of the compressed network weight matrix by minimizing the reconstruction error of the weight matrix output, and realizes the compression of the recurrent neural network.

S102: Calculate the Pearson correlation coefficient of each weight group and other weight groups respectively, use the Pearson correlation coefficient as the sampling probability of the weight group being sampled, and randomly select the corresponding weight group through the sampling probability according to the preset compression rate The weight group is compressed to obtain the compressed long-short-term memory network model.

The Pearson Correlation Coefficient (Pearson Correlation Coefficient) mentioned here is used to measure whether two data sets are on a line, to measure the linear relationship between fixed-distance variables, and to reflect the degree of linear correlation between two variables X and Y , the value of the Pearson correlation coefficient is between -1 and 1, and the larger the absolute value, the stronger the correlation.

Taking the target weight group as an example, calculate the Pearson correlation coefficients between the target weight group and other weight groups, and use the Pearson correlation coefficient as the sampling probability of the target weight group being sampled. According to the preset compression rate, the application randomly selects the corresponding weight group through the sampling probability to compress, and obtains the compressed long-short-term memory network model. For example, the sampled weight group can be set to 0, and the long-short time Compression of memory network models.

S103: Perform image processing on the input image by using the compressed long-short-term memory network model.

On the basis of the above-mentioned embodiments, in a specific embodiment, before using the compressed long-short-term memory network model to perform image processing on the input image, the steps shown in FIG. For parameter optimization of the memory network model, please refer to FIG. 2 below. FIG. 2 is a flowchart of another image processing method provided by the embodiment of the present application.

It specifically includes the following steps:

S201: Calculate the output feature map of the weight matrix of the preset long-short-term memory network model according to the weight group.

In a specific embodiment, the output feature map of the weight matrix of the preset long-short-term memory network model is calculated according to the weight group, which can be specifically implemented by performing the following steps:

And determine the corresponding output weight matrix

and the input weight matrix

According to the formula

and

are the output weights of the forget gate, input gate, update gate and output gate of the preset long-short-term memory network model respectively, W ^xl is the input weight matrix of the l-th layer of the preset long-short-term memory network model,

are the input weights of the forget gate, input gate, update gate and output gate of the preset long-short-term memory network model respectively, and FM ^hl is the output feature map of the output weight matrix of the l-th layer of the preset long-short-term memory network model, y _fh , y _ih , y _uh , y _oh are the output feature maps of the output weight matrix of the forget gate, input gate, update gate and output gate of the preset long short-term memory network model, respectively,

is the output value of the preset long-short-term memory network model at the last time step t-1, FM ^xl is the output feature map of the input weight matrix of the _l-th layer of the preset long-short-term memory network model, y _fx , y _ix , y _ux , y _ox are the output feature maps of the input weight matrix of the forget gate, input gate, update gate and output gate of the preset long short-term memory network model, respectively,

S202: Calculate the compressed output feature map of the weight matrix of the compressed long-short-term memory network model according to the weight group.

In a specific embodiment, the compressed output feature map of the weight matrix of the compressed long-short-term memory network model is calculated according to the weight group, which can be specifically implemented by performing the following steps:

and will output the value

As the input value of the compressed long-short-term memory network model;

Will

in,

is the output weight matrix after W ^hl compression,

for

The transpose matrix of

is the input weight matrix after W ^xl compression,

for

The transpose matrix of

S203: Determine the optimal weight matrix of the compressed long-short-term memory network model according to the output feature map and the compressed output feature map by least square method.

In a specific embodiment, the optimal weight matrix of the compressed long-short-term memory network model is determined according to the output feature map and the compressed output feature map by the least square method, which can be specifically implemented by performing the following steps:

and the output feature map local data of the input weight matrix

According to the formula

in,

is the F norm,

is the output feature map local data of the output weight matrix,

is the output feature map local data of the input weight matrix.

On the basis of the above-mentioned embodiments, the determination of the optimal input weight matrix of the l+1th layer can also be realized by performing the following steps:

According to the formula

is the compressed input weight matrix of W ^x(l+1) ,

for

The transpose matrix of

S204: Using the optimal weight matrix to optimize parameters of the compressed long-short-term memory network model.

In a specific embodiment, in order to further restore the network accuracy of the compressed long-short-term memory network model, after using the optimal weight matrix to optimize the parameters of the compressed long-short-term memory network model, the following steps can also be performed:

This application compresses the neurons of each hidden layer in the preset long-short-term memory network model by compressing the output weights and input weights of the neurons step by step, and the random compression process is performed layer by layer.

Based on the above technical solution, an image processing method provided by this application randomly selects the corresponding weight value group for compression according to the sampling probability determined by the Pearson correlation coefficient according to the preset compression rate, so that the application can compress according to the user-specified Compared with neuron pruning, the new weights of the compressed network weight matrix are obtained by minimizing the reconstruction error output by the weight matrix without relying on training, and then the compression of the recurrent neural network is realized.

Please refer to FIG. 3 , which is a structural diagram of an image processing system provided by an embodiment of the present application.

The system can include:

The grouping module 100 is used to group the weight matrix of the preset long short-term memory network model according to the inherent structured sparsity to obtain the corresponding weight group;

The compression module 200 is used to separately calculate the Pearson correlation coefficient of each weight group and other weight groups, and use the Pearson correlation coefficient as the sampling probability of the weight group being sampled, and randomly pass the sampling probability according to the preset compression rate. Select the corresponding weight group for compression to obtain the compressed long-short-term memory network model;

The image processing module 300 is configured to use the compressed long-short-term memory network model to perform image processing on the input image.

On the basis of the foregoing embodiments, in a specific embodiment, the system may also include:

The first calculation module is used to calculate the output feature map of the weight matrix of the preset long short-term memory network model according to the weight group;

The second calculation module is used to calculate the compressed output feature map of the weight matrix of the compressed long short-term memory network model according to the weight group;

A determination module is used to determine the optimal weight matrix of the compressed long-short-term memory network model according to the output feature map and the compressed output feature map by the least squares method;

The optimization module is used to optimize the parameters of the compressed long-short-term memory network model by using the optimal weight matrix.

On the basis of the foregoing embodiments, in a specific embodiment, the first calculation module may include:

And determine the corresponding output weight matrix

and the input weight matrix

According to the formula

and

On the basis of the foregoing embodiments, in a specific embodiment, the second calculation module may include:

The acquisition sub-module is used to obtain the output value of the l-th layer of the preset long-short-term memory network model at the previous time step t-1

and will output the value

As the input value of the compressed long-short-term memory network model;

The first determined sub-module is used to

in,

is the output weight matrix after W ^hl compression,

for

The transpose matrix of

is the input weight matrix after W ^xl compression,

for

The transpose matrix of

On the basis of the foregoing embodiments, in a specific embodiment, the determining module may include:

The recording sub-module is used to record the number of the compressed weight group in the preset long short-term memory network model as a set θ;

The first extraction sub-module is used to extract the local data of the output feature map of the output weight matrix of the preset long short-term memory network model at the previous time step t-1 according to the set θ

and the output feature map local data of the input weight matrix

The second determination sub-module is used to determine according to the formula

The third determination sub-module is used to determine according to the formula

in,

is the F norm,

is the output feature map local data of the output weight matrix,

is the output feature map local data of the input weight matrix.

On the basis of the above embodiments, in a specific embodiment, the determining module may also include:

The second extraction sub-module is used to extract the weight of the corresponding row of the input weight matrix W ^{x (l+1)} of the l+1 layer of the preset long-short-term memory network model according to the set θ

The fourth determination sub-module is used to determine according to the formula

is the compressed input weight matrix of W ^x(l+1) ,

for

The transpose matrix of

The retraining module is configured to perform a preset number of retrainings on the compressed long short-term memory network model in response to parameter optimization of all layers of the compressed long short-term memory network model.

Since the embodiments of the system part correspond to the embodiments of the method part, please refer to the description of the embodiments of the method part for the embodiments of the system part, and details will not be repeated here.

Please refer to FIG. 4 , which is a structural diagram of an image processing device provided by an embodiment of the present application.

The image processing device 400 may have relatively large differences due to different configurations or performances, and may include one or more than one processor (central processing units, CPU) 422 and memory 432, and one or more than one storage application program 442 or data 444 Storage medium 430 (such as one or more mass storage devices). Wherein, the memory 432 and the storage medium 430 may be temporary storage or persistent storage. The program stored in the storage medium 430 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the device. Furthermore, the processor 422 may be configured to communicate with the storage medium 430 , and execute a series of instruction operations in the storage medium 430 on the image processing device 400 .

The image processing device 400 may also include one or more power sources 424, one or more wired or wireless network interfaces 450, one or more input and output interfaces 458, and/or, one or more operating systems 441, such as Windows Server™, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

The steps in the image processing method described above in FIG. 1 to FIG. 2 are realized by the image processing device based on the structure shown in FIG. 4 .

Further, the embodiment of the present application also discloses a non-volatile computer-readable storage medium, in which computer-readable instructions are stored, and when the computer-readable instructions are loaded and executed by one or more processors, The steps in the image processing method disclosed in any of the foregoing embodiments are implemented.

Those skilled in the art can clearly understand that for the convenience and brevity of the description, the specific working process of the above-described system, device and module can refer to the corresponding process in the foregoing method embodiment, which will not be repeated here.

In the several embodiments provided in this application, it should be understood that the disclosed devices, devices and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of modules is only a logical function division. In actual implementation, there may be other division methods. For example, multiple modules or components can be combined or integrated. to another system, or some features may be ignored, or not implemented. In another point, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or modules may be in electrical, mechanical or other forms.

A module described as a separate component may or may not be physically separated, and a component shown as a module may or may not be a physical module, that is, it may be located in one place, or may also be distributed to multiple network modules. Part or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional module in each embodiment of the present application may be integrated into one processing module, each module may exist separately physically, or two or more modules may be integrated into one module. The above-mentioned integrated modules can be implemented in the form of hardware or in the form of software function modules.

If the integrated modules are realized in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or part of the contribution to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium , including several instructions to make a computer device (which may be a personal computer, a function calling device, or a network device, etc.) execute all or part of the steps of the methods in various embodiments of the present application. The aforementioned storage media include: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), magnetic disk or optical disc and other media that can store program codes. .

An image processing method, system, device, and readable storage medium provided by the present application have been introduced in detail above. In this paper, specific examples are used to illustrate the principles and implementation methods of the present application, and the descriptions of the above embodiments are only used to help understand the methods and core ideas of the present application. It should be pointed out that for those skilled in the art, without departing from the principle of the application, some improvements and modifications can be made to the application, and these improvements and modifications also fall within the protection scope of the claims of the application.

It should also be noted that in this specification, relative terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that these entities or operations There is no such actual relationship or order between the operations. Furthermore, the term "comprises", "comprises" or any other variation thereof is intended to cover a non-exclusive inclusion such that a process, method, article, or apparatus comprising a set of elements includes not only those elements, but also includes elements not expressly listed. other elements of or also include elements inherent in such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising a ..." does not preclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.

Claims

An image processing method, characterized in that, comprising:

grouping the weight matrix of the preset LSTM network model according to the inherent structured sparsity to obtain corresponding weight groups;

Calculate the Pearson correlation coefficient of each weight group and other weight groups separately, use the Pearson correlation coefficient as the sampling probability of the weight group being sampled, and randomly pass the sampling probability according to the preset compression rate Selecting the corresponding weight value group for compression to obtain the compressed long-short-term memory network model; and

Image processing is performed on the input image by using the compressed long-short-term memory network model.
The method according to claim 1, wherein, before utilizing the compressed long-short-term memory network model to perform image processing on the input image, it also includes:

Calculating the output feature map of the weight matrix of the preset long-short-term memory network model according to the weight group;

Calculate the compressed output feature map of the weight matrix of the compressed long-short-term memory network model according to the weight group;

determining the optimal weight matrix of the compressed long-short-term memory network model according to the output feature map and the compressed output feature map by a least squares method; and

Using the optimal weight matrix to optimize the parameters of the compressed long-short-term memory network model.
The method according to claim 2, wherein the calculation of the output feature map of the weight matrix of the preset long-short-term memory network model according to the weight group includes:

Calculate the input value of the activation function of the forgetting gate, input gate and output gate of the first layer of the preset long-short-term memory network model at time step t and the output value at the previous time step t-1
And determine the corresponding output weight matrix
and the input weight matrix
and

According to the formula
and
Calculate the output feature map of the weight matrix of the preset long short-term memory network model;

Wherein, W hl is the output weight matrix of the first layer of the preset long-short-term memory network model,
Respectively, the output weights of the forget gate, input gate, update gate and output gate of the preset long-short-term memory network model, W xl is the input weight matrix of the first layer of the preset long-short-term memory network model,
are respectively the input weights of the forget gate, input gate, update gate and output gate of the preset long-short-term memory network model, and FM hl is the output weight matrix of the first layer of the preset long-short-term memory network model Output feature map, y fh , y ih , y uh , y oh are the output feature maps of the output weight matrix of the forget gate, input gate, update gate and output gate of the preset long short-term memory network model, respectively,
is the output value of the preset long-short-term memory network model at the last time step t-1, FM xl is the output feature map of the input weight matrix of the first layer of the preset long-short-term memory network model, y fx , y ix , y ux , y ox are the output feature maps of the input weight matrix of the forget gate, input gate, update gate and output gate of the preset long-short-term memory network model, respectively,
is the input value of the first layer of the preset long-short-term memory network model at time step t.
The method according to claim 3, wherein the compressed output feature map of the weight matrix of the compressed long-short-term memory network model is calculated according to the weight group, comprising:

Obtain the output value of the l-th layer of the preset long-short-term memory network model at the previous time step t-1
and put the output value
As the input value of the compressed long-short-term memory network model; and

Will
As the compressed output feature map of the output weight matrix of the compressed long-short-term memory network model, and
As the compressed output feature map of the input weight matrix of the compressed long-short-term memory network model;

in,
is the output weight matrix after W hl compression,
for
The transpose matrix of
is the input weight matrix after W xl compression,
for
The transpose matrix of
is the output value of the first layer of the preset long-short-term memory network model at the last time step t-1, and T is the total number of time steps of the input data x.
The method according to any one of claims 2-4, wherein the optimal weight of the compressed long-short-term memory network model is determined according to the output feature map and the compressed output feature map by the least square method A matrix of values, including:

Recording the number of the compressed weight group in the preset long-short-term memory network model as a set θ;

Extract the local data of the output feature map of the output weight matrix of the preset long-short-term memory network model at the previous time step t-1 according to the set θ
and the output feature map local data of the input weight matrix

According to the formula
Determine the optimal output weight matrix of the first layer of the compressed long short-term memory network model
and

According to the formula
Determine the optimal input weight matrix of the first layer of the compressed long short-term memory network model

in,
is the F norm,
is the output feature map local data of the output weight matrix,
is the output feature map local data of the input weight matrix.
The method according to claim 5, further comprising:

Extract the weight of the input weight matrix W x (l+1) corresponding row of the l+1 layer of the preset long-short-term memory network model according to the set θ
and

According to the formula
Determine the optimal input weight matrix of the l+1 layer of the compressed long short-term memory network model

Wherein, FM x (l+1) is the output feature map of the input weight matrix of the l+1 layer of the preset long-short-term memory network model, and W x (l+1) is the preset long-short-term memory The input weight matrix of the l+1 layer of the network model,
is the compressed input weight matrix of W x(l+1) ,
for
The transpose matrix of
is the input value of the l+1th layer of the preset long-short-term memory network model at time step t.
The method according to any one of claims 2-6, wherein, after optimizing the parameters of the compressed long-short-term memory network model using the optimal weight matrix, further comprising:

Responding to the completion of parameter optimization of all layers of the compressed long-short-term memory network model, retraining the compressed long-short-term memory network model for a preset number of times.
An image processing system, characterized in that it comprises:

The grouping module is used to group the weight matrix of the preset long-short-term memory network model according to the inherent structured sparsity to obtain the corresponding weight group;

The compression module is used to separately calculate the Pearson correlation coefficient of each weight group and other weight groups, and use the Pearson correlation coefficient as the sampling probability of the weight group being sampled, and pass the The sampling probability randomly selects the corresponding weight group to compress, and obtains the compressed long-short-term memory network model; and

The image processing module is configured to use the compressed long-short-term memory network model to perform image processing on the input image.
An image processing device, characterized in that it comprises:

memory for storing computer readable instructions; and

One or more processors, configured to implement the steps of the image processing method according to any one of claims 1 to 7 when executing the computer-readable instructions.
A non-volatile computer-readable storage medium, wherein computer-readable instructions are stored on the non-volatile computer-readable storage medium, and when the computer-readable instructions are executed by one or more processors Implementing the steps of the image processing method described in any one of claims 1 to 7.