CN113362402B - Image processing method and device based on decomposition matrix - Google Patents
Image processing method and device based on decomposition matrix Download PDFInfo
- Publication number
- CN113362402B CN113362402B CN202110544946.4A CN202110544946A CN113362402B CN 113362402 B CN113362402 B CN 113362402B CN 202110544946 A CN202110544946 A CN 202110544946A CN 113362402 B CN113362402 B CN 113362402B
- Authority
- CN
- China
- Prior art keywords
- matrix
- image processing
- decomposition
- matrixes
- initial
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 239000011159 matrix material Substances 0.000 title claims abstract description 531
- 238000000354 decomposition reaction Methods 0.000 title claims abstract description 107
- 238000003672 processing method Methods 0.000 title claims abstract description 33
- 238000004364 calculation method Methods 0.000 claims abstract description 163
- 238000012545 processing Methods 0.000 claims abstract description 148
- 238000012549 training Methods 0.000 claims abstract description 13
- 230000015572 biosynthetic process Effects 0.000 claims description 50
- 238000003786 synthesis reaction Methods 0.000 claims description 50
- 238000000605 extraction Methods 0.000 claims description 11
- 238000004590 computer program Methods 0.000 claims description 9
- 238000000034 method Methods 0.000 abstract description 13
- 238000004891 communication Methods 0.000 description 5
- 238000010586 diagram Methods 0.000 description 3
- 230000002194 synthesizing effect Effects 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 2
- 230000002035 prolonged effect Effects 0.000 description 2
- 238000007796 conventional method Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T9/00—Image coding
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- General Engineering & Computer Science (AREA)
- Image Processing (AREA)
Abstract
The invention provides an image processing method and device based on a decomposition matrix, wherein the method comprises the following steps: inputting an image to be processed into an image processing model to obtain an image processing result output by the image processing model; the image processing model is obtained by training based on the sample image and the corresponding sample image processing result; the image processing model is a self-attention depth model with a hierarchical structure, the equivalent calculation matrix of each hierarchy in the image processing model is obtained by decomposing each initial calculation matrix based on a decomposition matrix, and the decomposition matrix is obtained based on a preset number of column matrixes in each hierarchy of initial calculation matrix. According to the method, based on the column matrixes with the preset number in the initial calculation matrixes of each level, the initial calculation matrixes are decomposed to obtain the equivalent calculation matrixes of each level, so that the dimension of the equivalent calculation matrixes is lower than that of the initial calculation matrixes, the matrix calculation amount of the model is further reduced, and the image processing efficiency is improved.
Description
Technical Field
The present invention relates to the field of image processing technologies, and in particular, to an image processing method and apparatus based on a decomposition matrix.
Background
Self-attention depth models such as the transfomer have found widespread application in the task of image processing. The self-attention depth model comprises multiple self-attention layers, each layer has a respective calculation matrix, and the self-attention values of the layers can be obtained based on the calculation matrices of the layers.
However, the calculation matrix of each layer has a large dimension, thereby increasing the amount of calculation of the model and further reducing the image processing efficiency.
Disclosure of Invention
The invention provides an image processing method and device based on a decomposition matrix, which are used for overcoming the defect of low image processing efficiency in the prior art.
The invention provides an image processing method based on a decomposition matrix, which comprises the following steps:
determining an image to be processed;
inputting the image to be processed into an image processing model to obtain an image processing result output by the image processing model;
the image processing model is obtained by training based on sample images and sample image processing results corresponding to the sample images; the image processing model is a self-attention depth model with a hierarchical structure, the equivalent calculation matrix of each level in the image processing model is obtained by decomposing each initial calculation matrix based on a decomposition matrix, the decomposition matrix is obtained based on a preset number of column matrixes in each level of initial calculation matrix, and the preset number is greater than the effective rank of the initial calculation matrix and less than the number of rows of the initial calculation matrix.
According to the image processing method based on the decomposition matrix, provided by the invention, a preset number of column matrixes are randomly extracted from the initial calculation matrix to form a column synthesis matrix;
randomly extracting the row matrixes with the preset number from the column synthesis matrix to form an initial decomposition matrix;
and if the rank of the initial decomposition matrix is greater than the effective rank of the initial calculation matrix, taking the initial decomposition matrix as the decomposition matrix.
According to the image processing method based on the decomposition matrix provided by the invention, the equivalent calculation matrix is determined based on the following steps:
performing singular value decomposition on the decomposition matrix to obtain a plurality of sub-matrixes, and determining characteristic values corresponding to the sub-matrixes;
selecting a plurality of sub-matrixes with the maximum eigenvalue larger than a preset value to form an eigenvalue matrix, and determining a left singular vector and a right singular vector corresponding to the eigenvalue matrix;
randomly extracting a preset number of row matrixes from the initial calculation matrix to form a row synthesis matrix;
determining the equivalent computation matrix based on the column synthesis matrix, the row synthesis matrix, the eigenvalue matrix, the left singular vector, and the right singular vector.
According to an image processing method based on a decomposition matrix provided by the present invention, the determining the equivalent computation matrix based on the column synthesis matrix, the row synthesis matrix, the eigenvalue matrix, the left singular vector, and the right singular vector includes:
matrix multiplication is carried out on the column synthesis matrix, the right singular vector and an inverse matrix of the eigenvalue matrix to obtain a first matrix;
performing matrix multiplication on the transposed matrix of the left singular vector and the row synthesis matrix to obtain a second matrix;
and taking a multiplication matrix of the first matrix and the second matrix as the equivalent calculation matrix.
According to the image processing method based on the decomposition matrix provided by the invention, the image to be processed is input to an image processing model, and an image processing result output by the image processing model is obtained, and the image processing method comprises the following steps:
inputting the image to be processed to a pixel extraction layer of the image processing model to obtain a pixel matrix output by the pixel extraction layer;
inputting the pixel matrix to a self-attention layer of the image processing model, and carrying out matrix multiplication on the pixel matrix and an equivalent calculation matrix by the self-attention layer to obtain a self-attention value output by the self-attention layer;
and inputting the self-attention value to an image processing layer of the image processing model to obtain an image processing result output by the image processing layer.
According to the image processing method based on the decomposition matrix provided by the invention, the matrix multiplication is carried out on the pixel matrix and the equivalent calculation matrix to obtain the self-attention value output by the self-attention layer, and the image processing method comprises the following steps:
matrix multiplication is carried out on the pixel matrix and the first matrix to obtain an intermediate matrix;
and carrying out matrix multiplication on the intermediate matrix and the second matrix to obtain a self-attention value output by the self-attention layer.
According to the image processing method based on the decomposition matrix, the initial calculation matrix comprises at least one of a query matrix, a key value matrix and a value matrix.
The present invention also provides an image processing apparatus based on a decomposition matrix, comprising:
an image determining unit for determining an image to be processed;
the image processing unit is used for inputting the image to be processed into an image processing model to obtain an image processing result output by the image processing model;
the image processing model is obtained by training based on sample images and sample image processing results corresponding to the sample images; the image processing model is a self-attention depth model with a hierarchical structure, the equivalent calculation matrix of each level in the image processing model is obtained by decomposing each initial calculation matrix based on a decomposition matrix, the decomposition matrix is obtained based on a preset number of column matrixes in each level of initial calculation matrix, and the preset number is greater than the effective rank of the initial calculation matrix and less than the number of rows of the initial calculation matrix.
The invention further provides an electronic device, which comprises a memory, a processor and a computer program stored on the memory and capable of running on the processor, wherein the processor executes the computer program to realize the steps of the image processing method based on the decomposition matrix.
The invention also provides a non-transitory computer-readable storage medium having stored thereon a computer program which, when being executed by a processor, carries out the steps of the decomposition matrix-based image processing method according to any one of the above.
According to the image processing method and device based on the decomposition matrix, the initial calculation matrixes are decomposed to obtain the equivalent calculation matrix of each level based on the preset number of column matrixes in the initial calculation matrixes of each level, so that the dimension of the equivalent calculation matrix is lower than that of the initial calculation matrix, the matrix operation amount of a model is further reduced, and the image processing efficiency is improved.
Drawings
In order to more clearly illustrate the technical solutions of the present invention or the prior art, the drawings needed for the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and those skilled in the art can also obtain other drawings according to the drawings without creative efforts.
FIG. 1 is a schematic flow chart of an image processing method based on a decomposition matrix according to the present invention;
FIG. 2 is a schematic structural diagram of an image processing apparatus based on a decomposition matrix according to the present invention;
fig. 3 is a schematic structural diagram of an electronic device provided in the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings, and it is obvious that the described embodiments are some, but not all embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The self-attention depth model for image processing includes multiple levels of self-attention layers, each of which may include a query matrix (W) Q ) Key-value matrix (W) K ) And a value matrix (W) V ). For the pixel matrix X of the image input to be processed, the corresponding self-attention value can be calculated by the following formula:
Y＝softmax(X×W Q ×(X×W k ) T )×(X×W v )；
however, the query matrix (W) Q ) Key-value matrix (W) K ) And a value matrix (W) V ) The matrix is usually a high-dimensional matrix (for example, a 1024 × 128-dimensional matrix), so that the matrix operation amount is increased, the time for image processing by the model is prolonged, and the efficiency of image processing is affected.
In view of the above, the present invention provides an image processing method based on a decomposition matrix. Fig. 1 is a schematic flow chart of an image processing method based on a decomposition matrix provided by the present invention, as shown in fig. 1, the method includes the following steps:
the image processing model is obtained by training based on the sample image and the corresponding sample image processing result; the image processing model is a self-attention depth model with a hierarchical structure, the equivalent calculation matrix of each level in the image processing model is obtained by decomposing each initial calculation matrix based on a decomposition matrix, the decomposition matrix is obtained based on a preset number of column matrixes in each level of initial calculation matrix, and the preset number is larger than the effective rank of the initial calculation matrix and smaller than the number of rows of the initial calculation matrix.
Specifically, the image to be processed refers to an image to be processed in accordance with the image processing task. The image may be an image to be classified or an image to be identified, which is not particularly limited in this embodiment of the present invention.
The image processing model comprises a plurality of self-attention layers, after model training is completed, an initial calculation matrix (such as a query matrix, a key value matrix, a value matrix and the like) corresponding to each layer generally has a high dimensionality, so that a decomposition matrix is obtained based on a preset number of column matrixes in the initial calculation matrix, and because the preset number is smaller than the number of rows of the initial calculation matrix, when the initial calculation matrix is decomposed based on the preset number of column matrixes, the dimensionality of an equivalent calculation matrix obtained by decomposition is lower than that of the initial calculation matrix, and the matrix operation amount is reduced. Meanwhile, the number of the column matrixes is greater than the effective rank of the initial calculation matrix, so that the effective information of the initial calculation matrix is reserved in the equivalent calculation matrix obtained through decomposition, namely the equivalent calculation matrix can be equivalent to the initial calculation matrix.
Therefore, after the image to be processed is input into the image processing model, the operation amount of the matrix in the model can be reduced, the image processing efficiency is improved, and the effective information of the initial calculation matrix is kept by the equivalent calculation matrix, so that the image can be accurately processed.
In the conventional method, after an image to be processed is input into an image processing model, a pixel matrix corresponding to the image to be processed is multiplied by a calculation matrix (such as a query matrix, a key value matrix, a value matrix and the like) of each self-attention layer to obtain a self-attention value of each layer, and then image processing is performed based on the self-attention value. However, the calculation matrix of each self-attention layer generally has a large dimension, so that the calculation amount of each layer of matrix is large, the time for image processing of the model is prolonged, and the efficiency of the image processing is affected. Meanwhile, the higher the dimensionality of the calculation matrix is, the larger the size of the space of the model required to be stored is, and the higher the corresponding hardware architecture cost required to execute the operation is. The method provided by the embodiment of the invention can not only improve the image processing efficiency, but also reduce the model storage size and save the cost of a hardware architecture.
It should be noted that before the image to be processed is input to the image processing model, the image processing model may be obtained by training in advance, and specifically, the following steps may be performed: firstly, a large number of sample images are collected, and corresponding sample image processing results are determined through manual marking. And then, training the initial model based on the sample image and the corresponding sample image processing result, thereby obtaining the initial image processing model. After the initial image processing model is trained, decomposing each initial calculation matrix based on the preset number of column matrixes in each level of initial calculation matrix to obtain the equivalent calculation matrix of the corresponding level, and replacing the corresponding initial calculation matrix with the equivalent calculation matrix with lower dimensionality of each level, so that the matrix operation amount can be reduced, and the image processing efficiency is improved.
According to the image processing method based on the random matrix, provided by the embodiment of the invention, each initial calculation matrix is decomposed to obtain the equivalent calculation matrix of each level based on the preset number of column matrixes in each level of initial calculation matrix, so that the dimension of the equivalent calculation matrix is lower than that of the initial calculation matrix, the matrix operation amount of a model is further reduced, and the image processing efficiency is improved.
Based on the above embodiment, the decomposition matrix is determined based on the following steps:
randomly extracting a preset number of column matrixes from the initial calculation matrix to form a column synthesis matrix;
randomly extracting a preset number of row matrixes from the column synthesis matrix to form an initial decomposition matrix;
and if the rank of the initial decomposition matrix is greater than the effective rank of the initial calculation matrix, taking the initial decomposition matrix as a decomposition matrix.
Specifically, a preset number of column matrixes are randomly extracted from the initial calculation matrix, the column matrixes form a column synthesis matrix, and the preset number is smaller than the number of rows of the initial calculation matrix and larger than the effective rank of the initial calculation matrix, so that the dimension of the column synthesis matrix is not only lower than that of the initial calculation matrix, but also the column synthesis matrix retains effective information of the initial calculation matrix.
After determining the column synthesis matrix, randomly extracting a preset number of row matrixes from the column synthesis matrix to form an initial decomposition matrix, wherein the row number and the column number of the initial decomposition matrix are equal to values corresponding to the preset number. If the rank of the initial decomposition matrix is greater than the effective rank of the initial calculation matrix, indicating that the initial decomposition matrix retains the effective information of the initial calculation matrix, taking the corresponding initial decomposition matrix as a decomposition matrix; if not, repeating the steps to extract the column matrix and the row matrix again according to the preset number or resetting the preset number and extracting the corresponding column matrix and row matrix until the rank of the initial decomposition matrix is greater than the effective rank of the initial calculation matrix.
The dimension of the decomposition matrix obtained based on the method is lower than that of the initial calculation matrix, and the rank of the decomposition matrix is greater than the effective rank of the initial calculation matrix, so that effective information of the initial calculation matrix is reserved, the equivalent calculation matrix obtained based on the decomposition matrix is equivalent to the initial calculation matrix, and the dimension of the equivalent calculation matrix is lower than that of the initial calculation matrix, so that the operation amount of the matrix can be reduced.
For example, for an initial computation matrix such as the query matrix W Q Is m × n (e.g., 1024 × 128), and the corresponding effective rank is r (e.g., 16), then W may be selected from Q In the random selection of k columns (m)>k>r) forming a column synthesis matrix S c Synthesizing the matrix S from the columns c Selecting k rows randomly to obtain an initial decomposition matrix S 2 In which S is 2 Is k × k. If S 2 Rank of>W Q Is valid, then S is 2 If not, repeating the above steps to reselect the corresponding S c And S 2 Up to S 2 Rank of>W Q Is determined.
Based on any of the above embodiments, the equivalent calculation matrix is determined based on the following steps:
performing singular value decomposition on the decomposition matrix to obtain a plurality of sub-matrixes, and determining characteristic values corresponding to the sub-matrixes;
selecting a plurality of sub-matrixes with the maximum eigenvalue larger than a preset value to form an eigenvalue matrix, and determining a left singular vector and a right singular vector corresponding to the eigenvalue matrix;
randomly extracting a preset number of row matrixes from the initial calculation matrix to form a row synthesis matrix;
and determining an equivalent calculation matrix based on the column synthesis matrix, the row synthesis matrix, the eigenvalue matrix, the left singular vector and the right singular vector.
Specifically, after the decomposition matrix is determined, singular Value Decomposition (SVD) is performed on the decomposition matrix, so that a plurality of sub-matrices can be obtained, and eigenvalues corresponding to the sub-matrices are determined. And selecting a plurality of sub-matrixes with the maximum eigenvalue larger than a preset value to form an eigenvalue matrix, and determining a left singular vector and a right singular vector corresponding to the eigenvalue matrix. R submatrices with the largest eigenvalue and larger than a preset value can be selected as the eigenvalue matrix, and r is the effective rank of the initial calculation matrix.
And simultaneously, randomly extracting a preset number of row matrixes from the initial calculation matrix to form a row synthesis matrix, and determining the equivalent calculation matrix based on the column synthesis matrix, the row synthesis matrix, the eigenvalue matrix, the left singular vector and the right singular vector.
For example, for an initial computation matrix such as the query matrix W Q Has a dimension of m × n (e.g., 1024 × 128), and has an effective rank of r (e.g., 16), from W Q In the random selection of k rows (m)>k>r) forming a column synthesis matrix S c Synthesizing the matrix S from the columns c Selecting k rows randomly to obtain an initial decomposition matrix S 2 In which S is 2 Is k × k. If S 2 Rank of>W Q Is valid, then S is 2 As a decomposition matrix, and for S 2 Singular value decomposition is carried out, r sub-matrixes with the maximum eigenvalue are selected to form an eigenvalue matrix E, and a left singular matrix corresponding to the matrix E is determinedAn eigenvector U and a right singular vector V. At the same time, from the matrix W Q Randomly selecting k rows to form a row synthesis matrix S R Based on the matrix S R The matrix S c And determining an equivalent calculation matrix by using the matrix E, the vector U and the vector V.
Based on any of the above embodiments, determining an equivalent computation matrix based on the column synthesis matrix, the row synthesis matrix, the eigenvalue matrix, the left singular vector, and the right singular vector includes:
matrix multiplication is carried out on the column synthesis matrix, the right singular vector and the inverse matrix of the eigenvalue matrix to obtain a first matrix;
matrix multiplication is carried out on the transposed matrix of the left singular vector and the row synthesis matrix to obtain a second matrix;
and taking a multiplication matrix of the first matrix and the second matrix as an equivalent calculation matrix.
Specifically, after the eigenvalue matrix E is obtained, the inverse matrix B corresponding to the matrix E may be determined -1 Thereby synthesizing a matrix S for the columns c Right singular vector V and inverse matrix B -1 Matrix multiplication is carried out to obtain a first matrix C, namely C = S c ×V×B -1 And the dimension of C is m × r (m represents the number of rows of the initial calculation matrix, and r represents the effective rank of the initial calculation matrix).
Transposed matrix U to left singular vector U T And a row synthesis matrix S R Matrix multiplication is carried out to obtain a second matrix R, namely R = U T ×S R R has dimension R × n (n represents the number of columns of the initial calculation matrix, and R represents the effective rank of the initial calculation matrix), so that the equivalent calculation matrix W Q * Is C × R, is related to the initial calculation matrix W Q Equivalent matrix, and equivalent calculation matrix W Q * Is much smaller than the initial calculation matrix W Q Therefore, the matrix operation amount of the model can be effectively reduced, and the image processing efficiency is improved.
Based on any of the above embodiments, inputting an image to be processed into an image processing model to obtain an image processing result output by the image processing model, including:
inputting an image to be processed to a pixel extraction layer of an image processing model to obtain a pixel matrix output by the pixel extraction layer;
inputting the pixel matrix into a self-attention layer of the image processing model, and performing matrix multiplication on the pixel matrix and the equivalent calculation matrix by the self-attention layer to obtain a self-attention value output by the self-attention layer;
and inputting the self-attention value into an image processing layer of the image processing model to obtain an image processing result output by the image processing layer.
Specifically, the equivalent calculation matrix W is determined based on the above-described embodiment Q * After = C × R, the initial calculation matrix (e.g. query matrix W) in the image processing model is calculated Q ) And replacing the two matrixes with a first matrix C and a second matrix R, and then inputting the image to be processed into a pixel extraction layer of the image processing model to obtain a pixel matrix X output by the pixel extraction layer. Suppose W Q Has a dimension m n of 1024 x 128 Q Is =16, then the dimension of the matrix C is m × R =1024 × 16, and the dimension of the matrix R is R × n =16 × 128, i.e. the sum of the dimensions of the matrix C and the matrix R is much smaller than the initial calculation matrix W Q For a pixel matrix X (e.g., k × m =512 × 1024), the process input to the self-attention layer to calculate the self-attention value is as follows:
calculating X multiplied by C, and multiplying with R to obtain X Q The calculation amount is reduced from k × m × n (512 × 01024 × 128) to k × m × r + k × r × n (512 × (1024 × 16+128 × 16)). By analogy, if the initial calculation matrix further comprises a key value matrix W k And the value matrix W v Then the same method is adopted to obtain the key value matrix W k Is calculated by the equivalent calculation matrix W k * And a value matrix W v Is calculated by the equivalent calculation matrix W v * Then the pixel matrix X and the matrix W are combined k * Multiplying to obtain a key value X k And combining the pixel matrix X with the matrix W v * Multiplying to obtain a value X v Finally, the self-attention value a = softmax (X) of the layer is calculated Q ×X k T )×X v And performing image processing based on the self-attention value.
Based on any of the above embodiments, matrix-multiplying the pixel matrix and the equivalent computation matrix to obtain the self-attention value output from the attention layer includes:
matrix multiplication is carried out on the pixel matrix and the first matrix to obtain an intermediate matrix;
and performing matrix multiplication on the intermediate matrix and the second matrix to obtain a self-attention value output from the attention layer.
Specifically, after the pixel matrix is input into the self-attention layer, the pixel matrix is firstly subjected to matrix multiplication with the first matrix to obtain an intermediate matrix, and then the intermediate matrix is subjected to matrix multiplication with the second matrix to obtain the self-attention value.
Therefore, in the embodiment of the invention, the initial calculation matrix in the image processing model is replaced by the first matrix and the second matrix, and the sum of the sizes of the first matrix and the second matrix is far smaller than that of the initial calculation matrix, so that the matrix operation amount and the storage size of the model can be greatly reduced, and the image processing efficiency is improved.
Based on any of the above embodiments, the initial computation matrix includes at least one of a query matrix, a key-value matrix, and a cost matrix.
Specifically, the initial computation matrix comprises a query matrix W Q Key-value matrix W k And the value matrix W v Obtaining the query matrix W based on the method of the above embodiment Q Equivalent matrix W of Q * The key value matrix W k Is calculated by the equivalent calculation matrix W k * And a value matrix W v Is calculated by the equivalent calculation matrix W v * Then the pixel matrix X and the matrix W are combined Q * Multiplying to obtain a query value X Q A pixel matrix X and a matrix W k * Multiplying to obtain a key value X k And the pixel matrix X and the matrix W v * Multiplying to obtain a value X v Finally, the self-attention value a = softmax (X) of the layer is calculated Q ×X k T )×X v And image processing is performed based on the self-attention value, so that the operation amount of the matrix can be reduced, and the image processing efficiency is improved.
The present invention provides a decomposition matrix-based image processing apparatus, and the decomposition matrix-based image processing apparatus described below and the decomposition matrix-based image processing method described above may be referred to in correspondence with each other.
Based on any of the embodiments, the present invention provides an image processing apparatus based on a decomposition matrix, as shown in fig. 2, the apparatus comprising:
an image determining unit 210 for determining an image to be processed;
the image processing unit 220 is configured to input an image to be processed to the image processing model, and obtain an image processing result output by the image processing model;
the image processing model is obtained by training based on the sample image and the corresponding sample image processing result; the image processing model is a self-attention depth model with a hierarchical structure, the equivalent calculation matrix of each level in the image processing model is obtained by decomposing each initial calculation matrix based on a decomposition matrix, the decomposition matrix is obtained based on a preset number of column matrixes in each level initial calculation matrix, and the preset number is greater than the effective rank of the initial calculation matrix and less than the number of rows of the initial calculation matrix.
Based on any embodiment above, still include:
a first forming unit, configured to randomly extract a preset number of column matrices from the initial calculation matrix to form a column synthesis matrix;
a second forming unit, configured to randomly extract the preset number of row matrices from the column synthesis matrix to form an initial decomposition matrix;
a decomposition matrix determining unit, configured to take the initial decomposition matrix as the decomposition matrix if the rank of the initial decomposition matrix is greater than the effective rank of the initial calculation matrix.
Based on any embodiment above, still include:
the decomposition unit is used for carrying out singular value decomposition on the decomposition matrix to obtain a plurality of sub-matrixes and determining characteristic values corresponding to the sub-matrixes;
the eigenvalue determining unit is used for selecting a plurality of sub-matrixes with the maximum eigenvalue and larger than a preset value to form an eigenvalue matrix and determining a left singular vector and a right singular vector corresponding to the eigenvalue matrix;
a third forming unit, configured to randomly extract a preset number of row matrices from the initial calculation matrix to form a row synthesis matrix;
an equivalent calculation matrix determination unit configured to determine the equivalent calculation matrix based on the column synthesis matrix, the row synthesis matrix, the eigenvalue matrix, the left singular vector, and the right singular vector.
Based on any of the above embodiments, the equivalent calculation matrix determining unit includes:
a first matrix determining unit, configured to perform matrix multiplication on the column synthesis matrix, the right singular vector, and an inverse matrix of the eigenvalue matrix to obtain a first matrix;
a second matrix determining unit, configured to perform matrix multiplication on the transposed matrix of the left singular vector and the row synthesis matrix to obtain a second matrix;
and the matrix multiplication unit is used for taking a multiplication matrix of the first matrix and the second matrix as the equivalent calculation matrix.
According to any of the above embodiments, the image processing unit 220 includes:
the pixel extraction unit is used for inputting the image to be processed to a pixel extraction layer of the image processing model to obtain a pixel matrix output by the pixel extraction layer;
the self-attention unit is used for inputting the pixel matrix to a self-attention layer of the image processing model, and the self-attention layer performs matrix multiplication on the pixel matrix and an equivalent calculation matrix to obtain a self-attention value output by the self-attention layer;
and the processing subunit is used for inputting the self-attention value into an image processing layer of the image processing model to obtain an image processing result output by the image processing layer.
Based on any embodiment above, the self-attention unit includes:
the first multiplying unit is used for carrying out matrix multiplication on the pixel matrix and the first matrix to obtain an intermediate matrix;
and the second multiplying unit is used for carrying out matrix multiplication on the intermediate matrix and the second matrix to obtain a self-attention value output by the self-attention layer.
Based on any of the above embodiments, the initial calculation matrix includes at least one of a query matrix, a key-value matrix, and a cost matrix.
Fig. 3 is a schematic structural diagram of an electronic device provided in the present invention, and as shown in fig. 3, the electronic device may include: a processor (processor) 310, a memory (memory) 320, a communication Interface (Communications Interface) 330 and a communication bus 340, wherein the processor 310, the memory 320 and the communication Interface 330 communicate with each other via the communication bus 340. The processor 310 may invoke logic instructions in the memory 320 to perform a decomposition matrix based image processing method comprising: determining an image to be processed; inputting the image to be processed into an image processing model to obtain an image processing result output by the image processing model; the image processing model is obtained by training based on sample images and sample image processing results corresponding to the sample images; the image processing model is a self-attention depth model with a hierarchical structure, the equivalent calculation matrix of each level in the image processing model is obtained by decomposing each initial calculation matrix based on a decomposition matrix, the decomposition matrix is obtained based on a preset number of column matrixes in each level of initial calculation matrix, and the preset number is greater than the effective rank of the initial calculation matrix and less than the number of rows of the initial calculation matrix.
In addition, the logic instructions in the memory 320 may be implemented in the form of software functional units and stored in a computer readable storage medium when the logic instructions are sold or used as independent products. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
In another aspect, the present invention also provides a computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions which, when executed by a computer, enable the computer to perform the decomposition matrix-based image processing method provided by the above methods, the method comprising: determining an image to be processed; inputting the image to be processed into an image processing model to obtain an image processing result output by the image processing model; the image processing model is obtained by training based on sample images and sample image processing results corresponding to the sample images; the image processing model is a self-attention depth model with a hierarchical structure, the equivalent calculation matrix of each level in the image processing model is obtained by decomposing each initial calculation matrix based on a decomposition matrix, the decomposition matrix is obtained based on a preset number of column matrixes in each level of initial calculation matrix, and the preset number is greater than the effective rank of the initial calculation matrix and less than the number of rows of the initial calculation matrix.
In yet another aspect, the present invention also provides a non-transitory computer-readable storage medium having stored thereon a computer program, which when executed by a processor is implemented to perform the decomposition matrix-based image processing methods provided above, the method comprising: determining an image to be processed; inputting the image to be processed into an image processing model to obtain an image processing result output by the image processing model; the image processing model is obtained by training based on sample images and sample image processing results corresponding to the sample images; the image processing model is a self-attention depth model with a hierarchical structure, the equivalent calculation matrix of each level in the image processing model is obtained by decomposing each initial calculation matrix based on a decomposition matrix, the decomposition matrix is obtained based on a preset number of column matrixes in each level of initial calculation matrix, and the preset number is greater than the effective rank of the initial calculation matrix and less than the number of rows of the initial calculation matrix.
The above-described embodiments of the apparatus are merely illustrative, and the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.
Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware. With this understanding in mind, the above-described technical solutions may be embodied in the form of a software product, which can be stored in a computer-readable storage medium such as ROM/RAM, magnetic disk, optical disk, etc., and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the methods described in the embodiments or some parts of the embodiments.
Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, and not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.
Claims (9)
1. An image processing method based on a decomposition matrix, comprising:
determining an image to be processed;
inputting the image to be processed into an image processing model to obtain an image processing result output by the image processing model;
the image processing model is obtained by training based on sample images and sample image processing results corresponding to the sample images; the image processing model is a self-attention depth model with a hierarchical structure, the equivalent calculation matrix of each level in the image processing model is obtained by decomposing each initial calculation matrix based on a decomposition matrix, the decomposition matrix is obtained based on a preset number of column matrixes in each level of initial calculation matrix, and the preset number is greater than the effective rank of the initial calculation matrix and less than the number of rows of the initial calculation matrix;
the equivalent calculation matrix is determined based on the following steps:
performing singular value decomposition on the decomposition matrix to obtain a plurality of sub-matrixes, and determining characteristic values corresponding to the sub-matrixes;
selecting a plurality of sub-matrixes with the maximum eigenvalue larger than a preset value to form an eigenvalue matrix, and determining a left singular vector and a right singular vector corresponding to the eigenvalue matrix;
randomly extracting a preset number of row matrixes from the initial calculation matrix to form a row synthesis matrix, and randomly extracting a preset number of column matrixes from the initial calculation matrix to form a column synthesis matrix;
determining the equivalent computation matrix based on the column synthesis matrix, the row synthesis matrix, the eigenvalue matrix, the left singular vector, and the right singular vector.
2. The decomposition matrix-based image processing method according to claim 1, wherein the decomposition matrix is determined based on:
randomly extracting the row matrixes with the preset number from the column synthesis matrix to form an initial decomposition matrix;
and if the rank of the initial decomposition matrix is greater than the effective rank of the initial calculation matrix, taking the initial decomposition matrix as the decomposition matrix.
3. The decomposition matrix-based image processing method of claim 1, wherein determining the equivalence computation matrix based on the column synthesis matrix, the row synthesis matrix, the eigenvalue matrix, the left singular vector, and the right singular vector comprises:
matrix multiplication is carried out on the column synthesis matrix, the right singular vector and the inverse matrix of the eigenvalue matrix to obtain a first matrix;
performing matrix multiplication on the transposed matrix of the left singular vector and the row synthesis matrix to obtain a second matrix;
and taking a multiplication matrix of the first matrix and the second matrix as the equivalent calculation matrix.
4. The decomposition matrix-based image processing method according to claim 3, wherein the inputting the image to be processed into an image processing model to obtain an image processing result output by the image processing model comprises:
inputting the image to be processed to a pixel extraction layer of the image processing model to obtain a pixel matrix output by the pixel extraction layer;
inputting the pixel matrix to a self-attention layer of the image processing model, and performing matrix multiplication on the pixel matrix and an equivalent calculation matrix by the self-attention layer to obtain a self-attention value output by the self-attention layer;
and inputting the self-attention value into an image processing layer of the image processing model to obtain an image processing result output by the image processing layer.
5. The decomposition matrix-based image processing method according to claim 4, wherein the matrix-multiplying the pixel matrix and the equivalent computation matrix to obtain the self-attention value output by the self-attention layer comprises:
matrix multiplication is carried out on the pixel matrix and the first matrix to obtain an intermediate matrix;
and carrying out matrix multiplication on the intermediate matrix and the second matrix to obtain a self-attention value output by the self-attention layer.
6. The decomposition matrix-based image processing method according to any one of claims 1 to 5, wherein the initial calculation matrix comprises at least one of a query matrix, a key value matrix, and a cost matrix.
7. An image processing apparatus based on a decomposition matrix, comprising:
an image determining unit for determining an image to be processed;
the image processing unit is used for inputting the image to be processed into an image processing model to obtain an image processing result output by the image processing model;
the image processing model is obtained by training based on sample images and sample image processing results corresponding to the sample images; the image processing model is a self-attention depth model with a hierarchical structure, the equivalent calculation matrix of each level in the image processing model is obtained by decomposing each initial calculation matrix based on a decomposition matrix, the decomposition matrix is obtained based on a preset number of column matrixes in each level of initial calculation matrix, and the preset number is greater than the effective rank of the initial calculation matrix and less than the number of rows of the initial calculation matrix;
further comprising:
the decomposition unit is used for carrying out singular value decomposition on the decomposition matrix to obtain a plurality of sub-matrixes and determining characteristic values corresponding to the sub-matrixes;
the eigenvalue determining unit is used for selecting a plurality of sub-matrixes with the maximum eigenvalue and larger than a preset value to form an eigenvalue matrix and determining a left singular vector and a right singular vector corresponding to the eigenvalue matrix;
a first forming unit, configured to randomly extract a preset number of column matrices from the initial calculation matrix to form a column synthesis matrix;
a third forming unit, configured to randomly extract a preset number of row matrices from the initial calculation matrix to form a row synthesis matrix;
an equivalence calculation matrix determination unit configured to determine the equivalence calculation matrix based on the column synthesis matrix, the row synthesis matrix, the eigenvalue matrix, the left singular vector, and the right singular vector.
8. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the steps of the decomposition matrix-based image processing method according to any one of claims 1 to 6 when executing the program.
9. A non-transitory computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the decomposition matrix-based image processing method according to any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110544946.4A CN113362402B (en) | 2021-05-19 | 2021-05-19 | Image processing method and device based on decomposition matrix |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110544946.4A CN113362402B (en) | 2021-05-19 | 2021-05-19 | Image processing method and device based on decomposition matrix |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113362402A CN113362402A (en) | 2021-09-07 |
| CN113362402B true CN113362402B (en) | 2023-03-17 |
Family
ID=77526546
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110544946.4A Active CN113362402B (en) | 2021-05-19 | 2021-05-19 | Image processing method and device based on decomposition matrix |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113362402B (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5812072A (en) * | 1994-06-03 | 1998-09-22 | Masters; John | Data conversion technique |
| CN105323037A (en) * | 2014-08-01 | 2016-02-10 | 中国移动通信集团公司 | Pre-coding method and device according to complex matrix |
| CN106447630A (en) * | 2016-09-18 | 2017-02-22 | 清华大学 | High-spectral image sharpening method based on probability matrix decomposition |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| AUPQ055999A0 (en) * | 1999-05-25 | 1999-06-17 | Silverbrook Research Pty Ltd | A method and apparatus (npage01) |
| CN108492273A (en) * | 2018-03-28 | 2018-09-04 | 深圳市唯特视科技有限公司 | A kind of image generating method based on from attention model |
| CN111340186B (en) * | 2020-02-17 | 2022-10-21 | 之江实验室 | Compressed representation learning method based on tensor decomposition |
-
2021
- 2021-05-19 CN CN202110544946.4A patent/CN113362402B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5812072A (en) * | 1994-06-03 | 1998-09-22 | Masters; John | Data conversion technique |
| CN105323037A (en) * | 2014-08-01 | 2016-02-10 | 中国移动通信集团公司 | Pre-coding method and device according to complex matrix |
| CN106447630A (en) * | 2016-09-18 | 2017-02-22 | 清华大学 | High-spectral image sharpening method based on probability matrix decomposition |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113362402A (en) | 2021-09-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109871532B (en) | Text theme extraction method and device and storage medium | |
| CN110222598B (en) | Video behavior identification method and device, storage medium and server | |
| CN109859288B (en) | Image coloring method and device based on generation countermeasure network | |
| CN104317902B (en) | Image search method based on local holding iterative quantization Hash | |
| FI3761235T3 (en) | Transposing neural network matrices in hardware | |
| CN115115905B (en) | High-mobility image countermeasure sample generation method based on generation model | |
| CN112508190A (en) | Method, device and equipment for processing structured sparse parameters and storage medium | |
| JP6107531B2 (en) | Feature extraction program and information processing apparatus | |
| CN116075821A (en) | Form convolution and acceleration | |
| CN112528318A (en) | Image desensitization method and device and electronic equipment | |
| CN111639699A (en) | Method, system and equipment for extracting image features and readable storage medium | |
| CN113362402B (en) | Image processing method and device based on decomposition matrix | |
| CN111680162A (en) | Knowledge graph embedding method, system and equipment based on tensor decomposition | |
| CN113361567B (en) | Image processing method, device, electronic equipment and storage medium | |
| CN111985487B (en) | Remote sensing image target extraction method, electronic equipment and storage medium | |
| CN113379657B (en) | Image processing method and device based on random matrix | |
| CN116993629B (en) | Smoothing method and device based on image decomposition, electronic equipment and storage medium | |
| CN111026087B (en) | Weight-containing nonlinear industrial system fault detection method and device based on data | |
| CN111695689B (en) | Natural language processing method, device, equipment and readable storage medium | |
| CN113505838B (en) | Image clustering method and device, electronic equipment and storage medium | |
| CN114565772B (en) | Method and device for extracting set features, electronic equipment and storage medium | |
| CN113177462B (en) | Target detection method suitable for court trial monitoring | |
| CN116433662B (en) | Neuron extraction method and device based on sparse decomposition and depth of field estimation | |
| CN114387626B (en) | Gesture classification method and device, electronic equipment and storage medium | |
| CN115170916B (en) | Image reconstruction method and system based on multi-scale feature fusion |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| EE01 | Entry into force of recordation of patent licensing contract |
Application publication date: 20210907 Assignee: Hangzhou Biren Technology Development Co.,Ltd. Assignor: Shanghai Bilin Intelligent Technology Co.,Ltd. Contract record no.: X2023310000122 Denomination of invention: Image processing method and device based on decomposition matrix Granted publication date: 20230317 License type: Common License Record date: 20230701 |
|
| EE01 | Entry into force of recordation of patent licensing contract | ||
| CP03 | Change of name, title or address |
Address after: 201114 room 1302, 13 / F, building 16, 2388 Chenhang Road, Minhang District, Shanghai Patentee after: Shanghai Bi Ren Technology Co.,Ltd. Country or region after: China Address before: 201114 room 1302, 13 / F, building 16, 2388 Chenhang Road, Minhang District, Shanghai Patentee before: Shanghai Bilin Intelligent Technology Co.,Ltd. Country or region before: China |
|
| CP03 | Change of name, title or address |