WO2023000526A1

WO2023000526A1 - Image interpolation model training method based on residual-guided policy

Info

Publication number: WO2023000526A1
Application number: PCT/CN2021/125577
Authority: WO
Inventors: 钟宝江; 苏润
Original assignee: 苏州大学
Priority date: 2021-07-22
Filing date: 2021-10-22
Publication date: 2023-01-26
Also published as: CN113538242B; CN113538242A

Abstract

An image interpolation model training method based on a residual-guided policy, an image interpolation method based on a residual-guided policy, a computer device, and a readable storage medium. On the basis of characteristics of a random forest level structure, the residual-guided policy is used to construct and train the image interpolation model. A pre-interpolated image is used as feature data, and a residual is used as label data to train a random forest; the random forest grows synchronously according to levels in the training process; an initial residual is a difference between a high-resolution image and the pre-interpolated image, and a later residual is a difference between an upper-level residual and an estimated residual; because the residual can be updated in iteration, the estimated residual of each level is the optimization of the previous-level residual; an image residual is converged to zero along with the increase of a training level; and finally, the image interpolation model with the determined mapping relationship of each level is obtained. A good interpolation effect can be obtained by using the image interpolation model to interpolate the low-resolution image.

Description

A Training Method of Image Interpolation Model Based on Residual Guidance Strategy

This application claims the priority of the Chinese patent application with the application number 202110830807.8 and the title of the invention "A Method for Training Image Interpolation Model Based on Residual Guidance Strategy" submitted to the China Patent Office on July 22, 2021, the entire content of which Incorporated in this application by reference.

technical field

The present application relates to the field of computer technology, in particular to an image interpolation model training method based on a residual guidance strategy, an image interpolation method based on a residual guidance strategy, computer equipment, and a readable storage medium.

Background technique

Due to limitations in hardware design and cost of image acquisition equipment, the resolution of some regions of interest in the acquired digital images may be low. Image interpolation is a method of restoring a low-resolution image to a high-resolution image and maintaining the details and structure of the original low-resolution image as much as possible.

Although the traditional bilinear and bicubic interpolation (Bicubic) methods can realize image interpolation, the interpolation results will have obvious artificial traces at the edge of the image, and there are many areas containing noise and blur. In order to improve interpolation performance, more prior information needs to be considered, such as edge-guided interpolation methods, and image interpolation methods based on local or non-local pixels or image blocks. However, according to the residual image analysis comparing the standard image and the poor interpolation results of various methods, it can be seen that the interpolation methods proposed in recent years have better interpolation results in smooth areas, but poor interpolation results in edge areas.

To sum up, the effect of the current image interpolation method is not ideal, and how to improve the image interpolation effect is an urgent problem to be solved by those skilled in the art.

Contents of the invention

The purpose of this application is to provide an image interpolation model training method based on a residual-guided strategy, an image interpolation method based on a residual-guided strategy, computer equipment, and a readable storage medium to solve the problem of poor interpolation effects of current image interpolation schemes Ideal question. The specific plan is as follows:

In the first aspect, the present application provides a method for training an image interpolation model based on a residual guidance strategy, including:

Obtaining a high-resolution image; down-sampling the high-resolution image to obtain a low-resolution image; generating a pre-interpolation image according to the low-resolution image;

Making a difference between the high-resolution image and the pre-interpolation image to obtain an initial residual;

Use the pre-interpolation image and the initial residual to train the random forest; during the training process, the random forest grows synchronously by layers, and the initial residual is used as the first-level residual, in the random forest At any level, learn the mapping relationship between the pre-interpolation image and the residual of the current level to generate an estimated residual, and make a difference between the residual of the current level and the estimated residual to obtain the residual of the next level;

When the training termination condition is reached, the random forest whose mapping relationship at each level is determined is output as an image interpolation model.

Optionally, the random forest is divided into multiple groups of random forests, and the training of the random forest by using the pre-interpolation image and the initial residual includes:

generating a feature vector of the pre-interpolated image;

According to the fixed point distribution mode, the feature vectors are grouped, wherein the number of groups of the feature vectors is equal to the number of groups of the random forest;

When training the random forest, a group of random forests are trained for each set of feature vectors and corresponding residual vectors in the initial residual.

Optionally, the generating the feature vector of the pre-interpolation image includes:

Filtering the pre-interpolation image by using a one-dimensional first-order gradient operator and a second-order gradient operator to generate four corresponding feature images; sampling the four feature images to obtain a feature vector of each sampling position.

Optionally, the sampling method of the four feature images is specifically: sampling at intervals with a step size of 1;

Correspondingly, there are four fixed point distribution patterns, and the number of groups of the feature vector and the number of groups of the random forest are both 4.

Optionally, the image interpolation model specifically includes the K-level random forest; the pre-interpolation image of the first-level random forest is an image generated using a preset interpolation algorithm, and for any k∈[2,K], the k-th level The pre-interpolation image of the random forest is the image obtained by sequential interpolation of the previous k-1 random forest.

Optionally, in the image interpolation model, the high-resolution images of random forests at different levels are different.

Optionally, the random forest grows synchronously by layers, including:

At any level of the random forest, judging whether there is an unprocessed target node;

If it exists, generate the first linear transformation from the feature vector contained in the target node to the residual vector contained in the target node, and then generate the second linear transformation from the feature vector contained in the target node to the target residual vector , wherein the target residual vector is a residual vector that intersects with the target node;

If it does not exist, it is judged whether the split termination condition is reached;

If it is reached, it is determined that the target node belongs to a leaf node, and the second linear transformation of the target node is recorded, and finally the second linear transformation of all leaf nodes is the difference between the pre-interpolation image and the sum of the residuals of each level Mapping relations;

If not, determine that the target node belongs to an internal node, and enter the next level through node splitting; in the process of node splitting, randomly select the splitting parameters and split the target node, and determine the optimal node according to the amount of error reduction before and after splitting optimal splitting parameter, record the optimal splitting parameter of the target node.

In the second aspect, the present application provides an image interpolation method based on a residual guidance strategy, including:

Obtain the low-resolution image to be interpolated;

generating a pre-interpolated image according to the low-resolution image;

Input the pre-interpolated image into the trained random forest; at any level of the random forest, generate an estimated residual according to the mapping relationship between the pre-interpolated image learned in the training process and the residual of the current level;

An interpolated image of the low-resolution image is generated according to the estimated residual of each level and the pre-interpolated image.

In a third aspect, the present application provides a computer device, which is characterized in that it includes:

memory: used to store computer programs;

Processor: configured to execute the computer program to implement the above-mentioned method for training an image interpolation model based on a residual-guided strategy, and/or the above-mentioned image interpolation method based on a residual-guided strategy.

In a fourth aspect, the present application provides a readable storage medium, where a computer program is stored on the readable storage medium, and when the computer program is executed by a processor, it is used to implement the image based on the residual guidance strategy as described above An interpolation model training method, and/or, an image interpolation method based on a residual guidance strategy as described above.

To sum up, this application provides an image interpolation model training method based on the residual guidance strategy, including: obtaining a high-resolution image; downsampling the high-resolution image to obtain a low-resolution image; generating Pre-interpolation image; make a difference between the high-resolution image and the pre-interpolation image to obtain the initial residual; use the pre-interpolation image and the initial residual to train the random forest; during the training process, the random forest grows synchronously by layer, and the initial residual As the residual of the first level, at any level of the random forest, learn the mapping relationship between the pre-interpolated image and the residual of the current level to generate an estimated residual, and make a difference between the residual of the current level and the estimated residual to obtain the next level Residual error; when the training termination condition is reached, output a random forest with determined mapping relationships at each level as an image interpolation model.

It can be seen that this method is based on the characteristics of the random forest hierarchy, and uses the residual guide strategy to build and train the image interpolation model. Specifically, the random forest is trained using the pre-interpolated image as the feature data and the residual as the label data. During the training process, the random forest grows synchronously by layer. The initial residual is the difference between the high-resolution image and the pre-interpolated image. The residual of the latter level is the difference between the residual of the previous level and the estimated residual of the previous level. Since the residual can be updated in iterations, the estimated residual of each level is an optimization of the residual of the previous level. With the increase of the training level, the image residual converges to zero, and finally an image interpolation model with definite mapping relationship at each level is obtained. Using it to interpolate low-resolution images can significantly improve the interpolation effect.

In addition, the present application also provides an image interpolation method based on a residual guidance strategy, a computer device and a readable storage medium, the technical effects of which are corresponding to those of the above method, and will not be repeated here.

Description of drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the accompanying drawings that need to be used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings in the following description are only For some embodiments of the present application, those of ordinary skill in the art can also obtain other drawings based on these drawings without creative effort.

FIG. 1 is an overall flow chart of Embodiment 1 of the image interpolation model training method based on the residual guidance strategy provided by the present application;

2 is a schematic diagram of the data preprocessing process in Embodiment 1 of the image interpolation model training method based on the residual guidance strategy provided by the present application;

Fig. 3 is a schematic diagram of the random forest layer-by-layer training process provided by the application;

FIG. 4 is a flowchart of Embodiment 1 of an image interpolation method based on a residual guidance strategy provided by the present application;

FIG. 5 is a schematic diagram of an image interpolation process according to a training result provided by the present application.

detailed description

In order to enable those skilled in the art to better understand the solution of the present application, the present application will be further described in detail below in conjunction with the drawings and specific implementation methods. Apparently, the described embodiments are only some of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the scope of protection of this application.

In a nutshell, the goal of image interpolation is to transform a low-resolution image into a high-resolution image while maintaining as much detail and structure as possible in the original low-resolution image. This application implements image interpolation through machine learning models. Usually, training a model is to let it learn the mapping relationship between the feature data X and the label data Y, but the model M we choose may have limitations in its learning ability, resulting in a bottleneck in its predictive ability, and Using the residual bootstrap strategy can help the model break through the limitations.

The residual, as the difference between the true value and the predicted value, reflects the insufficient prediction of the model for the data, and also provides a basis for revising the model. Using the residual instead of the true value as the label to participate in the training is no different from directly using the true value as the label, but the residual has a remarkable feature, that is, it can be updated in iterations. Therefore, this application allows the model to be expanded under the guidance of residuals, and the new model refines the previous residuals, and the final model obtained after repeated iterations will produce better prediction results. According to this strategy, the estimated value of the model output will continue to approach the true value. If the first-level model M ⁽¹⁾ cannot make perfect predictions, a new residual error must be generated, which is represented by ε, and then the second model M ⁽²⁾ is used to learn this part of the residual error. Let M ⁽²⁾ make up for the deficiency of M ⁽¹⁾ , and a new residual error will be generated at this time, but ideally, each component of this residual error will be smaller than before. And so on. Up to the nth-level model, the strength of each component of the remaining residual ε is almost 0 at this time, that is to say, after each iteration is perfected, the predicted value of the model approaches the true value. This is the overall idea of the residual guidance strategy.

The core of the present application is to provide an image interpolation model training method based on a residual guidance strategy, an image interpolation method based on a residual guidance strategy, computer equipment and a readable storage medium. Utilizing the characteristics of the random forest hierarchy, a residual bootstrap strategy is applied to build and train the model. Theoretically, as the training level increases, the image residual converges to zero, so that the interpolation effect can be improved. During the training process, each level saves the corresponding regression function, that is, the mapping relationship between the low-resolution image block and the residual image block at the current stage. In the interpolation stage, the residuals are predicted by using the mapping relationship of each level. In view of the convergence nature of model training, the estimated residuals of each level are the optimization of the residuals of the previous level. The reconstruction of the image guarantees the quality of the reconstruction. A large number of experimental results show that this application can provide interpolation images with high precision and good subjective feeling.

Embodiment 1 of the method for training an image interpolation model based on a residual guidance strategy provided by this application is introduced below.

Referring to Fig. 1, embodiment one comprises the following steps:

S11. Obtain a high-resolution image; down-sample the high-resolution image to obtain a low-resolution image; generate a pre-interpolation image based on the low-resolution image;

S12. Perform a difference between the high-resolution image and the pre-interpolation image to obtain an initial residual error;

S13. Use the pre-interpolation image and the initial residual to train the random forest; during the training process, the random forest grows synchronously by layer, the initial residual is used as the first-level residual, and at any level of the random forest, learn the pre-interpolation image and The mapping relationship between the residuals of the current level and then generate the estimated residuals, and make a difference between the residuals of the current level and the estimated residuals to obtain the residuals of the next level;

S14. When the training termination condition is reached, output a random forest with determined mapping relationships at each level as an image interpolation model.

Specifically, in order to improve efficiency, the above-mentioned pre-interpolated image and the initial residual are preprocessed before being used as training data. The preprocessing process includes but is not limited to: sampling the pre-interpolation image to obtain a feature vector for training; sampling the initial residual to obtain a residual vector for training.

In order to further improve the interpolation effect, the training data can be grouped according to certain rules. Correspondingly, the entire random forest is also divided into multiple groups of random forests to ensure that the number of groups of training data is equal to the number of groups of the random forest. During the training process, A set of training data trains a set of random forests. As a specific implementation method, considering that image interpolation must keep the value of the pixel at the fixed point position unchanged before and after interpolation, that is, the value of the pixel at the fixed point position of the image after interpolation corresponds to the value in the low-resolution image one-to-one , so the feature vectors can be grouped according to the fixed point distribution patterns contained in the sampling results of the pre-interpolated image blocks. Here, the fixed point distribution pattern contained in the sampling result is mainly affected by the sampling rules. Assuming that the sampling method is: sampling with a step size of 1 interval, then there are four fixed point distribution patterns contained in the sampling result. At this time, the number of groups of the feature vector is 4, and the number of groups of the random forest is also 4.

A way to improve the interpolation effect is mentioned above, which is to divide the training data and random forest into multiple groups. Next, another method to improve the interpolation effect is provided. In practical applications, these two methods can be combined or used alone.

In order to improve the interpolation effect, the image interpolation model can be allowed to include a multi-level random forest, that is, the structure of the image interpolation model is a cascaded random forest. Each level of random forest is trained as described above. It should be noted that during the training process, the pre-interpolation image of the first level of random forest is an image generated by using a preset interpolation algorithm. This embodiment does not limit the selection of What kind of image interpolation algorithm; for any k∈[2,K], the pre-interpolation image of the k-th random forest is the image obtained by sequential interpolation of the previous k-1 random forest.

On this basis, in order to avoid overfitting, different levels of random forests use different high-resolution images.

To sum up, this embodiment provides an image interpolation model training method based on the residual guidance strategy. The initial residual generated by the difference between the pre-interpolation image, the high-resolution image and the pre-interpolation image is used as the input of the random forest. During the process, the random forest grows layer by layer, and after the growth of each layer is completed, the residual is refined, that is, a new residual is generated, and the new residual will guide the growth of the next layer of the random forest. The initial residual is the difference between the high-resolution image and the pre-interpolated image, and the residual at the next level is the difference between the residual of the previous level and the estimated residual of the previous level. According to this idea of residual guidance, this embodiment constructs an image interpolation model based on random forest, and trains it based on the residual guidance strategy. The trained model can significantly improve the quality of image interpolation.

On the basis of the first embodiment above, the preprocessing process of the pre-interpolation image and the initial residual will be introduced in detail below. This is only provided as a feasible preprocessing manner, and this embodiment does not limit the preprocessing manners of the two.

As shown in Figure 2, the preprocessing process of the pre-interpolated image includes the following steps:

S21. Filter and sample the pre-interpolation image to obtain a feature vector of each sampling position;

Specifically, the above-mentioned filtering and sampling process can be specifically as follows: filter the pre-interpolation image with a one-dimensional first-order gradient operator and second-order gradient operator to generate four corresponding feature images; sample the four feature images to obtain A feature vector for each sampling location.

S22. Perform edge detection on the pre-interpolation image to obtain an edge image; sample the edge image to obtain an edge image block;

S23. Sampling the pre-interpolation image to obtain a pre-interpolation image block;

S24. Filter the feature vectors of all sampling positions according to the edge image block at each sampling position, and filter out feature vectors whose edge pixel intensity values are greater than 0;

S25. Group the filtered feature vectors according to the fixed point distribution patterns included in all pre-interpolated image blocks; record the number of groups of feature vectors as H, and the random forest is also divided into H groups of random forests;

S26. For any h∈[1,H], perform dimensionality reduction on the hth group of feature vectors to obtain feature vectors of pre-interpolation images used for training.

Figure 2 illustrates the generation process of the first-level residual, that is, the difference between the high-resolution image and the pre-interpolated image is obtained to obtain the initial residual. Figure 2 also illustrates the preprocessing process of the initial residual, including the following steps:

S31. Sampling the initial residual to obtain a residual image block at each sampling position;

S32. For any h∈[1,H], determine the residual image block corresponding to the hth group of feature vectors of the pre-interpolation image, and obtain the hth group of residual vectors of the initial residual. That is to say, the residual vector and the dimension-reduced feature vector are combined according to the sampling position, and the feature vector processed in S26 is associated with the residual vector one by one.

As mentioned above, a set of training data is used to train a set of random forests, that is, a set of feature vectors and their corresponding residual vectors are used to train a set of random forests. So, in detail, the training process is as follows: For any h ∈ [1, H], the h-th set of random forests is trained using the h-th set of feature vectors of the pre-interpolated image and the h-th set of residual vectors of the initial residuals.

It should be noted that, in FIG. 2 , the sampling methods of the pre-interpolation image, the edge image and the residual image are the same. As a specific implementation manner, a specific sampling rule may be: sampling at intervals with a step size of 1. At this time, there are 4 fixed point distribution patterns contained in the pre-interpolation image sampling results, and the number of groups of feature vectors is 4, that is, the value of H above is 4. Correspondingly, the random forest is divided into 4 groups.

On the basis of the first embodiment above, the training process of the random forest is introduced in detail below. This is only provided as a feasible training method, and this embodiment does not limit what kind of training method is adopted.

As described in S13, during the training process of this embodiment, the random forest grows synchronously by layers. As shown in Figure 3, in the first level, the pre-interpolation image X and the first level residual R ⁽¹⁾ are used as training data, and the nodes of the first level of random forest learn the mapping relationship between them, according to the mapping relationship The estimated residual for the first-level residual R ⁽¹⁾ can be obtained

For the first-level residual R ⁽¹⁾ and the estimated residual

Do the difference to obtain the refined residual F as the second level residual R ⁽²⁾ . In the second-level random forest, the pre-interpolated image X and the second-level residual R ⁽²⁾ are used as training data, and so on.

In the above S13, at any level of the random forest, the process of learning the mapping relationship between the pre-interpolation image and the residual of the current level and then generating the estimated residual, specifically includes the following steps:

S40. For any h∈[1, H], initialize the root nodes of all decision trees of the random forest of the hth group by using the hth group of feature vectors of the pre-interpolation image and the hth group of residual vectors of the initial residual;

S41. Control all the decision trees to grow synchronously by layers. At any level of the random forest, judge whether there is an unprocessed target node. If there is, go to S42, otherwise go to S43;

S42. Generate a first linear transformation from the feature vector contained in the target node to a residual vector contained in the target node, and then generate a second linear transformation from the feature vector contained in the target node to the target residual vector, wherein the target residual vector is The residual vector that intersects with the target node among the residual vectors contained in the root node;

S43, judging whether the split termination condition is met, if so, proceed to S44, otherwise proceed to S45;

S44. Determine that the target node belongs to a leaf node, and record the second linear transformation of the target node, and finally the second linear transformation of all leaf nodes is the mapping relationship between the pre-interpolation image and the sum of the residuals of each level;

S45. Determine that the target node belongs to an internal node, and enter the next level through node splitting; during the node splitting process, randomly select splitting parameters and split the target node, determine the optimal splitting parameter according to the amount of error reduction before and after splitting, and record the target node optimal splitting parameters.

It can be seen that in this embodiment, the node not only calculates the linear transformation between the feature vector contained in itself and the residual vector contained in itself, but also superimposes the linear transformation of all its ancestor nodes, and further calculates the feature vector contained in itself The linear transformation between the vector and the target residual vector, the target residual vector is the residual vector that intersects with this node among all the residual vectors. Therefore, in this embodiment, only leaf nodes need to record linear transformations, and internal nodes do not need to record their linear transformations. It can be understood that since the leaf nodes do not need to continue splitting, the leaf nodes do not need to record the optimal splitting parameters, only the internal nodes need to record the optimal splitting parameters.

In short, the linear transformation calculated at each level will be superimposed on the linear transformation of its child nodes (that is, the above-mentioned process of generating the second linear transformation based on the first linear transformation). From another perspective, the linear transformation calculated by the child nodes can be used for The refinement of the linear transformation of the parent node is based on the fact that the residual is additive, and the linear transformation calculated from the residual can also be superimposed.

The second embodiment of the image interpolation model training method based on the residual guidance strategy provided by the present application will be introduced in detail below.

In the second embodiment, the image interpolation model is a K-level cascaded random forest, and each level of random forest is divided into four groups of random forests. The input and output of embodiment two are as follows:

Input: training image dataset, maximum height/level L of random forest, number N of decision trees contained in random forest, maximum number of stages K of cascaded random forest.

Output: Trained Cascaded Random Forest

where each random forest contains N decision trees, namely

In this embodiment, the whole training process is divided into three stages: the data preparation stage, the first-level random forest training stage, and the remaining random forest training stages. Each stage is described below.

First, the data preparation phase

S401. Convert the high-resolution image from the RGB color space to the YCbCr color space, and then perform training only on the Y channel image.

S402. Down-sample the high-resolution image {I _Y } at intervals of one pixel to simulate a low-resolution image acquired under realistic conditions.

S403, use the Bicubic algorithm to pre-interpolate the low-resolution image so that it has the same size as the original image, and use the pre-interpolation image

Instead of low-resolution images as feature data to participate in training.

S404, by combining the corresponding high-resolution image {I _Y } with the pre-interpolation image

Do the difference to get the residual image {I _R }, and use it to replace the high-resolution image {I _Y } as the label data to participate in the training.

S405. Use the Canny edge detection function in Matlab to detect the edge of the pre-interpolation image to obtain the edge image {I _E }.

S406. Filter the pre-interpolation image according to the one-dimensional first-order gradient operator and second-order gradient operator to generate four corresponding feature images

Collect image blocks with a step size of 1 on the four feature images, and each position will generate four image blocks of size 5×5

These image blocks are vectorized (converted from 5×5 to 25×1), and stitched (from 4 25×1 to 100×1), and then the stitched vector is obtained

as feature vectors for training.

Among them, the forms of the first-order gradient operator and the second-order gradient operator are as follows:

S407, for the pre-interpolation image

The residual image {I _R } and the edge image {I _E } are sampled in the same way to obtain pre-interpolation image blocks, residual image blocks and edge image blocks.

S408. Each eigenvector x _i has a corresponding residual image block r _i , merge the eigenvectors into a matrix X=[x ₁ , x ₂ , . . . , x _D ], and merge the residual image blocks into a matrix , the two constitute a group of {X, R ⁽¹⁾ } to participate in random forest training, where D is the number of image blocks, and the superscript 0 indicates the number of layers in the decision tree, that is, the number of iterations performed. The form of the residual matrix is as follows:

S409. According to the edge image blocks, filter out the feature vectors whose intensity values of the edge pixels are greater than 0, and keep the corresponding feature vectors and residual image blocks.

S410. Group the feature vectors according to the fixed point distribution pattern included in the sampling result of the pre-interpolation image block. There are only four distribution modes here, so the eigenvectors are divided into four groups.

S411. Use PCA to reduce the dimensionality of the eigenvectors of different groups, and save the next four PCA matrices P _j (j=1, 2, 3, 4) and the dimensionality-reduced feature matrix

S412, finally obtain the training data, which are respectively used to train four groups of random forests in each level, and the form of the training data is as follows:

It can be understood that this embodiment divides the eigenvectors into four groups according to the fixed point mode, so there are four groups of training data matrices, and each group contains the matrix of the reduced eigenvectors

and the residual matrix

j = 1, 2, 3, 4. Each level of random forest contains 4 groups of random forests, and the 4 groups of random forests correspond to four different fixed point patterns. On the whole, all the feature vectors X are trained together in the first-level random forest, but in detail, all the feature vectors X are divided into four groups according to the pattern X ₁ , X ₂ , X ₃ , and X ₄ respectively train four groups in the same level of random forest random forest.

Second, the training phase of the first level random forest

Train the first level random forest according to the residual bootstrapping strategy

Include the following steps:

S51, the first-level random forest to be trained

in

For all decision trees in the random forest of the jth group, use the data

Initialize the root node.

S52, N decision trees in the jth group grow synchronously by layers, when all the decision trees in the random forest train the l (l=1, 2,..., L-1) layer, for the nth decision tree

If there is still an unprocessed node α, it will be split.

S53. When the current level l<L-1, the residual needs to be refined. For the nth decision tree

If there are also nodes with unrefined residuals

Then follow the steps below to refine the residual: According to

Estimation residuals for X _β in node β

Each of the residual vectors can be estimated by the following formula:

Afterwards, the refinement of the residuals in node β is completed according to the following formula:

Among them, x _i is the i-th eigenvector in X _β , and all

Flatten into a matrix by column

S54, finally, save the trained random forest

In the above S52, the node splitting process is as follows:

S521. Node α contains data

By solving the following formula to get from X _α to

The linear transformation of , that is, the first linear transformation mentioned above:

X _α to

The linear transformation form of is as follows:

S522. The ancestor nodes of node α have completed the training, and obtained corresponding linear transformations, and combined these linear transformations with

accumulated

That is, X _α to

The linear transformation of , that is, the second linear transformation mentioned above, where

Refers to

The residual vector that intersects with node α. Here α ⁽ⁱ⁾ refers to the ancestor node of node α, α ⁽⁰⁾ is the root node of node α, α ^(l-1) is the parent node of node α, and α ^(l) is node α.

S523. If the number of eigenvectors contained in node α is less than 200 or the current level l=L-1, no further splitting is performed, and node α is marked as a leaf node, and W _α is stored for use in the interpolation stage.

S524. Randomly select a series of splitting parameters {Θ ₁ , Θ ₂ , ..., Θ _p }, where Θ _p = {θ ₁ , Θ ₂ , τ}, θ _j (j=1, 2) means that in X _α In line θ _j of , classify the i-th feature vector according to the result of the split function S( _xi , Θ _p )= _xi [θ ₁ ] _-xi [θ ₂ ]-τ, if S( _xi , Θ _p )≥0, it will be classified into child node β, otherwise it will be classified into child node γ.

Wherein, p=1, 2, . . . , P, as a specific implementation manner, the value is as follows: P=6. τ represents a gray value threshold. In practical applications, the gray value is normalized to [0,1], so τ∈[0,1].

S525. Use the error reduction before and after splitting to select the optimal splitting parameter Θ _p , and finally select the parameter Θ _p with the largest error reduction G _α to split the node α. The error calculation method is:

in,

To use the linear transformation stored in node δ

The estimation error for the residual R _δ , D _δ is the number of eigenvectors in the node δ.

When the nth decision tree in the jth random forest group

After the l-layer split is completed, update

random forest in

In the same group of random forests, because the split parameters of the decision tree in the process of node splitting are randomly selected, even if the inputs of different decision trees are the same, the training results are different. That is to say, the training results of different decision trees in the same random forest are different. The training results specifically refer to: the optimal splitting parameters stored in the root node and internal nodes and the mapping relationship stored in the leaf nodes.

In addition, in the same group of random forests, the dimensions of the mapping relationship obtained in each decision tree are the same, so they are additive. If a set of random forests contains two decision trees, after the feature vector x is input into the random forest, it will be assigned to a leaf node in decision tree 1, and will also be assigned to a leaf node in decision tree 2, so that the feature vector x It corresponds to two mapping relationships W ₁ and W ₂ , and these two mapping relationships can respectively complete the mapping of the feature vector x

can add them up to get

It is also possible to add up the mapping relations first to obtain W=W ₁ +W ₂ , and then directly obtain

It is worth mentioning that the residual refinement mentioned in this embodiment refers to the process of determining the residual of the next level according to the residual of the current level.

Third, the training phase of the random forest of the remaining levels

Training Cascaded Random Forests with Residual Bootstrapping Strategies

(k=2, 3, . . . , K).

It should be noted that for training the kth (k>1) level random forest

The high-resolution images need to be different from the training random forest

The training data used when, and the pre-interpolated image is used

generated after sequential interpolation. The remaining training steps are the same as the first-level random forest training process, and will not be introduced here.

Finally, the output K-level cascaded random forest

That is, the image interpolation model whose mapping relations at all levels are determined. In practical applications, K may take a value of 4.

To sum up, this embodiment provides an image interpolation model training method based on the residual guidance strategy. The purpose is to obtain a high-resolution image from a low-resolution image, and to ensure that the interpolated image has both objective indicators and subjective perception. greatly improved. This embodiment mainly describes the implementation process of the offline training phase. During the construction of each decision tree, a series of node splitting and data refining steps are iteratively executed. The data refining phase includes data division and residual update. The updated residual will be for training at the next level. In addition, a cascading strategy is also introduced to further improve the quality of image interpolation, and the cascading strategy is analyzed from a high scale. It also uses image residuals to guide the training of the model.

The above two embodiments have introduced the training process of the image interpolation model, and the image interpolation process of the image interpolation model trained in the above manner will be introduced below.

First, the first embodiment of the image interpolation method based on the residual guidance strategy provided by the present application is introduced, as shown in FIG. 4 , this embodiment includes the following steps:

S61. Obtain a low-resolution image to be interpolated;

S62. Generate a pre-interpolation image according to the low-resolution image;

S63. Input the pre-interpolation image into the trained random forest; at any level of the random forest, generate an estimated residual according to the mapping relationship between the pre-interpolation image learned during the training process and the residual of the current level;

S64. Generate an interpolated image of the low-resolution image according to the estimated residuals of each level and the pre-interpolated image.

In short, for the online image interpolation stage, the given low-resolution images will be sequentially passed through the trained cascade random forest and the interpolation will be completed. As shown in Figure 5, in each level of random forest, the image is divided from top to bottom in each decision tree in the form of feature vectors, and each feature vector is passed to the leaf node, and then the linear transformation stored in it is used to generate an estimated residual value. Difference, the estimated residual after recombination is superimposed on the pre-interpolation image, and the interpolation result of the current level of random forest is obtained.

The second embodiment of the image interpolation method based on the residual guidance strategy provided by the present application will be introduced below.

The input and output of embodiment two are as follows:

Input: low-resolution image, trained cascaded random forest

Output: Interpolated image.

In this embodiment, the entire interpolation process is divided into two stages: a data preparation stage and an image interpolation stage. The two stages are described below.

First, the data preparation stage

S71. Convert the low-resolution image from the RGB color space to the YCbCr color space.

S72, generate a pre-interpolation image from the Y channel image I _Y

Specifically, when using the first level random forest

When interpolating, the Bicubic algorithm is used to pre-interpolate the low-resolution image to obtain a pre-interpolated image; when using other random forests, the interpolation result from the previous random forest is directly used as the pre-interpolated image.

S73. Use the Canny edge detection function in Matlab to detect the edge of the pre-interpolation image to obtain the edge image {I _E }.

S74, pre-interpolating the image according to the one-dimensional first-order gradient operator and the second-order gradient operator

Perform filtering to generate the corresponding four feature images

Vectorize these image blocks and stitch them together to get the stitched vector

S75, for the pre-interpolation image

Sample image blocks in the same way as the edge image {I _E } to obtain pre-interpolation image blocks, residual image blocks and edge image blocks.

S76. Combine the feature vectors into a matrix X=[x ₁ , x ₂ , . . . , x _D ], where D is the number of image blocks.

S77. According to the edge image block, filter out the feature vectors whose intensity values of the edge pixels are greater than 0, and keep the corresponding feature vectors.

S78. Group the feature vectors according to the fixed point distribution pattern contained in the sampling result of the pre-interpolation image block. There are only four distribution modes here, so the eigenvectors are divided into four groups.

S79. Using PCA in the training phase to reduce the dimensionality of the feature vectors of different groups, and save the dimensionality-reduced feature matrix

Second, the image interpolation stage

random forest at level k

Perform image interpolation according to the processed feature matrix, including the following steps:

S81. Random forest

in

For all the decision trees in the random forest of group j, the feature matrix is used

Initialize the root node;

S82. Feature matrix

Transfer from top to bottom until the leaf node, specifically, if it reaches the feature matrix of the internal node α

has not been passed down yet, according to the optimal split parameter Θ pair recorded by node α

to divide;

S83. When the feature matrix is passed to the leaf node ρ, use the linear transformation W _p stored in the leaf node to generate an estimated residual

S84, each decision tree

Finally, an estimated residual matrix is output

In order to distinguish the prediction results of different decision trees, the residual matrix estimated by the nth decision tree is recorded as

by random forest

The estimate for the residual is

S85. Reorganize the predicted residual vector into a residual image and process the overlapping area. The specific method is: prepare two zero matrices with the same size as the interpolated image, and one of them saves the residual image

The other saves the count of overlaps at each position, when all image blocks are placed in

After that, the final residual image is obtained by taking the average

S86. Will

After being superimposed with the pre-interpolated image I _X , the k-th random forest is obtained

interpolated image

If k < K, the image will be interpolated

Used as the next level of random forest

pre-interpolated image, otherwise the image will be interpolated

To restore a color image, the specific method is to use the Bicubic algorithm to interpolate the Cb channel and Cr channel images of the low-resolution image, and then combine the three-channel images and convert them to the RGB color space.

Experiments have proved that compared with other mainstream image interpolation algorithms, the random forest image interpolation method based on the residual guidance strategy in this embodiment has significantly improved the objective index of image interpolation results.

In addition, the present application also provides a computer device, including:

memory: used to store computer programs;

Processor: configured to execute the computer program to implement the above-mentioned method for training an image interpolation model based on a residual-guided strategy, and/or, the above-mentioned method for image interpolation based on a residual-guided strategy.

Finally, the present application provides a readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it is used to realize the image interpolation based on the residual guidance strategy as described above A model training method, and/or, an image interpolation method based on a residual-guided strategy as described above.

Each embodiment in this specification is described in a progressive manner, each embodiment focuses on the difference from other embodiments, and the same or similar parts of each embodiment can be referred to each other. As for the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and for the related information, please refer to the description of the method part.

The steps of the methods or algorithms described in connection with the embodiments disclosed herein may be directly implemented by hardware, software modules executed by a processor, or a combination of both. Software modules can be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other Any other known storage medium.

The scheme provided by the present application has been introduced in detail above, and the principle and implementation mode of the present application have been explained by using specific examples in this paper. The description of the above embodiments is only used to help understand the method and core idea of the present application; at the same time , For those of ordinary skill in the art, based on the idea of this application, there will be changes in the specific implementation and application scope. In summary, the content of this specification should not be construed as limiting the application.

Claims

A method for training an image interpolation model based on a residual guidance strategy, characterized in that it includes:

Acquiring a high-resolution image; downsampling the high-resolution image to obtain a low-resolution image; generating a pre-interpolation image according to the low-resolution image;

Making a difference between the high-resolution image and the pre-interpolation image to obtain an initial residual;

Use the pre-interpolation image and the initial residual to train the random forest; during the training process, the random forest grows synchronously by layers, and the initial residual is used as the first-level residual, in the random forest At any level, learn the mapping relationship between the pre-interpolation image and the residual of the current level to generate an estimated residual, and make a difference between the residual of the current level and the estimated residual to obtain the residual of the next level;

When the training termination condition is reached, the random forest whose mapping relationship at each level is determined is output as an image interpolation model.
The method according to claim 1, wherein the random forest is divided into multiple groups of random forests, and the random forest is trained using the pre-interpolation image and the initial residual, comprising:

generating a feature vector of the pre-interpolated image;

According to the fixed point distribution mode, the feature vectors are grouped, wherein the number of groups of the feature vectors is equal to the number of groups of the random forest;

When training the random forest, a group of random forests are trained for each set of feature vectors and corresponding residual vectors in the initial residual.
The method according to claim 2, wherein said generating the feature vector of said pre-interpolation image comprises:

Filtering the pre-interpolation image by using a one-dimensional first-order gradient operator and a second-order gradient operator to generate four corresponding feature images; sampling the four feature images to obtain a feature vector of each sampling position.
The method according to claim 3, wherein the sampling method of the four feature images is specifically: sampling at intervals with a step size of 1;

Correspondingly, there are 4 kinds of fixed point distribution patterns, and the number of groups of the feature vector and the number of groups of the random forest are 4.
The method according to claim 1, wherein the image interpolation model specifically comprises K-level random forests; the pre-interpolation image of the first-level random forest is an image generated by a preset interpolation algorithm, for any k∈ [2,K], the pre-interpolation image of the k-th random forest is the image obtained by sequential interpolation of the previous k-1 random forest.
The method according to claim 5, characterized in that, in the image interpolation model, the high-resolution images of random forests at different levels are different.
The method according to any one of claims 1 to 6, wherein the random forest grows synchronously by layers, comprising:

At any level of the random forest, judging whether there is an unprocessed target node;

If it exists, generate the first linear transformation from the feature vector contained in the target node to the residual vector contained in the target node, and then generate the second linear transformation from the feature vector contained in the target node to the target residual vector , wherein the target residual vector is a residual vector that intersects with the target node;

If it does not exist, it is judged whether the split termination condition is reached;

If it is reached, it is determined that the target node belongs to a leaf node, and the second linear transformation of the target node is recorded, and finally the second linear transformation of all leaf nodes is the difference between the pre-interpolation image and the sum of the residuals of each level Mapping relations;

If not, determine that the target node belongs to an internal node, and enter the next level through node splitting; in the process of node splitting, randomly select the splitting parameters and split the target node, and determine the optimal node according to the amount of error reduction before and after splitting optimal splitting parameter, record the optimal splitting parameter of the target node.
An image interpolation method based on a residual guidance strategy, characterized in that it comprises:

Obtain the low-resolution image to be interpolated;

generating a pre-interpolated image according to the low-resolution image;

Input the pre-interpolated image into the trained random forest; at any level of the random forest, generate an estimated residual according to the mapping relationship between the pre-interpolated image learned in the training process and the residual of the current level;

An interpolated image of the low-resolution image is generated according to the estimated residual of each level and the pre-interpolated image.
A computer device, characterized in that it includes:

memory: used to store computer programs;

Processor: used to execute the computer program, so as to realize the image interpolation model training method based on the residual guidance strategy as claimed in any one of claims 1 to 7, and/or, as claimed in claim 8. Image Interpolation Method for Difference-Guided Strategies.
A readable storage medium, characterized in that a computer program is stored on the readable storage medium, and when the computer program is executed by a processor, it is used to implement the residual-based An image interpolation model training method of a guidance strategy, and/or, an image interpolation method based on a residual guidance strategy as claimed in claim 8 .