CN115713585A - Texture image reconstruction method and device, computer equipment and storage medium - Google Patents
Texture image reconstruction method and device, computer equipment and storage medium Download PDFInfo
- Publication number
- CN115713585A CN115713585A CN202310013253.1A CN202310013253A CN115713585A CN 115713585 A CN115713585 A CN 115713585A CN 202310013253 A CN202310013253 A CN 202310013253A CN 115713585 A CN115713585 A CN 115713585A
- Authority
- CN
- China
- Prior art keywords
- texture image
- texture
- target
- image
- enhanced
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Abstract
The application relates to a texture image reconstruction method, a texture image reconstruction device, a computer device, a storage medium and a computer program product. The method comprises the following steps: respectively carrying out frequency decomposition on the target texture image and the texture image set corresponding to the target texture image to obtain a first target texture image and a second target texture image corresponding to the target texture image, and a first texture image set and a second texture image set corresponding to the texture image sets; the frequency corresponding to the first texture image set is smaller than the frequency corresponding to the second texture image set, and the frequency corresponding to the first target texture image is smaller than the frequency corresponding to the second target texture image; performing image enhancement on the first target texture image based on the first texture image set to obtain a first enhanced texture image; performing image enhancement on the second target texture image based on the second texture image set to obtain a second enhanced texture image; and fusing the first enhanced texture image and the second enhanced texture image to obtain a reconstructed texture image corresponding to the target texture image, thereby improving the quality of texture image reconstruction.
Description
Technical Field
The present application relates to the field of image processing technologies, and in particular, to a texture image reconstruction method, apparatus, computer device, storage medium, and computer program product.
Background
With the development of computer technology, people's work and life have brought about great changes. For example, original animation production or game production is usually performed based on a planar object, and now with the development of science and technology, more and more scenes supporting three-dimensional objects appear, and accordingly, the quality requirement on texture images is higher and higher.
In conventional techniques, a large number of texture image samples need to be collected to train a machine learning model for optimizing the texture images. However, the training effect of the model is closely related to the training sample, the individual difference of the texture image is large, the model with excellent characteristics is difficult to obtain through training, a universal model is difficult to obtain through training, and the problem that the reconstruction quality of the texture image is poor exists.
Disclosure of Invention
In view of the above, it is necessary to provide a texture image reconstruction method, an apparatus, a computer device, a computer readable storage medium, and a computer program product capable of improving the reconstruction quality of a texture image in view of the above technical problems.
The application provides a texture image reconstruction method. The method comprises the following steps:
acquiring a target texture image and a texture image set corresponding to the target texture image; the texture presented by the texture image in the texture image set and the texture presented by the target texture image are matched with each other, and the texture image set is obtained based on texture images with different resolutions;
respectively carrying out frequency decomposition on the texture image set and the target texture image to obtain a first texture image set and a second texture image set corresponding to the texture image set, and a first target texture image and a second target texture image corresponding to the target texture image; the frequency corresponding to the first texture image set is smaller than the frequency corresponding to the second texture image set, and the frequency corresponding to the first target texture image is smaller than the frequency corresponding to the second target texture image;
performing image enhancement on the first target texture image based on the first texture image set to obtain a first enhanced texture image corresponding to the first target texture image;
performing image enhancement on the second target texture image based on the second texture image set to obtain a second enhanced texture image corresponding to the second target texture image;
and fusing the first enhanced texture image and the second enhanced texture image to obtain a reconstructed texture image corresponding to the target texture image.
The application also provides a texture image reconstruction device. The device comprises:
the texture image acquisition module is used for acquiring a target texture image and a texture image set corresponding to the target texture image; the texture presented by the texture image in the texture image set and the texture presented by the target texture image are matched with each other, and the texture image set is obtained based on texture images with different resolutions;
an image decomposition module, configured to perform frequency decomposition on the texture image set and the target texture image, respectively, to obtain a first texture image set and a second texture image set corresponding to the texture image set, and a first target texture image and a second target texture image corresponding to the target texture image; the frequency corresponding to the first texture image set is smaller than the frequency corresponding to the second texture image set, and the frequency corresponding to the first target texture image is smaller than the frequency corresponding to the second target texture image;
a first image enhancement module, configured to perform image enhancement on the first target texture image based on the first texture image set, to obtain a first enhanced texture image corresponding to the first target texture image;
a second image enhancement module, configured to perform image enhancement on the second target texture image based on the second texture image set, to obtain a second enhanced texture image corresponding to the second target texture image;
and the image fusion module is used for fusing the first enhanced texture image and the second enhanced texture image to obtain a reconstructed texture image corresponding to the target texture image.
A computer device comprising a memory storing a computer program and a processor implementing the steps of the above texture image reconstruction method when the computer program is executed.
A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the steps of the above-mentioned texture image reconstruction method.
A computer program product comprising a computer program which, when being executed by a processor, carries out the steps of the above-mentioned texture image reconstruction method.
The texture image reconstruction method, the texture image reconstruction device, the computer equipment, the storage medium and the computer program product are used for acquiring the target texture image and a texture image set corresponding to the target texture image, wherein textures represented by the texture images in the texture image set are matched with textures represented by the target texture image, the texture image set is obtained based on texture images with different resolutions, and the texture reconstruction is carried out on the target texture image by means of the texture image set. Respectively carrying out frequency decomposition on the texture image set and the target texture image to obtain a first texture image set and a second texture image set corresponding to the texture image set, and a first target texture image and a second target texture image corresponding to the target texture image, decomposing the texture image set into a first texture image set representing low-frequency components of the image and a second texture image set representing high-frequency components of the image through frequency decomposition, and decomposing the first target texture image into a first target texture image representing the low-frequency components of the image and a second target texture image representing the high-frequency components of the image. The first target texture image is subjected to image enhancement based on the first texture image set, so that a first enhanced texture image corresponding to the first target texture image can be obtained, which is equivalent to enhancement of low-frequency components in the target texture image, such as enhancement of illumination. And performing image enhancement on the second target texture image based on the second texture image set to obtain a second enhanced texture image corresponding to the second target texture image, which is equivalent to enhancing high-frequency components in the target texture image, for example, enhancing texture details. And fusing the first enhanced texture image and the second enhanced texture image to obtain a reconstructed texture image corresponding to the target texture image. Therefore, the low-quality target texture image can be converted into the reconstructed texture image with high definition and rich detail information, the definition of the generated reconstructed texture image is enhanced while the original texture information is kept, more texture details are enhanced, and the reconstruction quality of the texture image is greatly improved.
Drawings
FIG. 1 is a diagram illustrating an exemplary embodiment of a texture image reconstruction method;
FIG. 2 is a flowchart illustrating a texture image reconstruction method according to an embodiment;
FIG. 3 is a flowchart illustrating image enhancement of a first target texture image based on a first texture image set in one embodiment;
FIG. 4 is a schematic flow diagram illustrating a process for obtaining an intermediate texture feature map based on a first texture image set and a first target texture image in one embodiment;
FIG. 5 is a flowchart illustrating attention processing on an intermediate texture feature map to obtain a target texture feature map according to an embodiment;
FIG. 6 is a flowchart illustrating image enhancement of a second target texture image based on a second texture image set in one embodiment;
FIG. 7 is a flowchart illustrating a texture image reconstructing method according to another embodiment;
FIG. 8 is a block diagram of a texture reconstruction model in an embodiment;
FIG. 9 is a block diagram illustrating an exemplary texture image reconstruction apparatus;
FIG. 10 is a diagram showing an internal structure of a computer device in one embodiment;
fig. 11 is an internal configuration diagram of a computer device in another embodiment.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.
The texture image reconstruction method provided by the embodiment of the application can be applied to the application environment shown in fig. 1. Wherein the terminal 102 communicates with the server 104 via a network. The data storage system may store data that the server 104 needs to process. The data storage system may be integrated on the server 104, or may be placed on the cloud or other server. The terminal 102 may be, but not limited to, various desktop computers, notebook computers, smart phones, tablet computers, internet of things devices and portable wearable devices, and the internet of things devices may be smart speakers, smart televisions, smart air conditioners, smart car-mounted devices, and the like. The portable wearable device can be a smart watch, a smart bracelet, a head-mounted device, and the like. The server 104 may be implemented as a stand-alone server or a server cluster composed of a plurality of servers or a cloud server.
The terminal and the server can be used independently to execute the texture image reconstruction method provided in the embodiment of the present application.
For example, the server obtains a target texture image and a texture image set corresponding to the target texture image, wherein the texture presented by the texture image in the texture image set and the texture presented by the target texture image are matched with each other, and the texture image set is obtained based on texture images with different resolutions. The server carries out frequency decomposition on the texture image set and the target texture image respectively to obtain a first texture image set and a second texture image set corresponding to the texture image set, and a first target texture image and a second target texture image corresponding to the target texture image, wherein the frequency corresponding to the first texture image set is smaller than the frequency corresponding to the second texture image set, and the frequency corresponding to the first target texture image is smaller than the frequency corresponding to the second target texture image. The server performs image enhancement on the first target texture image based on the first texture image set to obtain a first enhanced texture image corresponding to the first target texture image, performs image enhancement on the second target texture image based on the second texture image set to obtain a second enhanced texture image corresponding to the second target texture image, and fuses the first enhanced texture image and the second enhanced texture image to obtain a reconstructed texture image corresponding to the target texture image.
The terminal and the server can also be cooperatively used for executing the texture image reconstruction method provided in the embodiment of the application.
For example, the server acquires a target texture image matched with the terminal, and acquires a texture image set corresponding to the target texture image. The server carries out frequency decomposition on the texture image set and the target texture image respectively to obtain a first texture image set and a second texture image set corresponding to the texture image set, and a first target texture image and a second target texture image corresponding to the target texture image. The server performs image enhancement on the first target texture image based on the first texture image set to obtain a first enhanced texture image corresponding to the first target texture image, performs image enhancement on the second target texture image based on the second texture image set to obtain a second enhanced texture image corresponding to the second target texture image, and fuses the first enhanced texture image and the second enhanced texture image to obtain a reconstructed texture image corresponding to the target texture image. And the server sends the reconstructed texture image to the terminal. The terminal can render and display the reconstructed texture image.
In one embodiment, as shown in fig. 2, a texture image reconstruction method is provided, which is applied to a computer device for example, the computer device may be a terminal or a server, and the method is executed by the terminal or the server itself, or may be implemented through interaction between the terminal and the server. Referring to fig. 2, the texture image reconstruction method includes the steps of:
step S202, obtaining a target texture image and a texture image set corresponding to the target texture image; the texture presented by the texture image in the texture image set and the texture presented by the target texture image are matched with each other, and the texture image set is obtained based on the texture images with different resolutions.
Wherein the texture image is an image used to characterize the surface of the object. Texture images, which may also be referred to as texture maps, enable objects to appear more realistic when the texture is mapped onto the surface of the object in a particular way. In a three-dimensional scene, the texture image, also called UV image, is a three-dimensional unfolded surface image. The UV is the abbreviation of UV texture mapping coordinates, and defines the information of the position of each point on the image, U and V are the coordinates of the image in the horizontal and vertical directions of the display, and the value is generally 0-1. Each point in the UV image is associated with the three-dimensional model and can determine the position of the surface texture map, i.e. each point in the UV image can correspond exactly to the surface of the model object to construct a stereo object. For example, a face texture image may be used to generate a three-dimensional face; the hair texture image can be used to generate three-dimensional hair; and so on.
The target texture image refers to a texture image to be enhanced and reconstructed. The target texture image may be an arbitrary texture image. The texture presented by the texture image in the texture image set and the texture presented by the target texture image are matched with each other, that is, the texture image in the texture image set and the target texture image are correspondingly matched with each other. The textures presented by the different texture images are matched with each other, which means that the image similarity between the different texture images is greater than the preset similarity. In one embodiment, the mutually matching textures refer to the same texture, i.e., the texture image set to which the target texture image corresponds includes texture images having the same texture as the target texture image. The texture image set corresponding to the target texture image is obtained based on texture images with different resolutions. For matching textures (e.g., the same texture), different resolution texture images are adapted to different devices, and the different resolution texture images are used for rendering presentation on different devices. In one embodiment, in order to facilitate subsequent data processing, a plurality of initial texture images having matching textures with a target texture image may be obtained, different initial texture images correspond to different resolutions, the resolutions of the initial texture images are increased to the same resolution to obtain updated texture images, and the updated texture images are combined into a texture image set.
It is understood that the texture image set may be obtained first, and then any one texture image may be selected from the texture image set as the target texture image. Or the target texture image may be obtained first, and then the texture image set corresponding to the target texture image may be obtained.
Specifically, the computer device may obtain the target texture image and a texture image set corresponding to the target texture image locally or from another device through a network, and perform texture reconstruction on the target texture image based on the texture image set corresponding to the target texture image, so as to convert the target texture image into a clearer reconstructed texture image with more details.
In one embodiment, the target texture image may be a texture image corresponding to the virtual object. The virtual object is an object which can be stored in a computer device and is realized by data, and the virtual object specifically may include at least one of a virtual character, a virtual animal, a virtual plant, a virtual object, and the like. The computer device may construct in advance a library of object texture images including various texture images required for the virtual object. The computer device may obtain at least one texture image from an object texture image library corresponding to the virtual object as a target texture image, and perform texture reconstruction on the target texture image based on a texture image set corresponding to the target texture image to obtain a reconstructed texture image corresponding to the target texture image. When a certain virtual object is displayed, the display effect of the virtual object can be effectively enhanced by loading at least one reconstructed texture image corresponding to the virtual object, and the display quality is improved.
In one embodiment, the target texture image may be a texture image corresponding to a virtual object in the game. The computer device may pre-construct a library of game texture images that includes various texture images required for virtual objects in the game. The computer device can obtain at least one texture image from the game texture image library as a target texture image, and carries out texture reconstruction on the target texture image based on a texture image set corresponding to the target texture image to obtain a reconstructed texture image corresponding to the target texture image. When any game is started, the display effect of the game picture can be effectively enhanced by loading at least one reconstructed texture image required by the game, and the display quality is improved.
In one embodiment, the target texture image may be an abnormal quality texture image. The texture image may be quality evaluated to determine the image quality of the texture image. Texture reconstruction may not be required for texture images that meet quality requirements, e.g., high quality texture images. For texture images which do not meet the quality requirement, for example, texture images with insufficient illumination, the texture reconstruction method of the present application may be adopted to improve the image quality.
Step S204, respectively carrying out frequency decomposition on the texture image set and the target texture image to obtain a first texture image set and a second texture image set corresponding to the texture image set, and a first target texture image and a second target texture image corresponding to the target texture image; the frequency corresponding to the first texture image set is smaller than the frequency corresponding to the second texture image set, and the frequency corresponding to the first target texture image is smaller than the frequency corresponding to the second target texture image.
Wherein the frequency decomposition is used to decompose the image into a high frequency part and a low frequency part. The high-frequency part of the image may also be referred to as a high-frequency component of the image, and refers to a place where the intensity (brightness or gray scale) of the image changes drastically, for example, an edge, a contour, or the like of the image. The high-frequency components of the image are used for representing detail information and local information of the image. The low-frequency part of the image may also be referred to as a low-frequency component of the image, and refers to a place where the intensity (brightness or gray scale) of the image changes smoothly, for example, a dark area with low light in the image, a background, and the like. The low-frequency components of the image are used for representing the overall information and the global information of the image.
And performing frequency decomposition on the texture image set to obtain a first texture image set and a second texture image set, wherein the frequency corresponding to the first texture image set is less than the frequency corresponding to the second texture image set, that is, the first texture image set corresponds to the low-frequency part of the texture image set, and the second texture image set corresponds to the high-frequency part of the texture image set. The texture image set comprises a plurality of texture images, the first texture image set comprises first texture images corresponding to the texture images in the texture image set respectively, and the second texture image set comprises second texture images corresponding to the texture images in the texture image set respectively.
And performing frequency decomposition on the target texture image to obtain a first target texture image and a second target texture image, wherein the frequency corresponding to the first target texture image is less than the frequency corresponding to the second target texture image, namely, the first target texture image corresponds to the low-frequency part of the target texture image, and the second target texture image corresponds to the high-frequency part of the target texture image.
Specifically, the computer device performs frequency decomposition on the texture image set to obtain a first texture image set and a second texture image set corresponding to the texture image set, wherein the frequency corresponding to the first texture image set is smaller than the frequency corresponding to the second texture image set. And the computer equipment carries out frequency decomposition on the target texture image to obtain a first target texture image and a second target texture image corresponding to the target texture image, wherein the frequency corresponding to the first target texture image is less than the frequency corresponding to the second target texture image.
It will be appreciated that there are many ways to separate the high frequency components and the low frequency components from the image, for example, using filters.
Step S206, the first target texture image is subjected to image enhancement based on the first texture image set, and a first enhanced texture image corresponding to the first target texture image is obtained.
The image enhancement of the first target texture image refers to enhancing the low-frequency component of the target texture image to achieve the effect of enhancing the global information of the target texture image, for example, enhancing the illumination of the image. The first texture image set comprises low-frequency components of texture images of corresponding matching textures of the target texture image, the original resolutions of the texture images are different, therefore, the contained low-frequency information is different, the low-frequency components contain more information than the low-frequency components of the target texture image, and the low-frequency components of the target texture image can be supplemented. And performing image enhancement on the first target texture image based on the first texture image set, and supplementing low-frequency components of the target texture image by using the low-frequency components of the texture images with various resolutions so as to obtain a first enhanced texture image. The first enhanced texture image refers to the first target texture image after image enhancement.
Specifically, in consideration of the fact that image information reflected by the low frequency component and the high frequency component of the image are different, in order to guarantee texture reconstruction quality, the low frequency component and the high frequency component may be enhanced separately. And aiming at the low-frequency component, the computer equipment performs image enhancement on the first target texture image based on the first texture image set to obtain a first enhanced texture image corresponding to the first target texture image.
In one embodiment, the computer device performs convolution processing on the first texture image set and the first target texture image respectively to obtain initial texture feature maps corresponding to the first texture image set and the first target texture image respectively, and obtains initial texture feature maps corresponding to the first texture image set and the first target texture image respectively based on the initial texture feature maps corresponding to the first texture image set and the first target texture image respectively.
And step S208, performing image enhancement on the second target texture image based on the second texture image set to obtain a second enhanced texture image corresponding to the second target texture image.
The image enhancement of the second target texture image refers to enhancing high-frequency components of the target texture image so as to achieve the effect of enhancing local information of the target texture image, for example, enhancing details of the image. The second texture image set comprises high-frequency components of texture images of corresponding matching textures of the target texture image, the original resolutions of the texture images are different, so that the contained high-frequency information is different, the high-frequency components contain more information than the high-frequency components of the target texture image, and the high-frequency components of the target texture image can be supplemented. And performing image enhancement on the second target texture image based on the second texture image set, and supplementing high-frequency components of the target texture image by using the high-frequency components of the texture images with various resolutions so as to obtain a second enhanced texture image. The second enhanced texture image refers to a second target texture image after image enhancement.
Specifically, for the high-frequency component, the computer device performs image enhancement on the second target texture image based on the second texture image set to obtain a second enhanced texture image corresponding to the second target texture image.
In one embodiment, the high frequency components may reflect some noise information of the image in addition to the contour information of the image, and in order to better enhance the contour information, the high frequency components may be further enhanced by low frequency components, which help to assist in determining the contour information of the image. Therefore, the computer device performs image enhancement on the second target texture image based on the second texture image set and the reference texture image to obtain a second enhanced texture image corresponding to the second target texture image, wherein the reference texture image includes at least one of the first target texture image or the first enhanced texture image. In one embodiment, the average texture image is obtained by performing an average calculation on the second texture image set and the second target texture image. Then, a mask texture image is obtained based on the average texture image and the reference texture image, for example, the reference texture image and the average texture image are compared in terms of pixel values, the pixel value of a pixel point with a smaller pixel value in the average texture image is set to 1, and the pixel value of a pixel point with a larger pixel value or the same pixel value in the average texture image is set to 0, so that a binary mask texture image is obtained. And finally, fusing the mask texture image and the average texture image to obtain a second enhanced texture image. In one embodiment, deriving the mask texture image based on the average texture image and the reference texture image comprises: and splicing the average texture image and the reference texture image to obtain a spliced texture image, and performing residual error processing on the spliced texture image to obtain a mask texture image.
And S210, fusing the first enhanced texture image and the second enhanced texture image to obtain a reconstructed texture image corresponding to the target texture image.
Specifically, the computer device fuses a first enhanced texture image and a second enhanced texture image obtained through image enhancement to obtain a reconstructed texture image corresponding to the target texture image. For example, the first enhanced texture image and the second enhanced texture image are added to obtain a reconstructed texture image. Compared with the target texture image, the reconstructed texture image has richer high-frequency information and low-frequency information on the basis of the same texture corresponding to the target texture image. Subsequently, the reconstructed texture image may be used for displaying, for example, the original display target texture image may be changed to display the reconstructed texture image, so as to improve the display effect.
In the texture image reconstruction method, the target texture image and a texture image set corresponding to the target texture image are obtained, the texture presented by the texture image in the texture image set is matched with the texture presented by the target texture image, the texture image set is obtained based on texture images with different resolutions, and the texture reconstruction is performed on the target texture image by means of the texture image set. Respectively carrying out frequency decomposition on the texture image set and the target texture image to obtain a first texture image set and a second texture image set corresponding to the texture image set, and a first target texture image and a second target texture image corresponding to the target texture image, decomposing the texture image set into a first texture image set representing low-frequency components of the image and a second texture image set representing high-frequency components of the image through frequency decomposition, and decomposing the first target texture image into a first target texture image representing the low-frequency components of the image and a second target texture image representing the high-frequency components of the image. The first target texture image is subjected to image enhancement based on the first texture image set, so that a first enhanced texture image corresponding to the first target texture image can be obtained, which is equivalent to enhancement of low-frequency components in the target texture image, such as enhancement of illumination. And performing image enhancement on the second target texture image based on the second texture image set to obtain a second enhanced texture image corresponding to the second target texture image, which is equivalent to enhancing high-frequency components in the target texture image, for example, enhancing texture details. And fusing the first enhanced texture image and the second enhanced texture image to obtain a reconstructed texture image corresponding to the target texture image. Therefore, the low-quality target texture image can be converted into the reconstructed texture image with high definition and rich detail information, the definition of the generated reconstructed texture image is enhanced while the original texture information is kept, more texture details are enhanced, and the reconstruction quality of the texture image is greatly improved.
In one embodiment, frequency decomposition is performed on the texture image set and the target texture image respectively to obtain a first texture image set and a second texture image set corresponding to the texture image set, and a first target texture image and a second target texture image corresponding to the target texture image, including:
performing Gaussian decomposition on the texture image set to obtain a first texture image set and a second texture image set corresponding to the texture image set; and carrying out Gaussian decomposition on the target texture image to obtain a first target texture image and a second target texture image corresponding to the target texture image.
The gaussian decomposition refers to a frequency decomposition mode realized based on gaussian filtering. And performing Gaussian filtering on the texture image to obtain a first texture image representing the low-frequency component of the texture image, and taking the difference value between the texture image and the first texture image as a second texture image representing the high-frequency component of the texture image. The gaussian filtering is a process of performing weighted average on the whole image, and the value of each pixel point is obtained by performing weighted average on the value of each pixel point and other pixel values in the neighborhood. The Gaussian filtering is to use each pixel in a Gaussian kernel scanning image and use the weighted average gray value of the pixels in the neighborhood determined by the Gaussian kernel to replace the value of the central pixel point determined by the Gaussian kernel. The gaussian kernel is obtained by sampling a gaussian function, and the gaussian function refers to a probability density function of gaussian distribution. Gaussian filtering is equivalent to a low-pass filter, and after weighted averaging, high-frequency components with severe changes, such as edges, stripes, noise points, etc., will become smooth (i.e., "suppressed"), while the original smooth place where weighted averaging is performed will still be smooth and have little change (i.e., "passed").
Specifically, the computer device performs a gaussian decomposition on the texture image set, resulting in a first texture image set representing high frequency components of the texture image set and a second texture image set representing low frequency components of the texture image set. The computer device performs Gaussian decomposition on the target texture image to obtain a first target texture image representing high frequency components of the target texture image and a second target texture image representing low frequency components of the target texture image.
In the above embodiment, the high-frequency component and the low-frequency component in the image can be decomposed quickly by gaussian decomposition. The low-frequency component and the high-frequency component of the image reflect different image information, and the low-frequency component and the high-frequency component of the target texture image are subsequently and respectively subjected to image enhancement, so that the texture reconstruction quality can be improved. The low-frequency component of the texture image set contains more information, the low-frequency component of the target texture image is subjected to image enhancement based on the low-frequency component of the texture image set so as to improve the definition, the high-frequency component of the texture image set contains more information, the high-frequency component of the target texture image is subjected to image enhancement based on the high-frequency component of the texture image set so as to increase the texture details, and the enhanced low-frequency component and the enhanced high-frequency component are fused so as to obtain a reconstructed texture image with high definition and rich detail information.
In one embodiment, as shown in fig. 3, performing image enhancement on the first target texture image based on the first texture image set to obtain a first enhanced texture image corresponding to the first target texture image, includes:
step S302, performing convolution processing on the first texture image set and the first target texture image, respectively, to obtain initial texture feature maps corresponding to the first texture image set and the first target texture image, respectively.
Step S304, the initial texture feature images corresponding to the first texture image set and the first target texture image are spliced to obtain an intermediate texture feature image.
And step S306, performing attention processing on the intermediate texture feature map to obtain a target texture feature map.
Step S308, based on the target texture feature map and the first target texture image, a first enhanced texture image corresponding to the target texture image is obtained.
Wherein the convolution process is used to extract image features. The process of performing convolution processing on the image is a process of sliding on the image by using a convolution kernel, wherein the convolution kernel is sequentially convolved with image blocks at corresponding positions on the image. The convolution of the convolution kernel and the image block at the corresponding position on the image means that weighted summation is carried out on pixel values of pixel points in the image block, wherein the weight of the weighted summation is determined by the convolution kernel, that is, the pixel values of the pixel points in the image block are multiplied by the numerical value of the corresponding position in the convolution kernel, and then all the multiplied values are added to be used as the convolution result of the image pixel point corresponding to the middle pixel point of the convolution kernel.
The stitching process is mainly used for stitching images. The splicing treatment can be directly splicing different images, or splicing different images after carrying out some preprocessing.
Attention processing is used to further extract image features to filter extraneous information to focus, enhance accent information, e.g., remove noise in the image, illuminate dark areas in the image. Attention processing may be implemented using various attention mechanisms applied to the image, for example, using a channel attention mechanism.
Specifically, when the low-frequency component is subjected to image enhancement, the computer device performs convolution processing on the first texture image set and the first target texture image respectively to obtain initial texture feature maps corresponding to the first texture image set and the first target texture image respectively, and spectral features of the images can be extracted through the convolution processing. And then, the computer equipment splices the initial texture feature maps corresponding to the first texture image set and the first target texture image respectively to obtain an intermediate texture feature map, and spectral features of different images can be integrated through splicing. Furthermore, the computer equipment carries out attention processing on the intermediate texture feature map to obtain a target texture feature map, and spectral features of image depth can be extracted and obtained through the attention processing. And finally, the computer equipment obtains a first enhanced texture image corresponding to the target texture image based on the target texture feature image and the first target texture image. The target texture feature map contains frequency information that is lacking in the first target texture image, and may supplement the frequency information of the first target texture image.
In the above embodiment, the first texture image set and the first target texture image are respectively subjected to convolution processing to obtain initial texture feature maps corresponding to the first texture image set and the first target texture image, the initial texture feature maps corresponding to the first texture image set and the first target texture image are respectively subjected to stitching processing to obtain an intermediate texture feature map, the intermediate texture feature map is subjected to attention processing to obtain a target texture feature map, and a first enhanced texture image corresponding to the target texture image is obtained based on the target texture feature map and the first target texture image. The convolution processing is used for extracting the spectral features of the image, the attention processing is used for enhancing the low-frequency information of the image, and the first enhanced texture image obtained through the processing has more low-frequency information than the first target texture image and has better quality than the first target texture image, for example, the illumination in the first target texture image is recovered, and the first enhanced texture image is helpful for improving the reconstruction quality of the texture image.
In one embodiment, performing convolution processing on the first texture image set and the first target texture image respectively to obtain initial texture feature maps corresponding to the first texture image set and the first target texture image respectively includes:
performing convolution processing on the first texture image set based on at least two first convolution kernels to obtain at least two first convolution feature maps, and splicing the at least two first convolution feature maps to obtain an initial texture feature map corresponding to the first texture image set; the at least two first convolution cores comprise at least two sizes of first convolution cores; performing convolution processing on the first target texture image based on at least two second convolution kernels to obtain at least two second convolution characteristic graphs, and splicing the at least two second convolution characteristic graphs to obtain an initial texture characteristic graph corresponding to the first target texture image; the at least two second convolution kernels include at least two sizes of second convolution kernels.
And the data in the convolution kernel is used for determining the weight value in the weighted summation. The size of the convolution kernel is used to determine the area of pixels that need to be weighted and summed, i.e., to determine the image block size. The convolution kernels with different sizes are used for extracting the spectral features with different scales, and the spectral features with different sizes correspond to the image space information with different scales.
The first convolution kernel is used for performing convolution processing on the first texture image set, and the second convolution kernel is used for performing convolution processing on the first target texture image. In one embodiment, the first convolution kernel and the second convolution kernel may be the same convolution kernel.
Specifically, the process of performing convolution processing on the first texture image set and the first target texture image is the same. The computer equipment obtains first convolution kernels with at least two sizes, convolution processing is carried out on the first texture image set based on each first convolution kernel to obtain different first convolution feature maps, and finally, each first convolution feature map is spliced to obtain an initial texture feature map corresponding to the first texture image set. And the computer equipment acquires second convolution kernels with at least two sizes, performs convolution processing on the first target texture image based on the second convolution kernels to obtain different second convolution characteristic diagrams, and finally splices the second convolution characteristic diagrams to obtain an initial texture characteristic diagram corresponding to the first target texture image.
In the above embodiment, multi-scale feature information can be extracted through convolution kernels of different sizes, and the multi-scale feature information is helpful for improving the accuracy of subsequent data processing. And performing convolution processing on the first texture image set through the first convolution cores with different sizes, so that the multi-scale joint spatial spectrum can be extracted. The first target texture image is checked by a second convolution kernel of different sizes, enabling extraction of multi-scale spatial and spectral features.
In one embodiment, the process of stitching the initial texture feature maps corresponding to the first texture image set and the first target texture image, respectively, to obtain an intermediate texture feature map includes:
splicing the initial texture feature maps corresponding to the first texture image set and the first target texture image respectively to obtain a first texture feature map, and rectifying the first texture feature map to obtain a second texture feature map; and performing convolution processing on the second texture feature map to obtain a third texture feature map, performing up-sampling processing on the third texture feature map to obtain a fourth texture feature map, and performing rectification processing on the fourth texture feature map to obtain an intermediate texture feature map.
The rectification processing is used for correcting the pixel value and mapping the pixel value into a preset range. The rectification process may be performed by an activation function, for example, by a RELU (Rectified Linear Unit).
The upsampling process is used to increase the resolution of the image, i.e., to convert the image from a smaller size to a larger size. Specifically, the size of the original image may be enlarged, so that a plurality of areas to be supplemented are left, and then a pixel value corresponding to the area to be supplemented is calculated through a certain interpolation algorithm, thereby realizing the enlargement of the image. For example, a pixel value corresponding to a region to be supplemented is calculated through a bilinear interpolation algorithm; calculating a pixel value corresponding to a region to be supplemented through a nearest neighbor interpolation algorithm; and so on.
Specifically, when the initial texture feature maps corresponding to the first texture image set and the first target texture image are spliced, the computer device firstly splices the initial texture feature maps corresponding to the first texture image set and the first target texture image respectively to obtain a first texture feature map, then carries out rectification processing on the first texture feature map to obtain a second texture feature map, and can normalize the pixel value of the feature maps through rectification processing so as to avoid overlarge pixel value difference between pixels. And then, carrying out convolution processing on the second texture feature map by the computer equipment to obtain a third texture feature map, and further extracting feature information of the image through further convolution processing. Furthermore, the computer device performs upsampling processing on the third texture feature map, and converts the third texture feature map into a fourth texture feature map with higher resolution, and the upsampling processing is beneficial to facilitating subsequent attention processing. And finally, rectifying the fourth texture feature map by the computer equipment to obtain an intermediate texture feature map, wherein new pixel values can be introduced due to the up-sampling treatment, and the pixel values of the feature map can be normalized again through the rectifying treatment, so that the overlarge pixel value difference between pixel points is avoided.
In the above embodiment, the initial texture feature maps corresponding to the first texture image set and the first target texture image are respectively stitched to obtain a first texture feature map, the first texture feature map is rectified to obtain a second texture feature map, the second texture feature map is convolved to obtain a third texture feature map, the third texture feature map is up-sampled to obtain a fourth texture feature map, and the fourth texture feature map is rectified to obtain an intermediate texture feature map. The rectification processing is used for standardizing the pixel value size of the characteristic graph, the convolution processing is used for further extracting characteristic information, and the characteristic expression capability of the intermediate texture characteristic graph obtained through the processing is stronger, so that the accuracy of subsequent data processing is improved.
In one embodiment, the calculation process for obtaining the intermediate texture feature map based on the first texture image set and the first target texture image is as follows:
F SL =[H L3 (I L ),H L5 (I L ),H L7 (I L )]
F SC =[H C3 (C L ),H C5 (C L ),H C7 (C L )]
F S =ReLu([F SL , F SC ])
F 0 =ReLu(H conv (F S ))
wherein H L3 ，H L5 And H L7 Representing 2D convolution kernels having convolution kernel sizes of 3 x 3,5 x 5, and 7 x 7, respectively. I.C. A L Representing a first target texture image. F SL And representing an initial texture feature map corresponding to the first target texture image. H C3 ，H C5 And H C7 Representing 2D convolution kernels having convolution kernel sizes of 3 × 3,5 × 5, and 7 × 7, respectively. C L Representing a first set of texture maps. F SC An initial texture feature map corresponding to the first texture image set is represented.
[]Indicating splicing, e.g. [ F ] SL , F SC ]Representation splicing F SL And F SC . ReLu denotes the ReLu function. H conv Indicating that one 3 x 3 convolution is used and 1 x 1 pixel is filled to enlarge the spatial resolution of the feature map. F 0 Indicating what is obtained by final treatmentAnd (5) intermediate texture feature maps.
I 0 Representing the target texture image, C 0 Representing the texture image set to which the target texture image corresponds. Referring to FIG. 4, II 0 Of the low-frequency component I L Inputting a first sub-module which consists of three convolution kernels with different sizes and is used for 2D convolution to extract multi-scale space information, simultaneously executing the three convolution, and connecting the extracted feature maps to form a feature map F SL . Will C 0 Low frequency component C of L Inputting a second sub-module, wherein the second sub-module is also composed of three convolution kernels with different sizes and is used for 2D convolution to extract multi-scale joint space information, the three convolutions are executed simultaneously, and the extracted feature maps are connected to form a feature map F SC 。F SL And F SC Are consistent in size. The characteristic diagram F is obtained by connecting the outputs of the two sub-modules and then carrying out rectification processing S . Feature map F S Further extracting channel characteristics from the input convolution layer, and rectifying to obtain a characteristic diagram F 0 。
In one embodiment, the attention processing is performed on the intermediate texture feature map to obtain a target texture feature map, and the method includes:
sequentially performing at least two times of ordered attention processing on the intermediate texture feature map to obtain at least two ordered attention texture feature maps; splicing at least two ordered attention texture feature maps to obtain a first spliced texture feature map, and performing convolution processing on the first spliced texture feature map to obtain a convolution texture feature map; acquiring an ending attention texture feature map from at least two ordered attention texture feature maps, and fusing the ending attention texture feature map and the convolution texture feature map to obtain a fused texture feature map; and splicing the at least two ordered attention texture feature maps and the fusion texture feature map to obtain a second spliced texture feature map, and performing convolution processing on the second spliced texture feature map to obtain a target texture feature map.
The performing of the ordered attention processing on the intermediate texture feature map means performing continuous attention processing on the intermediate texture feature map, for example, performing a first attention processing on the intermediate texture feature map to obtain a first attention texture feature map, performing a second attention processing on the first attention texture feature map to obtain a second attention texture feature map, and performing a third attention processing on the second attention texture feature map to obtain a third attention texture feature. The number of attention treatments can be set according to actual needs.
The final attention texture map refers to the attention texture map obtained through the last attention processing in at least two ordered attention texture maps. For example, if the number of times of attention processing is four, the attention texture map obtained by the fourth attention processing is used as the end attention texture map.
Specifically, the computer device sequentially performs at least two times of ordered attention processing on the intermediate texture feature map to obtain at least two ordered attention texture feature maps, wherein the initial attention processing can extract shallow feature information to obtain a primary attention texture feature map, and the subsequent attention processing can extract deep feature information to obtain a high-level attention texture feature map. And then, the computer equipment splices at least two ordered attention texture feature maps to obtain a first spliced texture feature map, performs convolution processing on the first spliced texture feature map to obtain a convolution texture feature map, and can reduce the resolution of the first spliced texture feature map through the convolution processing. And the computer equipment acquires an ending attention texture feature map from at least two ordered attention texture feature maps, fuses the ending attention texture feature map and the convolution texture feature map to obtain a fused texture feature map, and the fused texture feature map has stronger feature expression capability. And finally, the computer equipment splices the at least two ordered attention texture feature maps and the fusion texture feature map to obtain a second spliced texture feature map, and performs convolution processing on the second spliced texture feature map to obtain a target texture feature map. The information contained in the attention texture feature maps of the various levels can be integrated by the convolution process.
In the foregoing embodiment, the intermediate texture feature map is sequentially subjected to at least two times of ordered attention processing to obtain at least two ordered attention texture feature maps, the at least two ordered attention texture feature maps are spliced to obtain a first spliced texture feature map, the first spliced texture feature map is subjected to convolution processing to obtain a convolution texture feature map, an ending attention texture feature map is obtained from the at least two ordered attention texture feature maps, the ending attention texture feature map and the convolution texture feature map are fused to obtain a fused texture feature map, the at least two ordered attention texture feature maps and the fused texture feature map are spliced to obtain a second spliced texture feature map, and the second spliced texture feature map is subjected to convolution processing to obtain the target texture feature map. The continuous attention processing can gradually extract deep feature information, enhance low-frequency information and remove noise in the image, the convolution processing can further extract the feature information, and the feature expression capability of the target texture feature map obtained through the data processing is higher, so that the accuracy of subsequent data processing is improved.
In one embodiment, the calculation formula for performing attention processing on the intermediate texture feature map to obtain the target texture feature map is as follows:
F i =H TAB,i (F i-1 )=H TAB,i (H TAB,i-1 (⋯(H TAB,1 (F 0 ))⋯))
F n,F =H t ([F n-1 ,F n-2 ,⋯,F n-C ,⋯,F 1 ])
F n = F n,F +F n−1
F D =H DF ([F 0 ,F 1 ,⋯,F N ])
wherein, F i And an attention texture feature map representing the output of the ith channel attention processing. H TAB,i Represents the process of the ith channel attention process, which consists of several standard operations, namely 2D convolution, global average pooling, and activation functions. [ F ] n-1 ,F n-2 ,⋯,F n-C ,⋯,F 1 ]Is represented by F n-1 ,F n-2 ,⋯,F n-C ,⋯,F 1 Series connection of characteristic diagrams. Ht denotes a convolution operation for reducing the dimensionality of the feature map.H DF Representing a convolution operation for integrating features at different levels. F n And showing the target texture feature graph obtained by final processing.
Referring to FIG. 5, the intermediate texture feature map is processed n-1 times by the standard channel attention process, and each time the channel attention process is performed, the corresponding attention texture feature map is output, i.e. output F n-1 ,F n-2 ,⋯,F n-C ,⋯,F 1 。
The nth special channel attention processing is performed based on the output of the 1 st to n-1 th channel attention processing, and the output F n . In particular, after depth features are extracted with successive channel attention processing, the output of the 1 st to n-1 st channel attention processing, i.e., [ F ] is integrated using a feature fusion function n-1 ,F n-2 ,⋯,F n-C ,⋯,F 1 ]Further, the dimensionality of the connected feature map, H, is reduced by transition layers composed of a single convolutional layer having the size of the convolutional kernel t ([F n-1 ,F n-2 ,⋯,F n-C ,⋯,F 1 ]). This transition layer reduces the computational burden on the entire network, making the network easy to use. To further enhance the model representation capability, an intra block residual learning strategy is applied, F n,F And F n−1 Fusion to give F n 。
Finally, the 1 st to n th channel attention processing outputs are connected in series, and feature information of different levels is integrated through convolution operation, so that a target texture feature map, namely F, is obtained D =H DF ([F 0 ,F 1 ,⋯,F N ])。
In one embodiment, obtaining a first enhanced texture image corresponding to the target texture image based on the target texture feature map and the first target texture image includes:
performing convolution processing on the target texture feature map to obtain a supplementary texture feature map; and fusing the supplementary texture feature image and the first target texture image to obtain a first enhanced texture image corresponding to the target texture image.
Specifically, the computer device performs convolution processing on the target texture feature map to obtain a complementary texture feature map, and the convolution processing can reconstruct low-frequency components to supplement some frequency information for the first target texture image. And then, fusing the supplementary texture feature image and the first target texture image to obtain a first enhanced texture image corresponding to the target texture image.
In the above embodiment, the convolution processing is performed on the target texture feature map to obtain a supplementary texture feature map, and the supplementary texture feature map and the first target texture image are fused to obtain a first enhanced texture image corresponding to the target texture image. The frequency information of the first target texture image missing can be recovered through convolution processing, the supplementary texture feature image and the first target texture image are fused, and a first enhanced texture image containing rich information can be obtained.
In one embodiment, the calculation formula for obtaining the first enhanced texture image based on the target texture feature map and the first target texture image is as follows:
I R =H R (F D )
wherein, F D Representing the target texture feature map, H R Representing convolution operations, I R Representing the restored residual, i.e. the complementary texture map. I.C. A L Representing a first target texture image and a second target texture image,
representing a first enhanced texture image, i.e. a first target texture image enhanced by low frequency information.
In one embodiment, referring to fig. 6, performing image enhancement on the second target texture image based on the second texture image set to obtain a second enhanced texture image corresponding to the second target texture image, includes:
step S602, performing average processing on the second texture image set and the second target texture image to obtain an average texture image.
Step S604, the average texture image, the first target texture image and the first enhanced texture image are spliced to obtain a spliced texture image.
Step S606, residual error processing is carried out on the spliced texture image to obtain a mask texture image.
Step S608, the mask texture image and the average texture image are fused to obtain a second enhanced texture image corresponding to the target texture image.
The averaging processing refers to mean value calculation and is used for calculating pixel mean values among pixel points at the same position of different images. The stitching process is mainly used for stitching images. The splicing treatment can be directly splicing different images, or splicing different images after carrying out some preprocessing.
The residual processing is data processing implemented by a residual network. The mask texture image can be obtained by residual processing. The mask texture image is a binarized image used to determine contour information in the image.
Specifically, the computer device may perform mean value calculation on the second texture image set and the second target texture image to obtain an average texture image, and perform stitching processing on the average texture image, the first target texture image, and the first enhanced texture image to obtain a stitched texture image. And then, the computer equipment performs residual error processing on the spliced texture image to obtain a mask texture image, wherein the residual error processing is used for performing frequency supplementation on the average texture image, the first target texture image and the first enhanced texture image in the spliced texture image, and then performing pixel value comparison on the first target texture image after the frequency supplementation, the first enhanced texture image and the average texture image after the frequency supplementation to obtain the mask texture image. And finally, fusing the mask texture image and the average texture image by the computer equipment to obtain a second enhanced texture image corresponding to the target texture image. The mask texture image is a binary image, the average texture image and the mask texture image are fused, the pixel values of the pixel points reflecting the contour information in the average texture image can be reserved, and the pixel values of the pixel points reflecting other information are set to be zero.
In the above embodiment, the second texture image set and the second target texture image are averaged to obtain an average texture image, the first target texture image and the first enhanced texture image are spliced to obtain a spliced texture image, the spliced texture image is subjected to residual error processing to obtain a mask texture image, and the mask texture image and the average texture image are fused to obtain a second enhanced texture image corresponding to the target texture image. The average processing is beneficial to thinning the second target texture image and increasing frequency information, the residual processing is beneficial to recovering the missing frequency information of the second target texture image, and the second enhanced texture image obtained through the processing contains more accurate contour information and is beneficial to improving the quality of texture image reconstruction.
In one embodiment, the obtaining the stitched texture image by stitching the average texture image, the first target texture image, and the first enhanced texture image includes:
respectively carrying out up-sampling processing on the first target texture image and the first enhanced texture image to obtain a first up-sampling texture image corresponding to the first target texture image and a second up-sampling texture image corresponding to the first enhanced texture image; the resolutions of the first up-sampling texture image, the second up-sampling texture image and the average texture image are consistent; and splicing the average texture image, the first up-sampling texture image and the second up-sampling texture image to obtain a spliced texture image.
Specifically, the first target texture image and the first enhanced texture image represent low-frequency components of an image, and the resolution of the low-frequency components obtained through frequency decomposition is usually reduced, so that in order to facilitate stitching, upsampling processing may be performed first, and the resolutions of the images to be stitched are unified, and then stitching is performed.
When the average texture image, the first target texture image and the first enhanced texture image are spliced, the computer device respectively performs up-sampling processing on the first target texture image and the first enhanced texture image, converts the resolution of the first target texture image and the first enhanced texture image to be consistent with the resolution of the average texture image, and obtains a first up-sampling texture image corresponding to the first target texture image and a second up-sampling texture image corresponding to the first enhanced texture image. And then, the computer equipment splices the average texture image, the first up-sampling texture image and the second up-sampling texture image to obtain a spliced texture image.
In the above embodiment, the first target texture image and the first enhanced texture image are respectively subjected to upsampling processing to obtain a first upsampled texture image corresponding to the first target texture image and a second upsampled texture image corresponding to the first enhanced texture image, the resolutions of the first upsampled texture image, the second upsampled texture image and the average texture image are the same, and the average texture image, the first upsampled texture image and the second upsampled texture image are spliced to obtain a spliced texture image. The resolution ratios of the images are unified and then spliced, so that the accuracy and the efficiency of splicing are improved.
In one embodiment, the fusing the mask texture image and the average texture image to obtain a second enhanced texture image corresponding to the target texture image includes:
and performing pixel-by-pixel fusion on the mask texture image and the average texture image to obtain a second enhanced texture image corresponding to the target texture image.
The pixel-by-pixel fusion refers to the fusion of pixel values of pixel points at the same position of different images.
Specifically, the mask texture image is a binarized image, the binarized image is an image composed of pixel values 0 and 1, and the mask texture image and the average texture image are fused pixel by the computer equipment, so that a second enhanced texture image corresponding to the target texture image can be obtained.
In the above embodiment, the mask texture image and the average texture image are fused pixel by pixel to obtain the second enhanced texture image corresponding to the target texture image, so that the quality of the second enhanced texture image can be ensured.
In one embodiment, I H Representing a second target texture image, C H Representing a second set of texture maps. To achieve reliable reconstruction of the high-frequency components, pair I H And C H Averaging to obtain a refined high-frequency component I Mean . Further to I Mean Is processed to obtain
，
Representing a second enhanced texture image. To match I Mean Resolution (I) of Mean ∈R H×w×1 ) To 1, pair L （I L ∈R H /2×w/2×1 ) And
（
∈R H/2×w/2×1 ) Upsampling is performed. Then connected in series with I Mean And up-sampled I L 、
Inputting the images after series connection into a lightweight network consisting of residual blocks, and outputting the network I Mask ，I Mask ∈R H×w×1 。
Further, a calculation formula for obtaining the second enhanced texture image by fusing the mask texture image and the average texture image is as follows:
where 8855denotes pixel-by-pixel multiplication.
In one embodiment, fusing the first enhanced texture image and the second enhanced texture image to obtain a reconstructed texture image corresponding to the target texture image, including:
performing up-sampling processing on the first enhanced texture image, and fusing the first enhanced texture image and the second enhanced texture image after the up-sampling processing to obtain a fused texture image; and performing convolution processing on the fusion texture image to obtain a reconstructed texture image corresponding to the target texture image.
Specifically, the first enhanced texture image represents the low-frequency component of the image, and the resolution of the low-frequency component obtained by frequency decomposition is usually reduced, so for the convenience of fusion, the upsampling process may be performed first, and the resolutions of the images to be fused are unified and then fused. And in order to refine the image and smooth the image, the computer equipment further performs convolution processing on the fusion texture image to finally obtain a reconstructed texture image corresponding to the target texture image.
In the above embodiment, the first enhanced texture image is subjected to upsampling, the first enhanced texture image and the second enhanced texture image subjected to upsampling are fused to obtain a fused texture image, and the fused texture image is subjected to convolution processing to obtain a reconstructed texture image corresponding to the target texture image. The resolution of the image is fused again and again, so that the accuracy and the efficiency of fusion can be guaranteed. The frequency band information of the reconstructed texture can be further refined through convolution processing, and the quality of the reconstructed texture image is further improved.
In one embodiment, the calculation formula for fusing the first enhanced texture image and the second enhanced texture image to obtain the reconstructed texture image is as follows:
I E =H R (
+Upscale(
))
wherein Upscale is an upsampling process in which first, the upsampling process is performed
Is enlarged, zero is filled in the pixel value corresponding to the area to be supplemented, and then a gaussian kernel is used for convolution adjustmentThe size of the whole image is equal to
Are consistent in size. H R Is a simple convolution operation for further refining the reconstructed texture frequency band information. I is E Representing the resulting reconstructed texture image.
In one embodiment, as shown in fig. 7, the texture image reconstruction method further includes:
step S702, inputting the texture image set and the target texture image into a texture reconstruction model; the texture reconstruction model comprises an image decomposition network, a first image enhancement network, a second image enhancement network and an image reconstruction network.
Step S704, the texture image set and the target texture image are input into an image decomposition network for frequency decomposition, so as to obtain a first texture image set and a second texture image set corresponding to the texture image set, and a first target texture image and a second target texture image corresponding to the target texture image.
Step S706, inputting the first texture image set and the first target texture image into a first image enhancement network to obtain a first enhanced texture image corresponding to the target texture image.
Step S708, the second texture image set and the second target texture image are input to a second image enhancement network, so as to obtain a second enhanced texture image corresponding to the target texture image.
Step S710, inputting the first enhanced texture image and the second enhanced texture image into an image reconstruction network to obtain a reconstructed texture image corresponding to the target texture image.
The texture reconstruction model is a neural network model for performing texture reconstruction. The input data of the texture reconstruction model is a target texture image and a texture image set corresponding to the target texture image, and the output data is a reconstructed texture image corresponding to the target texture image.
The texture reconstruction model comprises an image decomposition network, a first image enhancement network, a second image enhancement network and an image reconstruction network. The image decomposition network is used for carrying out frequency decomposition, the first image enhancement network is used for carrying out image enhancement of low-frequency components, the second image enhancement network is used for carrying out image enhancement of high-frequency components, and the image reconstruction network is used for fusing the enhanced low-frequency components and the enhanced high-frequency components.
Specifically, the computer device inputs the texture image set and the target texture image into the texture reconstruction model, the texture image set and the target texture image are input into an image decomposition network in the texture reconstruction model for frequency decomposition, and the image decomposition network outputs a first texture image set and a second texture image set corresponding to the texture image set and a first target texture image and a second target texture image corresponding to the target texture image. The first texture image set and the first target texture image are input to a first image enhancement network in the texture reconstruction model for image enhancement, and the first image enhancement network outputs a first enhanced texture image corresponding to the target texture image. And inputting the second texture image set and the second target texture image into a second image enhancement network in the texture reconstruction model for image enhancement, and outputting a second enhanced texture image corresponding to the target texture image by the second image enhancement network. It will be appreciated that the first image enhancement network and the second image enhancement network may perform data processing in parallel. And inputting the first enhanced texture image and the second enhanced texture image into an image reconstruction network in the texture reconstruction model, and outputting a reconstructed texture image corresponding to the target texture image by the image reconstruction network. And finally, outputting the reconstructed texture image by the texture reconstruction model.
In the above embodiment, the texture image set and the target texture image are input into the texture reconstruction model, and accurate texture reconstruction can be rapidly achieved through the image decomposition network, the first image enhancement network, the second image enhancement network and the image reconstruction network in the texture reconstruction model, and the reconstructed texture image corresponding to the target texture image is output.
In one embodiment, obtaining a target texture image comprises:
acquiring a game texture image library; and acquiring the game texture image with abnormal illumination from the game texture image library as a target texture image.
Wherein, the game texture image library comprises various texture images required by the game. For example, the game texture image library may include various texture images required by a game character, various texture images required by a game environment, and the like.
Specifically, in the field of games, texture reconstruction is performed on game texture images with abnormal rendering, and the reconstructed texture images obtained by texture reconstruction are used for replacing the abnormal game texture images, so that the display quality of game pictures is improved. The computer device can obtain a game texture image library locally or from other devices, obtain an abnormal game texture image from the game texture image library as a target texture image, and perform texture reconstruction on the target texture image based on a texture image set corresponding to the target texture image to obtain a high-quality reconstructed texture image.
In a game, many objects often have insufficient illumination due to multiple rendering and rendering sequence problems, and in order to improve the illumination rendering effect of the image, the computer device acquires a game texture image with abnormal illumination from the game texture image library as a target texture image, for example, acquires the game texture image with insufficient illumination as the target texture image, acquires a game texture image with low illumination as the target texture image, and performs texture reconstruction on the target texture image based on a texture image set corresponding to the target texture image to obtain a high-quality reconstructed texture image.
The game texture image with abnormal illumination can be a game texture image with an illumination abnormal label, and the illumination abnormal label can be manually labeled in advance or obtained by evaluating through an algorithm for evaluating the illumination effect.
In one embodiment, texture reconstruction is performed during the game testing phase. In the game testing stage, a game texture image obtained through rendering is obtained, the game texture image with abnormal illumination is obtained from the game texture image as a target texture image, and texture reconstruction is carried out on the target texture image to obtain a reconstructed texture image. In the online application stage of the game, a user starts a game, loads the reconstructed texture image instead of the game texture image with abnormal illumination, and shows the reconstructed texture image to the user so as to improve the display effect of a game picture.
In the above embodiment, the game texture image with abnormal illumination is obtained from the game texture image library as the target texture image, and texture reconstruction is performed on the target texture image based on the texture image set corresponding to the target texture image, so that illumination of the target texture image can be restored, illumination of the target texture image can be enhanced, and a reconstructed texture image with a better illumination effect corresponding to the target texture image can be obtained.
In a specific embodiment, the texture image reconstruction method of the present application may be applied to a game scene to optimize performance of texture resources in a game. In a game, many objects cause insufficient illumination effect due to multiple times of rendering and rendering sequence problems, and the method provides a low-light game texture color space enhanced model (which can be called a texture reconstruction model) which can improve the rendering effect of a game texture image, so that the reconstructed game texture image can contain more abundant and more detailed textures.
Referring to fig. 8, the model includes a gaussian pyramid network, an illumination enhancement branch, a high-frequency refinement branch, and a gaussian reconstruction network.
Server obtains game texture image set C 0 Selecting a low-light game texture image from the game texture image set as a target texture image I 0 . Wherein, the game texture image set C 0 The game texture image is obtained based on game texture images with different resolutions and corresponding to the same texture, and the game texture images with different resolutions and corresponding to the same texture are adapted to different terminals. C 0 ∈R H×w×k Is represented by C 0 Comprising k texture images, I 0 ∈R H×w×1 。
Server sends C 0 And I 0 Inputting a Gaussian pyramid network, performing Gaussian decomposition by a Gaussian decomposition module in the Gaussian pyramid network, wherein the Gaussian decomposition module represents a standard Gaussian decomposition process, C 0 Is decomposed into C H And C L ，C H Represents a high frequency component, C H ∈R H×w×1 ，C L Representing a low frequency component, C L ∈R H/2×w/2×1 。I 0 Is decomposed into I H And I L ，I H Represents a high frequency component, I H ∈R H×w×1 ，I L Represents a low frequency component, I L ∈R H/2×w/2×1 。
The illumination enhancement branch aims at enlightening the low-frequency component of the low-light game texture color space and recovering illumination. The illumination enhancement branch includes a low light feature extraction module, an illumination enhancement module (which may also be referred to as a max-desired module), and a reconstruction module. Will I L And C L Low light feature extraction module in input illumination enhancement branch, low light feature extraction module output F 0 And the low-light feature extraction module is responsible for extracting the multi-scale space and the spectral features of the game texture color space. F is to be 0 Input illumination enhancement module, output F of illumination enhancement module D The illumination enhancement module is used for illuminating dark areas and removing various noises in the low-light game texture color space. F is to be D Input reconstruction module, reconstruction module output
And the reconstruction module is used for reconstructing the low-frequency component of the low-light game texture color space. In the reconstruction module, based on F D Generation of I R Is shown by R And I L Are added to obtain
，
Representing the low frequency components of the game texture color space band plus light.
∈R H/2×w/2×1 。
The high frequency refinement branch aims at restoring the texture details and reducing the artifacts in the reconstruction. Will I H And C H The high frequency refinement branch is input. To take advantage of the high frequency properties of the game texture color space, in the high frequency refinement branch, I is calculated H And C H Mean value of (1) Mean To replace I H Refine the high frequency component becauseI Mean Relates to I H Missing portions of texture information. To match I Mean Resolution of (2), to I L And
using an upsampling operation, I Mean And up-sampled I L 、
After splicing, the input of the lightweight network consisting of three residual blocks and the output I of the lightweight network Mask . Will I Mean And I Mask Multiplication pixel by pixel to obtain
。
∈R H×w×1 。
Due to the reversible nature of the gaussian pyramid, the image can be reconstructed by sequential mirroring operations. Will be provided with
And
and inputting the Gaussian reconstruction network. To match with
Resolution of (2), to
Performing an upsampling operation on
And after up-sampling
Adding the data and carrying out convolution operation, thereby refining the reconstructed texture frequency band information by one step to obtain I E Thus, the reconstruction of the texture is completed.
The method is a brand-new low-light generation technology, solves the problem of overlapping between actual low light and unreal low light in a game engine, can illuminate dark areas with low light in texture images, and can simultaneously inhibit various noises and keep the fidelity of frequency spectrums. The method is already applied to texture resource performance optimization work of the MOBA (Multiplayer Online Battle Arena), and can greatly improve the display effect of the final game picture and improve the quality of the game picture.
In addition, through testing, compared with the traditional method, the loading time and the GPU consumption of the method are less. Specifically, the texture resources are extracted by using a debug tool, the extracted texture resources are sent to the model in the method, the original texture is replaced after optimization, and the texture loading process is operated again.
It can be understood that the method can also be applied to scenes such as movie and television special effects, visual design, VR (Virtual Reality), industrial simulation, digital text creation and the like. The digital text creation body can comprise a rendered building or tourist attraction and the like. It is understood that processing of texture images may be involved in movie special effects, visualization design, VR and digital text creation, among other scenarios. The texture image reconstruction method can be used for reconstructing the texture image in each scene. The texture image reconstruction method can be used for reconstructing the texture image, the definition of the texture image can be enhanced, more texture details are enhanced, and the quality of the texture image is greatly improved, so that the picture display effects in scenes such as movie and television special effects, visual designs, VR (Virtual Reality), industrial simulation, digital text creation and the like are improved.
For example, industrial simulation refers to the simulation demonstration of industrial processes and industrial products. In an industrial simulation scene, simulation demonstration of an industrial production environment may be involved, for example, three-dimensional digital modeling is performed on plants, equipment and facilities, a low-quality texture image can be searched from each texture image corresponding to an industrial simulation model to serve as a target texture image.
For example, digital creative work refers to creation, production, transmission and service based on digital technology with cultural creative content as the core. In a digital text creation scene, three-dimensional digital modeling of buildings with cultural representative significance may be involved, for example, three-dimensional digital modeling of a museum or a historical building is performed, a low-quality texture image can be searched from each texture image corresponding to a building model to serve as a target texture image. It should be understood that, although the steps in the flowcharts related to the embodiments described above are shown in sequence as indicated by the arrows, the steps are not necessarily performed in sequence as indicated by the arrows. The steps are not limited to being performed in the exact order illustrated and, unless explicitly stated herein, may be performed in other orders. Moreover, at least a part of the steps in the flowcharts related to the embodiments described above may include multiple steps or multiple stages, which are not necessarily performed at the same time, but may be performed at different times, and the execution order of the steps or stages is not necessarily sequential, but may be rotated or alternated with other steps or at least a part of the steps or stages in other steps.
Based on the same inventive concept, the embodiment of the present application further provides a texture image reconstruction device for implementing the texture image reconstruction method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme described in the above method, so specific limitations in one or more embodiments of the texture image reconstruction device provided below can be referred to the limitations of the texture image reconstruction method in the foregoing, and details are not described here.
In one embodiment, as shown in fig. 9, there is provided a texture image reconstruction apparatus including: a texture image acquisition module 902, an image decomposition module 904, a first image enhancement module 906, a second image enhancement module 908, and an image fusion module 910, wherein:
a texture image obtaining module 902, configured to obtain a target texture image and a texture image set corresponding to the target texture image; the texture presented by the texture image in the texture image set and the texture presented by the target texture image are matched with each other, and the texture image set is obtained based on the texture images with different resolutions.
An image decomposition module 904, configured to perform frequency decomposition on the texture image set and the target texture image respectively to obtain a first texture image set and a second texture image set corresponding to the texture image set, and a first target texture image and a second target texture image corresponding to the target texture image; the frequency corresponding to the first texture image set is smaller than the frequency corresponding to the second texture image set, and the frequency corresponding to the first target texture image is smaller than the frequency corresponding to the second target texture image.
The first image enhancement module 906 is configured to perform image enhancement on the first target texture image based on the first texture image set to obtain a first enhanced texture image corresponding to the first target texture image.
The second image enhancement module 908 is configured to perform image enhancement on the second target texture image based on the second texture image set, so as to obtain a second enhanced texture image corresponding to the second target texture image.
And an image fusion module 910, configured to fuse the first enhanced texture image and the second enhanced texture image to obtain a reconstructed texture image corresponding to the target texture image.
In one embodiment, the image decomposition module 904 is further configured to:
performing Gaussian decomposition on the texture image set to obtain a first texture image set and a second texture image set corresponding to the texture image set; and carrying out Gaussian decomposition on the target texture image to obtain a first target texture image and a second target texture image corresponding to the target texture image.
In one embodiment, the first image enhancement module 906 is further for:
performing convolution processing on the first texture image set and the first target texture image respectively to obtain initial texture feature maps corresponding to the first texture image set and the first target texture image respectively; splicing the initial texture feature maps corresponding to the first texture image set and the first target texture image respectively to obtain an intermediate texture feature map; performing attention processing on the intermediate texture feature map to obtain a target texture feature map; and obtaining a first enhanced texture image corresponding to the target texture image based on the target texture feature image and the first target texture image.
In one embodiment, the first image enhancement module 906 is further for:
performing convolution processing on the first texture image set based on at least two first convolution kernels to obtain at least two first convolution feature maps, and splicing the at least two first convolution feature maps to obtain an initial texture feature map corresponding to the first texture image set; the at least two first convolution cores comprise at least two sizes of first convolution cores; performing convolution processing on the first target texture image based on at least two second convolution kernels to obtain at least two second convolution characteristic graphs, and splicing the at least two second convolution characteristic graphs to obtain an initial texture characteristic graph corresponding to the first target texture image; the at least two second convolution kernels comprise at least two sizes of second convolution kernels.
In one embodiment, the first image enhancement module 906 is further for:
splicing the initial texture feature maps corresponding to the first texture image set and the first target texture image respectively to obtain a first texture feature map, and rectifying the first texture feature map to obtain a second texture feature map; and performing convolution processing on the second texture feature map to obtain a third texture feature map, performing up-sampling processing on the third texture feature map to obtain a fourth texture feature map, and performing rectification processing on the fourth texture feature map to obtain an intermediate texture feature map.
In one embodiment, the first image enhancement module 906 is further for:
sequentially performing at least two times of ordered attention processing on the intermediate texture feature map to obtain at least two ordered attention texture feature maps; splicing at least two ordered attention texture feature maps to obtain a first spliced texture feature map, and performing convolution processing on the first spliced texture feature map to obtain a convolution texture feature map; acquiring an ending attention texture feature map from at least two ordered attention texture feature maps, and fusing the ending attention texture feature map and the convolution texture feature map to obtain a fused texture feature map; and splicing the at least two ordered attention texture feature maps and the fusion texture feature map to obtain a second spliced texture feature map, and performing convolution processing on the second spliced texture feature map to obtain a target texture feature map.
In one embodiment, the first image enhancement module 906 is further for:
performing convolution processing on the target texture feature map to obtain a supplementary texture feature map; and fusing the supplementary texture feature image and the first target texture image to obtain a first enhanced texture image corresponding to the target texture image.
In one embodiment, the second image enhancement module 908 is further configured to:
averaging the second texture image set and the second target texture image to obtain an average texture image; splicing the average texture image, the first target texture image and the first enhanced texture image to obtain a spliced texture image; residual error processing is carried out on the spliced texture image to obtain a mask texture image; and fusing the mask texture image and the average texture image to obtain a second enhanced texture image corresponding to the target texture image.
In one embodiment, the second image enhancement module 908 is further configured to:
respectively carrying out up-sampling processing on the first target texture image and the first enhanced texture image to obtain a first up-sampling texture image corresponding to the first target texture image and a second up-sampling texture image corresponding to the first enhanced texture image; the resolutions of the first up-sampling texture image, the second up-sampling texture image and the average texture image are consistent; and splicing the average texture image, the first up-sampling texture image and the second up-sampling texture image to obtain a spliced texture image.
In one embodiment, the second image enhancement module 908 is further configured to:
and performing pixel-by-pixel fusion on the mask texture image and the average texture image to obtain a second enhanced texture image corresponding to the target texture image.
In one embodiment, the image fusion module 910 is further configured to:
performing up-sampling processing on the first enhanced texture image, and fusing the first enhanced texture image and the second enhanced texture image after the up-sampling processing to obtain a fused texture image; and performing convolution processing on the fusion texture image to obtain a reconstructed texture image corresponding to the target texture image.
In one embodiment, the texture image reconstruction device is further configured to:
inputting the texture image set and the target texture image into a texture reconstruction model; the texture reconstruction model comprises an image decomposition network, a first image enhancement network, a second image enhancement network and an image reconstruction network; inputting the texture image set and the target texture image into an image decomposition network for frequency decomposition to obtain a first texture image set and a second texture image set corresponding to the texture image set, and a first target texture image and a second target texture image corresponding to the target texture image; inputting the first texture image set and the first target texture image into a first image enhancement network to obtain a first enhanced texture image corresponding to the target texture image; inputting the second texture image set and the second target texture image into a second image enhancement network to obtain a second enhanced texture image corresponding to the target texture image; and inputting the first enhanced texture image and the second enhanced texture image into an image reconstruction network to obtain a reconstructed texture image corresponding to the target texture image.
In one embodiment, the texture image acquisition module 902 is further configured to:
acquiring a game texture image library; and acquiring the game texture image with abnormal illumination from the game texture image library as a target texture image.
The texture image reconstruction device acquires the target texture image and a texture image set corresponding to the target texture image, the texture presented by the texture image in the texture image set is matched with the texture presented by the target texture image, the texture image set is obtained based on texture images with different resolutions, and the texture reconstruction is performed on the target texture image by means of the texture image set. The method comprises the steps of respectively carrying out frequency decomposition on a texture image set and a target texture image to obtain a first texture image set and a second texture image set corresponding to the texture image set, and a first target texture image and a second target texture image corresponding to the target texture image, decomposing the texture image set into a first texture image set representing low-frequency components of the image and a second texture image set representing high-frequency components of the image through frequency decomposition, and decomposing the first target texture image into a first target texture image representing the low-frequency components of the image and a second target texture image representing the high-frequency components of the image. The first target texture image is subjected to image enhancement based on the first texture image set, so that a first enhanced texture image corresponding to the first target texture image can be obtained, which is equivalent to enhancing low-frequency components in the target texture image, for example, enhancing illumination. And performing image enhancement on the second target texture image based on the second texture image set to obtain a second enhanced texture image corresponding to the second target texture image, which is equivalent to enhancing high-frequency components in the target texture image, for example, enhancing texture details. And fusing the first enhanced texture image and the second enhanced texture image to obtain a reconstructed texture image corresponding to the target texture image. Therefore, the low-quality target texture image can be converted into the reconstructed texture image with high definition and rich detail information, the definition of the generated reconstructed texture image is enhanced while the original texture information is kept, more texture details are enhanced, and the reconstruction quality of the texture image is greatly improved.
The modules in the texture image reconstruction device can be wholly or partially implemented by software, hardware and a combination thereof. The modules can be embedded in a hardware form or independent of a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.
In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as shown in fig. 10. The computer device comprises a processor, a memory, an Input/Output (I/O) interface and a communication interface. The processor, the memory and the input/output interface are connected through a system bus, and the communication interface is connected to the system bus through the input/output interface. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The database of the computer device is used for storing texture images, models and the like. The input/output interface of the computer device is used for exchanging information between the processor and an external device. The communication interface of the computer device is used for connecting and communicating with an external terminal through a network. The computer program is executed by a processor to implement a texture image reconstruction method.
In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as shown in fig. 11. The computer apparatus includes a processor, a memory, an input/output interface, a communication interface, a display unit, and an input device. The processor, the memory and the input/output interface are connected by a system bus, and the communication interface, the display unit and the input device are connected by the input/output interface to the system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program to run on the non-volatile storage medium. The input/output interface of the computer device is used for exchanging information between the processor and an external device. The communication interface of the computer device is used for carrying out wired or wireless communication with an external terminal, and the wireless communication can be realized through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. The computer program is executed by a processor to implement a texture image reconstruction method. The display unit of the computer equipment is used for forming a visual and visible picture, and can be a display screen, a projection device or a virtual reality imaging device, the display screen can be a liquid crystal display screen or an electronic ink display screen, the input device of the computer equipment can be a touch layer covered on the display screen, a key, a track ball or a touch pad arranged on the shell of the computer equipment, and can also be an external keyboard, a touch pad or a mouse and the like.
It will be appreciated by those skilled in the art that the configurations shown in fig. 10 and 11 are block diagrams of only some of the configurations relevant to the present disclosure, and do not constitute a limitation on the computing devices to which the present disclosure may be applied, and that a particular computing device may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.
In an embodiment, a computer device is further provided, which includes a memory and a processor, the memory stores a computer program, and the processor implements the steps of the above method embodiments when executing the computer program.
In an embodiment, a computer-readable storage medium is provided, in which a computer program is stored which, when being executed by a processor, carries out the steps of the above-mentioned method embodiments.
In an embodiment, a computer program product is provided, comprising a computer program which, when executed by a processor, performs the steps in the above-described method embodiments.
It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) referred to in the present application are information and data authorized by the user or fully authorized by each party, and the collection, use and processing of the related data need to comply with the relevant laws and regulations and standards of the relevant countries and regions.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above may be implemented by hardware instructions of a computer program, which may be stored in a non-volatile computer-readable storage medium, and when executed, may include the processes of the embodiments of the methods described above. Any reference to memory, databases, or other media used in the embodiments provided herein can include at least one of non-volatile and volatile memory. The nonvolatile Memory may include a Read-Only Memory (ROM), a magnetic tape, a floppy disk, a flash Memory, an optical Memory, a high-density embedded nonvolatile Memory, a resistive Random Access Memory (ReRAM), a Magnetic Random Access Memory (MRAM), a Ferroelectric Random Access Memory (FRAM), a Phase Change Memory (PCM), a graphene Memory, and the like. Volatile Memory can include Random Access Memory (RAM), external cache Memory, and the like. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM), among others. The databases referred to in various embodiments provided herein may include at least one of relational and non-relational databases. The non-relational database may include, but is not limited to, a block chain based distributed database, and the like. The processors referred to in the various embodiments provided herein may be, without limitation, general purpose processors, central processing units, graphics processors, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, or the like.
The technical features of the above embodiments can be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the above embodiments are not described, but should be considered as the scope of the present specification as long as there is no contradiction between the combinations of the technical features.
The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the present application. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, and these are all within the scope of protection of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.
Claims (17)
1. A method of texture image reconstruction, the method comprising:
acquiring a target texture image and a texture image set corresponding to the target texture image; the texture presented by the texture image in the texture image set and the texture presented by the target texture image are matched with each other, and the texture image set is obtained based on texture images with different resolutions;
respectively carrying out frequency decomposition on the texture image set and the target texture image to obtain a first texture image set and a second texture image set corresponding to the texture image set, and a first target texture image and a second target texture image corresponding to the target texture image; the frequency corresponding to the first texture image set is smaller than the frequency corresponding to the second texture image set, and the frequency corresponding to the first target texture image is smaller than the frequency corresponding to the second target texture image;
performing image enhancement on the first target texture image based on the first texture image set to obtain a first enhanced texture image corresponding to the first target texture image;
performing image enhancement on the second target texture image based on the second texture image set to obtain a second enhanced texture image corresponding to the second target texture image;
and fusing the first enhanced texture image and the second enhanced texture image to obtain a reconstructed texture image corresponding to the target texture image.
2. The method according to claim 1, wherein the frequency decomposition of the texture image set and the target texture image to obtain a first texture image set and a second texture image set corresponding to the texture image set, and a first target texture image and a second target texture image corresponding to the target texture image respectively comprises:
performing Gaussian decomposition on the texture image set to obtain a first texture image set and a second texture image set corresponding to the texture image set;
and carrying out Gaussian decomposition on the target texture image to obtain a first target texture image and a second target texture image corresponding to the target texture image.
3. The method according to claim 1, wherein the image enhancing the first target texture image based on the first texture image set to obtain a first enhanced texture image corresponding to the first target texture image comprises:
performing convolution processing on the first texture image set and the first target texture image respectively to obtain initial texture feature maps corresponding to the first texture image set and the first target texture image respectively;
splicing the initial texture feature images corresponding to the first texture image set and the first target texture image respectively to obtain an intermediate texture feature image;
attention processing is carried out on the intermediate texture feature map to obtain a target texture feature map;
and obtaining a first enhanced texture image corresponding to the target texture image based on the target texture feature map and the first target texture image.
4. The method according to claim 3, wherein the convolving the first texture image set and the first target texture image to obtain initial texture feature maps corresponding to the first texture image set and the first target texture image, respectively, comprises:
performing convolution processing on the first texture image set based on at least two first convolution kernels to obtain at least two first convolution feature maps, and splicing the at least two first convolution feature maps to obtain an initial texture feature map corresponding to the first texture image set; the at least two first convolution cores comprise at least two sizes of first convolution cores;
performing convolution processing on the first target texture image based on at least two second convolution kernels to obtain at least two second convolution feature maps, and splicing the at least two second convolution feature maps to obtain an initial texture feature map corresponding to the first target texture image; the at least two second convolution kernels comprise at least two sizes of second convolution kernels.
5. The method according to claim 3, wherein the step of performing stitching processing on the initial texture feature maps corresponding to the first texture image set and the first target texture image, respectively, to obtain an intermediate texture feature map comprises:
splicing the initial texture feature maps corresponding to the first texture image set and the first target texture image respectively to obtain a first texture feature map, and rectifying the first texture feature map to obtain a second texture feature map;
and performing convolution processing on the second texture feature map to obtain a third texture feature map, performing up-sampling processing on the third texture feature map to obtain a fourth texture feature map, and performing rectification processing on the fourth texture feature map to obtain an intermediate texture feature map.
6. The method according to claim 3, wherein said performing attention processing on the intermediate texture feature map to obtain a target texture feature map comprises:
sequentially carrying out at least two times of ordered attention processing on the intermediate texture feature map to obtain at least two ordered attention texture feature maps;
splicing the at least two ordered attention texture feature maps to obtain a first spliced texture feature map, and performing convolution processing on the first spliced texture feature map to obtain a convolution texture feature map;
acquiring an ending attention texture feature map from the at least two ordered attention texture feature maps, and fusing the ending attention texture feature map and the convolution texture feature map to obtain a fused texture feature map;
and splicing the at least two ordered attention texture feature maps and the fusion texture feature map to obtain a second spliced texture feature map, and performing convolution processing on the second spliced texture feature map to obtain a target texture feature map.
7. The method according to claim 3, wherein obtaining a first enhanced texture image corresponding to the target texture image based on the target texture feature map and the first target texture image comprises:
performing convolution processing on the target texture feature map to obtain a supplementary texture feature map;
and fusing the supplementary texture feature map and the first target texture image to obtain a first enhanced texture image corresponding to the target texture image.
8. The method according to claim 1, wherein the image enhancing the second target texture image based on the second texture image set to obtain a second enhanced texture image corresponding to the second target texture image comprises:
carrying out average processing on the second texture image set and the second target texture image to obtain an average texture image;
splicing the average texture image, the first target texture image and the first enhanced texture image to obtain a spliced texture image;
performing residual error processing on the spliced texture image to obtain a mask texture image;
and fusing the mask texture image and the average texture image to obtain a second enhanced texture image corresponding to the target texture image.
9. The method according to claim 8, wherein the stitching the average texture image, the first target texture image and the first enhanced texture image to obtain a stitched texture image comprises:
respectively performing upsampling processing on the first target texture image and the first enhanced texture image to obtain a first upsampled texture image corresponding to the first target texture image and a second upsampled texture image corresponding to the first enhanced texture image; the resolutions of the first upsampled texture image, the second upsampled texture image and the average texture image are consistent;
and splicing the average texture image, the first up-sampling texture image and the second up-sampling texture image to obtain a spliced texture image.
10. The method according to claim 8, wherein the fusing the mask texture image and the average texture image to obtain a second enhanced texture image corresponding to the target texture image comprises:
and performing pixel-by-pixel fusion on the mask texture image and the average texture image to obtain a second enhanced texture image corresponding to the target texture image.
11. The method according to claim 1, wherein the fusing the first enhanced texture image and the second enhanced texture image to obtain a reconstructed texture image corresponding to the target texture image comprises:
performing upsampling processing on the first enhanced texture image, and fusing the upsampled first enhanced texture image and the second enhanced texture image to obtain a fused texture image;
and performing convolution processing on the fusion texture image to obtain a reconstructed texture image corresponding to the target texture image.
12. The method of claim 1, further comprising:
inputting the texture image set and the target texture image into a texture reconstruction model; the texture reconstruction model comprises an image decomposition network, a first image enhancement network, a second image enhancement network and an image reconstruction network;
inputting the texture image set and the target texture image into the image decomposition network for frequency decomposition to obtain a first texture image set and a second texture image set corresponding to the texture image set, and a first target texture image and a second target texture image corresponding to the target texture image;
inputting the first texture image set and the first target texture image into the first image enhancement network to obtain a first enhanced texture image corresponding to the target texture image;
inputting the second texture image set and the second target texture image into the second image enhancement network to obtain a second enhanced texture image corresponding to the target texture image;
and inputting the first enhanced texture image and the second enhanced texture image into the image reconstruction network to obtain a reconstructed texture image corresponding to the target texture image.
13. The method of any one of claims 1 to 12, wherein the obtaining a target texture image comprises:
acquiring a game texture image library;
and acquiring the game texture image with abnormal illumination as a target texture image from the game texture image library.
14. A texture image reconstruction apparatus, characterized in that the apparatus comprises:
the texture image acquisition module is used for acquiring a target texture image and a texture image set corresponding to the target texture image; the texture presented by the texture image in the texture image set and the texture presented by the target texture image are matched with each other, and the texture image set is obtained based on texture images with different resolutions;
an image decomposition module, configured to perform frequency decomposition on the texture image set and the target texture image, respectively, to obtain a first texture image set and a second texture image set corresponding to the texture image set, and a first target texture image and a second target texture image corresponding to the target texture image; the frequency corresponding to the first texture image set is smaller than the frequency corresponding to the second texture image set, and the frequency corresponding to the first target texture image is smaller than the frequency corresponding to the second target texture image;
a first image enhancement module, configured to perform image enhancement on the first target texture image based on the first texture image set, to obtain a first enhanced texture image corresponding to the first target texture image;
a second image enhancement module, configured to perform image enhancement on the second target texture image based on the second texture image set, to obtain a second enhanced texture image corresponding to the second target texture image;
and the image fusion module is used for fusing the first enhanced texture image and the second enhanced texture image to obtain a reconstructed texture image corresponding to the target texture image.
15. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor realizes the steps of the method of any one of claims 1 to 13 when executing the computer program.
16. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the steps of the method according to any one of claims 1 to 13.
17. A computer program product comprising a computer program, characterized in that the computer program realizes the steps of the method of any one of claims 1 to 13 when executed by a processor.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310013253.1A CN115713585B (en) | 2023-01-05 | 2023-01-05 | Texture image reconstruction method, apparatus, computer device and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310013253.1A CN115713585B (en) | 2023-01-05 | 2023-01-05 | Texture image reconstruction method, apparatus, computer device and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115713585A true CN115713585A (en) | 2023-02-24 |
| CN115713585B CN115713585B (en) | 2023-05-02 |
Family
ID=85236180
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310013253.1A Active CN115713585B (en) | 2023-01-05 | 2023-01-05 | Texture image reconstruction method, apparatus, computer device and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115713585B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115953330A (en) * | 2023-03-13 | 2023-04-11 | 腾讯科技（深圳）有限公司 | Texture optimization method, device, equipment and storage medium for virtual scene image |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20130315504A1 (en) * | 2012-05-23 | 2013-11-28 | Samsung Electronics Co., Ltd. | Method and apparatus for reconstructing an image |
| CN113284051A (en) * | 2021-07-23 | 2021-08-20 | 之江实验室 | Face super-resolution method based on frequency decomposition multi-attention machine system |
| CN113674165A (en) * | 2021-07-27 | 2021-11-19 | 浙江大华技术股份有限公司 | Image processing method, image processing device, electronic equipment and computer readable storage medium |
| CN115131256A (en) * | 2021-03-24 | 2022-09-30 | 华为技术有限公司 | Image processing model, and training method and device of image processing model |
| WO2022241995A1 (en) * | 2021-05-18 | 2022-11-24 | 广东奥普特科技股份有限公司 | Visual image enhancement generation method and system, device, and storage medium |
-
2023
- 2023-01-05 CN CN202310013253.1A patent/CN115713585B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20130315504A1 (en) * | 2012-05-23 | 2013-11-28 | Samsung Electronics Co., Ltd. | Method and apparatus for reconstructing an image |
| CN115131256A (en) * | 2021-03-24 | 2022-09-30 | 华为技术有限公司 | Image processing model, and training method and device of image processing model |
| WO2022241995A1 (en) * | 2021-05-18 | 2022-11-24 | 广东奥普特科技股份有限公司 | Visual image enhancement generation method and system, device, and storage medium |
| CN113284051A (en) * | 2021-07-23 | 2021-08-20 | 之江实验室 | Face super-resolution method based on frequency decomposition multi-attention machine system |
| CN113674165A (en) * | 2021-07-27 | 2021-11-19 | 浙江大华技术股份有限公司 | Image processing method, image processing device, electronic equipment and computer readable storage medium |
Non-Patent Citations (1)
| Title |
|---|
| 晋杰 等: "基于权重小波变换的图像多分辨率重建算法" * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115953330A (en) * | 2023-03-13 | 2023-04-11 | 腾讯科技（深圳）有限公司 | Texture optimization method, device, equipment and storage medium for virtual scene image |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115713585B (en) | 2023-05-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Jam et al. | A comprehensive review of past and present image inpainting methods | |
| CN109919869A (en) | A kind of image enchancing method, device and storage medium | |
| CN113838176B (en) | Model training method, three-dimensional face image generation method and three-dimensional face image generation equipment | |
| CN114820905B (en) | Virtual image generation method and device, electronic equipment and readable storage medium | |
| US20220377300A1 (en) | Method for displaying objects, electronic device, and storage medium | |
| Akimoto et al. | 360-degree image completion by two-stage conditional gans | |
| CN115713585B (en) | Texture image reconstruction method, apparatus, computer device and storage medium | |
| CN115100337A (en) | Whole body portrait video relighting method and device based on convolutional neural network | |
| CN117058007A (en) | Object class repair in digital images using class-specific repair neural networks | |
| CN116091630A (en) | Method and device for training image generation model | |
| US20230051749A1 (en) | Generating synthesized digital images utilizing class-specific machine-learning models | |
| CN116797768A (en) | Method and device for reducing reality of panoramic image | |
| CN114782460B (en) | Image segmentation model generation method, image segmentation method and computer equipment | |
| CN116664422A (en) | Image highlight processing method and device, electronic equipment and readable storage medium | |
| US11948245B2 (en) | Relighting images and video using learned lighting and geometry | |
| CN115953524A (en) | Data processing method and device, computer equipment and storage medium | |
| CN115578497A (en) | Image scene relighting network structure and method based on GAN network | |
| Jiang et al. | Parallel adaptive guidance network for image inpainting | |
| CN116071478B (en) | Training method of image reconstruction model and virtual scene rendering method | |
| US20240144520A1 (en) | Generating three-dimensional human models representing two-dimensional humans in two-dimensional images | |
| US20240144623A1 (en) | Modifying poses of two-dimensional humans in two-dimensional images by reposing three-dimensional human models representing the two-dimensional humans | |
| US20240135612A1 (en) | Generating shadows for placed objects in depth estimated scenes of two-dimensional images | |
| US11769227B2 (en) | Generating synthesized digital images utilizing a multi-resolution generator neural network | |
| US20230055204A1 (en) | Generating colorized digital images utilizing a re-colorization neural network with local hints | |
| US20240144586A1 (en) | Generating shadows for objects in two-dimensional images utilizing a plurality of shadow maps |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| REG | Reference to a national code |
Ref country code: HK Ref legal event code: DE Ref document number: 40080507 Country of ref document: HK |