CN116977169A - Data processing method, apparatus, device, readable storage medium, and program product - Google Patents
Data processing method, apparatus, device, readable storage medium, and program product Download PDFInfo
- Publication number
- CN116977169A CN116977169A CN202310811145.9A CN202310811145A CN116977169A CN 116977169 A CN116977169 A CN 116977169A CN 202310811145 A CN202310811145 A CN 202310811145A CN 116977169 A CN116977169 A CN 116977169A
- Authority
- CN
- China
- Prior art keywords
- model
- module
- superdivision
- processing
- feature
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 26
- 238000012545 processing Methods 0.000 claims abstract description 370
- 238000003754 machining Methods 0.000 claims abstract description 145
- 238000000034 method Methods 0.000 claims abstract description 114
- 238000012549 training Methods 0.000 claims abstract description 35
- 238000000605 extraction Methods 0.000 claims description 39
- 230000008569 process Effects 0.000 claims description 37
- 238000010586 diagram Methods 0.000 claims description 27
- 230000004927 fusion Effects 0.000 claims description 27
- 230000004913 activation Effects 0.000 claims description 24
- 238000005070 sampling Methods 0.000 claims description 24
- 238000004891 communication Methods 0.000 claims description 14
- 238000004590 computer program Methods 0.000 claims description 12
- 238000007499 fusion processing Methods 0.000 claims description 10
- 230000009467 reduction Effects 0.000 claims description 3
- 238000005516 engineering process Methods 0.000 abstract description 35
- 238000013473 artificial intelligence Methods 0.000 abstract description 19
- 230000006870 function Effects 0.000 description 29
- 238000004364 calculation method Methods 0.000 description 23
- 230000000694 effects Effects 0.000 description 23
- 238000011282 treatment Methods 0.000 description 20
- 238000011161 development Methods 0.000 description 9
- 230000018109 developmental process Effects 0.000 description 9
- 230000001965 increasing effect Effects 0.000 description 9
- 238000013528 artificial neural network Methods 0.000 description 8
- 230000009466 transformation Effects 0.000 description 8
- 238000013135 deep learning Methods 0.000 description 6
- 238000010801 machine learning Methods 0.000 description 6
- 238000011160 research Methods 0.000 description 6
- 230000003993 interaction Effects 0.000 description 5
- 238000013527 convolutional neural network Methods 0.000 description 3
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 230000002349 favourable effect Effects 0.000 description 3
- 239000011159 matrix material Substances 0.000 description 3
- 238000012805 post-processing Methods 0.000 description 3
- 230000006399 behavior Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000004422 calculation algorithm Methods 0.000 description 2
- 238000006243 chemical reaction Methods 0.000 description 2
- 230000006835 compression Effects 0.000 description 2
- 238000007906 compression Methods 0.000 description 2
- 238000000354 decomposition reaction Methods 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 230000008034 disappearance Effects 0.000 description 2
- 238000013507 mapping Methods 0.000 description 2
- 230000001537 neural effect Effects 0.000 description 2
- 238000011176 pooling Methods 0.000 description 2
- 238000007781 pre-processing Methods 0.000 description 2
- 238000013138 pruning Methods 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 230000008485 antagonism Effects 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 230000003190 augmentative effect Effects 0.000 description 1
- 230000033228 biological regulation Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000012937 correction Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000003708 edge detection Methods 0.000 description 1
- 230000002708 enhancing effect Effects 0.000 description 1
- 238000004880 explosion Methods 0.000 description 1
- 230000006698 induction Effects 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 238000003058 natural language processing Methods 0.000 description 1
- 230000006855 networking Effects 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 238000013139 quantization Methods 0.000 description 1
- 238000011084 recovery Methods 0.000 description 1
- 238000011946 reduction process Methods 0.000 description 1
- 230000002787 reinforcement Effects 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
- 238000013526 transfer learning Methods 0.000 description 1
- 238000000844 transformation Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/0464—Convolutional networks [CNN, ConvNet]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/048—Activation functions
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/09—Supervised learning
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Biophysics (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Biomedical Technology (AREA)
- Health & Medical Sciences (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Life Sciences & Earth Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Image Processing (AREA)
Abstract
The embodiment of the application provides a data processing method, a device, equipment, a readable storage medium and a program product, which can be applied to fields or scenes such as cloud technology, artificial intelligence, image super-resolution processing and the like, and the method comprises the following steps: performing superdivision processing on the sample image by using the initial superdivision model to obtain a predicted superdivision image; training an initial superdivision model according to the predicted superdivision image and the reference superdivision image to obtain a first superdivision model; performing re-parameterization on a module to be processed in the first superdivision model to generate a second superdivision model; the second superdivision model is used for generating a target superdivision image of the image to be processed, and the module to be processed is a characteristic rough machining network after model parameter adjustment in the initial superdivision model and part or all of modules in the characteristic fine machining network; the parameter number of the second superdivision model is smaller than that of the first superdivision model, and the method can improve the model capacity of the superdivision model and the quality of superdivision images obtained through the superdivision model.
Description
Technical Field
The present application relates to the field of computer technology, and in particular, to a data processing method, a data processing apparatus, a computer device, a computer readable storage medium, and a computer program product.
Background
Image super processing is one of the important directions in the field of computer vision research, and the purpose of the image super processing is to recover a high-resolution image from a low-resolution image. In recent years, with the development of deep learning technology, many image superdivision methods based on complex superdivision models have been proposed and exhibit good performance. However, due to limited computing power of the terminal device, the complex superdivision model cannot be well configured on the terminal device, so that the superdivision model with a simple structure is generally configured on the terminal device, which results in lower model capacity of the superdivision model on the terminal device and further poor quality of the superdivision image obtained through the superdivision model. Therefore, how to increase the model capacity of the superdivision model and to increase the quality of the superdivision image obtained by the superdivision model is a problem that needs to be solved at present.
Disclosure of Invention
The application provides a data processing method, a device, equipment, a readable storage medium and a program product, which can improve the model capacity of a super-division model and the quality of a super-division image obtained by the super-division model.
In a first aspect, the present application provides a data processing method, the method comprising:
performing superdivision processing on the sample image by using the initial superdivision model to obtain a predicted superdivision image;
performing model parameter adjustment on the initial superdivision model according to the predicted superdivision image and a reference superdivision image corresponding to the sample image to obtain a first superdivision model;
performing re-parameterization on the to-be-processed module in the first super-division model to generate a second super-division model, wherein the second super-division model is used for generating a target super-division image of the to-be-processed image;
the initial super-division model comprises a characteristic rough machining network, a characteristic fine machining network and a characteristic fusion network, and the to-be-processed module is a part of or all of the characteristic rough machining network and the characteristic fine machining network after the model is subjected to parameter adjustment; the second superdivision model has a smaller number of parameters than the first superdivision model.
In a second aspect, the present application provides another data processing method, the method comprising:
acquiring an image to be processed;
performing superprocessing on the image to be processed by using a second superdivision model to obtain a target superdivision image;
The second superdivision model is generated by carrying out re-parameterization on a to-be-processed module in a first superdivision model, the first superdivision model is obtained by carrying out model training on an initial superdivision model by utilizing a sample image, the initial superdivision model comprises a characteristic rough machining network, a characteristic fine machining network and a characteristic fusion network, and the to-be-processed module is a part of or all of the modules in the characteristic rough machining network and the characteristic fine machining network after model adjustment during model training on the initial superdivision model; the second superdivision model has a smaller number of parameters than the first superdivision model.
In a third aspect, the present application provides a data processing apparatus, the apparatus comprising:
the superdivision processing module is used for superdivision processing of the sample image by using the initial superdivision model to obtain a predicted superdivision image;
the training module is used for carrying out model parameter adjustment on the initial superminute model according to the prediction superminute image and the reference superminute image corresponding to the sample image to obtain a first superminute model;
the re-parameterization module is used for carrying out re-parameterization processing on the to-be-processed module in the first super-division model to generate a second super-division model, and the second super-division model is used for generating a target super-division image of the to-be-processed image;
The initial super-division model comprises a characteristic rough machining network, a characteristic fine machining network and a characteristic fusion network, and the to-be-processed module is a part of or all of the characteristic rough machining network and the characteristic fine machining network after the model is subjected to parameter adjustment; the second superdivision model has a smaller number of parameters than the first superdivision model.
In a fourth aspect, the present application provides another data processing apparatus, the apparatus comprising:
the super processing module is used for acquiring an image to be processed;
the superprocessing module is further used for superprocessing the image to be processed by using a second superdivision model to obtain a target superdivision image;
the second superdivision model is generated by carrying out re-parameterization on a to-be-processed module in a first superdivision model, the first superdivision model is obtained by carrying out model training on an initial superdivision model by utilizing a sample image, the initial superdivision model comprises a characteristic rough machining network, a characteristic fine machining network and a characteristic fusion network, and the to-be-processed module is a part of or all of the modules in the characteristic rough machining network and the characteristic fine machining network after model adjustment during model training on the initial superdivision model; the second superdivision model has a smaller number of parameters than the first superdivision model.
In a fifth aspect, the present application provides a computer device comprising: the device comprises a processor, a storage device and a communication interface, wherein the processor, the communication interface and the storage device are mutually connected, the storage device stores executable program codes, and the processor is used for calling the executable program codes so as to realize the data processing method.
In a sixth aspect, the present application provides a computer readable storage medium storing a computer program comprising program instructions for execution by a processor for performing a data processing method as described above.
In a seventh aspect, the present application provides a computer program product comprising a computer program or computer instructions for execution by a processor for performing the data processing method described above.
According to the embodiment of the application, an initial super-division model is utilized to perform super-division processing on a sample image to obtain a predicted super-division image; then performing model parameter adjustment on the initial superdivision model according to the predicted superdivision image and the reference superdivision image to obtain a first superdivision model, so that the first superdivision model has higher model precision and better superdivision effect; and performing re-parameterization processing on the module to be processed in the first superdivision model to generate a second superdivision model, so that the second superdivision model with smaller parameter quantity can be used for representing the first superdivision model with larger parameter quantity, and the second superdivision model is applied to the terminal equipment to perform the image superdivision task, thereby ensuring that the second superdivision model and the first superdivision model have the same processing effect. Compared with the method for configuring the superdivision model with a simple structure on the terminal equipment, the method provided by the embodiment of the application enables the terminal equipment to configure the superdivision model with larger original model capacity, improves the model capacity of the superdivision model on the terminal equipment, improves the quality of the superdivision image obtained by the superdivision model, and improves the utilization rate of computing resources on the terminal equipment.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the description of the embodiments will be briefly described below, and it will be obvious to those skilled in the art that other drawings can be obtained from these drawings without inventive effort.
FIG. 1 is a schematic diagram of the architecture of a data processing system provided by an exemplary embodiment of the present application;
FIG. 2 is a flow chart of a method for processing data according to an exemplary embodiment of the present application;
FIG. 3 is a flow chart of another data processing method according to an exemplary embodiment of the present application;
FIG. 4A is a schematic diagram of a first superdivision model according to an exemplary embodiment of the present application;
FIG. 4B is a schematic diagram of a feature roughing module and a re-parameterization process provided by an exemplary embodiment of the present application;
FIG. 4C is a schematic illustration of a first process sub-module and a re-parameterization process provided by an exemplary embodiment of the present application;
FIG. 4D is a schematic diagram of a second process sub-module and re-parameterization process provided by an exemplary embodiment of the present application;
FIG. 4E is a schematic diagram of a third process sub-module according to an exemplary embodiment of the present application;
FIG. 5 is a flow chart of another data processing method according to an exemplary embodiment of the present application;
FIG. 6 is a schematic block diagram of a data processing apparatus provided in accordance with an exemplary embodiment of the present application;
fig. 7 is a schematic block diagram of a computer device provided in an exemplary embodiment of the present application.
Detailed Description
The following description of the embodiments of the present application will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present application, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application.
It should be noted that the descriptions of "first," "second," and the like in the embodiments of the present application are for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a technical feature defining "first", "second" may include at least one such feature, either explicitly or implicitly.
The embodiment of the application can be applied to various fields or scenes such as cloud computing, cloud internet of things, cloud games, artificial intelligence, vehicle-mounted scenes, intelligent traffic, auxiliary driving, image super-resolution processing and the like, and a few typical fields or scenes are described below.
Cloud computing (closed computing) refers to the delivery and usage mode of an IT infrastructure, meaning that required resources are obtained in an on-demand, easily scalable manner through a network; generalized cloud computing refers to the delivery and usage patterns of services, meaning that the required services are obtained in an on-demand, easily scalable manner over a network. Such services may be IT, software, internet related, or other services. Cloud Computing is a product of fusion of traditional computer and network technology developments such as Grid Computing (Grid Computing), distributed Computing (Distributed Computing), parallel Computing (Parallel Computing), utility Computing (Utility Computing), network storage (Network Storage Technologies), virtualization (Virtualization), load balancing (Load balancing), and the like. With the development of the internet, real-time data flow and diversification of connected devices, and the promotion of demands of search services, social networks, mobile commerce, open collaboration and the like, cloud computing is rapidly developed. Unlike the previous parallel distributed computing, the generation of cloud computing will promote the revolutionary transformation of the whole internet mode and enterprise management mode in concept. According to the method and the device for obtaining the data, the data such as the sample image, the image to be processed, the initial superdivision model, the first superdivision model and the second superdivision model can be stored on the cloud server, and when the different data are needed to be used, the data can be directly obtained on the cloud server, so that the data obtaining speed is greatly improved.
The Cloud IOT aims to connect information perceived by sensing equipment in the traditional IOT and accepted instructions into the Internet, networking is truly realized, mass data storage and operation are realized through a Cloud computing technology, the current running states of all 'objects' are perceived in real time due to the fact that the things are connected with each other, a large amount of data information can be generated in the process, how to collect the information, how to screen useful information in the mass information and make decision support for subsequent development, and the Cloud is a key problem affecting the development of the IOT, and the Internet of things Cloud based on Cloud computing and Cloud storage technology is also a powerful support for the technology and application of the IOT.
Cloud gaming (Cloud gaming), which may also be referred to as game on demand, is an online gaming technology based on Cloud computing technology. Cloud gaming technology enables lightweight devices (thin clients) with relatively limited graphics processing and data computing capabilities to run high quality games. In a cloud game scene, the game is not run in a player game terminal, but is run in a cloud server, the cloud server renders the game scene into a video and audio stream, and the video and audio stream is transmitted to the player game terminal through a network. The player game terminal does not need to have strong graphic operation and data processing capability, and only needs to have basic streaming media playing capability and the capability of acquiring player input instructions and sending the player input instructions to the cloud server. The method is applied to cloud games, when corresponding service demands exist, the second super-score model is configured on the mobile terminal, and the second super-score model is utilized to perform super-processing on game images in cloud game services, so that super-resolution images are obtained, the quality of the game images is improved, and the game experience of users is improved.
Artificial intelligence (Artificial Intelligence, AI) is the theory, method, technique and application system that uses a digital computer or a machine controlled by a digital computer to simulate, extend and extend human intelligence, sense the environment, acquire knowledge and use the knowledge to obtain optimal results. In other words, artificial intelligence is an integrated technology of computer science that attempts to understand the essence of intelligence and to produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence, i.e. research on design principles and implementation methods of various intelligent machines, enables the machines to have functions of sensing, reasoning and decision.
The artificial intelligence technology is a comprehensive subject, and relates to the technology with wide fields, namely the technology with a hardware level and the technology with a software level. Artificial intelligence infrastructure technologies generally include, for example, sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technologies, pre-training model technologies, operation/interaction systems, mechatronics, and the like. The pre-training model is also called a large model and a basic model, and can be widely applied to all large-direction downstream tasks of artificial intelligence after fine adjustment. The artificial intelligence software technology mainly comprises a computer vision technology, a voice processing technology, a natural language processing technology, machine learning/deep learning and other directions. The scheme provided by the embodiment of the application relates to a computer vision and technical machine learning technology which belongs to an artificial intelligence technology, and the scheme is described below:
Computer Vision (CV) is a science of studying how to "look" a machine, and more specifically, to replace a human eye with a camera and a Computer to perform machine Vision such as recognition and measurement on a target, and further perform graphic processing to make the Computer process an image more suitable for human eye observation or transmission to an instrument for detection. As a scientific discipline, computer vision research-related theory and technology has attempted to build artificial intelligence systems that can acquire information from images or multidimensional data. The large model technology brings important innovation for the development of computer vision technology, and a pre-trained model in the vision fields of swin-transformer, viT, V-MOE, MAE and the like can be rapidly and widely applied to downstream specific tasks through fine tuning. Computer vision techniques typically include image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content/behavior recognition, three-dimensional object reconstruction, 3D techniques, virtual reality, augmented reality, synchronous positioning, and map construction, among others, as well as common biometric recognition techniques such as face recognition, fingerprint recognition, and others.
Machine Learning (ML) is a multi-domain interdisciplinary, involving multiple disciplines such as probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory, etc. It is specially studied how a computer simulates or implements learning behavior of a human to acquire new knowledge or skills, and reorganizes existing knowledge structures to continuously improve own performance. Machine learning is the core of artificial intelligence, a fundamental approach to letting computers have intelligence, which is applied throughout various areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, confidence networks, reinforcement learning, transfer learning, induction learning, teaching learning, and the like. The pre-training model is the latest development result of deep learning, and integrates the technology. Specifically, the method combines the technologies of image processing, artificial neural network and the like, trains an initial superdivision model by using a sample image to obtain a first superdivision model for processing, and then carries out heavy parameterization processing on a module to be processed in the first superdivision model to generate a second superdivision model so that the second superdivision model can process an image superdivision task aiming at the image to be processed on terminal equipment.
With research and advancement of artificial intelligence technology, research and application of artificial intelligence technology is being developed in various fields, such as common smart home, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned, autopilot, unmanned, digital twin, virtual man, robot, artificial Intelligence Generated Content (AIGC), conversational interactions, smart medical, smart customer service, game AI, etc., and it is believed that with the development of technology, artificial intelligence technology will be applied in more fields and with increasing importance value.
The application scenario of the present application will be described below:
image super processing is one of the important directions in the field of computer vision research, and the purpose of the image super processing is to recover a high-resolution image from a low-resolution image. In recent years, with the development of deep learning technology, many image superdivision methods based on complex superdivision models of deep learning have been proposed and exhibit good performance. However, because the computing power of the terminal device (such as the mobile terminal) is limited, the complex superdivision model cannot be well configured on the terminal device, so that the superdivision model with a simple structure is generally configured on the terminal device.
For example, when designing the superdivision model of the terminal device, considering the calculation limit of the terminal device, the number of convolution layers and the number of channels of the superdivision model are generally set smaller, the calculation amount of the corresponding superdivision model is also relatively smaller, taking the mobile phone a12 chip as an example, the calculation capability of the neural processing unit (NPU, neural Processing Unit) of the a12 chip is 5TOPS (equal to 5000 GOPS), and if the calculation amount of the superdivision model is 2 glips, and the processing speed of the superdivision model on the NPU of the a12 chip is 4ms (i.e., the time spent by reasoning about 4ms by the superdivision model each time), the calculation capability of the superdivision model can be estimated to be 2 glips (1000/4) =500 GOPS, and then the utilization rate of the calculation capability of the NPU of the superdivision model on the a12 chip is 500GOPS/5000 gops=10%, which indicates that the calculation capability of the chip of the terminal device is not fully excavated for the light-weight superdivision model of the common terminal device is lower.
Moreover, according to the rooline theory, when the superdivision model enters a calculation bottleneck region (calculation-Bound), the calculation force of the upper calculation platform can be fully utilized; when the superdivision model is in a bandwidth bottleneck region (Memory-Bound) region, even if the calculation amount of the superdivision model is small, the superdivision model in the region cannot fully utilize the calculation force of the upper calculation platform due to the bandwidth limitation of the calculation platform, so that the superdivision model is difficult to exert the whole calculation capability of the calculation platform.
By increasing the size of the convolution kernel in the superdivision model (e.g., changing the convolution kernel size from 3*3 to 7*7), the computation of the superdivision model can be increased to some extent, but increasing the size of the convolution kernel in the superdivision model also increases the parameters of the superdivision model by an order of magnitude, which reduces the processing speed of the superdivision model (which does not necessarily increase the utilization), making it difficult for the superdivision model to run on some low-performance, low-memory embedded devices. The method can only configure the superdivision model with low model capacity on the terminal equipment, and can hardly singly improve the superdivision effect of the superdivision model by improving the complexity of the superdivision model (such as improving the size of a convolution kernel in the superdivision model).
Based on this, the embodiment of the application provides a heavy parameter scheme with intensive calculation, and the heavy parameter scheme is implemented by performing heavy parameterization on a module to be processed in a first superdivision model with larger model capacity (larger corresponding parameter amount), generating a second superdivision model with smaller model capacity (smaller corresponding parameter amount), and applying the second superdivision model to terminal equipment to perform an image superdivision task of an image to be processed, so that the model capacity of the superdivision model is improved, and the quality of the superdivision image obtained by the superdivision model (namely, the superdivision effect of the superdivision model is improved). The method realizes that the computing capacity of the terminal equipment chip is fully excavated on the premise of limited model parameter storage capacity of the terminal equipment, and the second superdivision model module generated by the re-parameterization comprises a depth convolution layer, wherein the depth convolution layer is used for reducing parameter quantity required by feature processing, the size (such as 7*7) of a depth convolution kernel in the depth convolution layer is larger than that of an original convolution kernel (such as 3*3), and the receptive field of the superdivision model is increased, so that the superdivision effect of the superdivision model is further improved.
It will be appreciated that in the specific embodiments of the present application, related data of a sample image, an image to be processed, an initial superdivision model, a first superdivision model, a second superdivision model, etc. are involved, and when the above embodiments of the present application are applied to specific products or technologies, the collection, use and processing of related data are required to comply with related laws and regulations and standards of related countries and regions.
The application will be illustrated by the following examples:
with reference now to FIG. 1, a diagram illustrating an architecture of a data processing system is depicted in accordance with an illustrative embodiment of the present application. The data processing system may in particular comprise a terminal device 101 and a server 102. Wherein the terminal device 101 and the server 102 are connected through a network, for example, a local area network, a wide area network, a mobile internet, etc. The server 102 may generate a second superdivision model, and send the second superdivision model to the terminal device 101, where the terminal device 101 performs an image superdivision task for the image to be processed on the device by using the second superdivision model.
In one embodiment, the server 102 inputs the sample image into an initial superscore model, performs superprocessing on the sample image using the initial superscore model, and the initial superscore model outputs a predicted superscore image; then, the server 102 performs model parameter adjustment on the initial superminute model according to the predicted superminute image and the reference superminute image corresponding to the sample image (as in the step of fig. 1, training the initial superminute model) to obtain a first superminute model; finally, the server 102 performs a re-parameterization process on the first superdivision model (for example, on a module to be processed in the first superdivision model), so as to generate a second superdivision model. The server 102 may transmit the second superscore model to the terminal device 101, so that the terminal device 101 generates a target superscore image of the acquired image to be processed using the second superscore model.
In an embodiment, after receiving the second superdivision model sent by the server 102, the terminal device 101 may configure the second superdivision model on the device, and then, the terminal device 101 may acquire the image to be processed; and then, performing superprocessing on the image to be processed by using the configured second superdivision model to obtain a target superdivision image.
The Terminal device 101 is also called a Terminal (Terminal), a User Equipment (UE), an access Terminal, a subscriber unit, a mobile device, a user Terminal, a wireless communication device, a user agent, or a user equipment. The terminal device may be, but is not limited to, a smart home appliance, a handheld device (e.g., a smart phone, a tablet computer) with wireless communication function, a computing device (e.g., a personal computer (personal computer, PC), a vehicle-mounted terminal, a smart voice interaction device, a wearable device or other smart apparatus, etc.
The server 102 may be an independent physical server, a server cluster or a distributed system formed by a plurality of physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, a content delivery network (Content Delivery Network, CDN), basic cloud computing services such as big data and an artificial intelligence platform.
In an embodiment, the architecture of the data processing system according to the present application may further include a database, where the database is used to store data of the sample image, the predicted superminute image, the reference superminute image, the image to be processed, the target superminute image, and the like, and may also be used to store data related to the initial superminute model, the first superminute model, and the second superminute model, where the data may be recorded in the database by different database tables. For example, the database may be a database provided in the server, that is, may be a database built in or self-contained in the server; the database may also be a peripheral database connected to the server, for example, a cloud database (i.e., a database deployed in the cloud), and specifically may be deployed based on any one of a private cloud, a public cloud, a hybrid cloud, an edge cloud, and so on, so that the cloud database has different focused functions.
It will be understood that the architecture schematic diagram of the system described in the embodiment of the present application is for more clearly describing the technical solution of the embodiment of the present application, and does not constitute a limitation on the technical solution provided by the embodiment of the present application. For example, the data processing method provided by the embodiment of the present application may be performed by a server 102, but may also be performed by a server or a server cluster other than the server 102 and capable of communicating with the terminal device 101 and/or the server 102. Those of ordinary skill in the art will recognize that the number of terminal devices and servers in fig. 1 is merely illustrative. Any number of terminal devices and servers may be configured according to service implementation needs. Moreover, with the evolution of the system architecture and the appearance of new service scenarios, the technical scheme provided by the embodiment of the application is also applicable to similar technical problems. In the subsequent embodiments, the terminal device 101 will be referred to as a terminal device, and the server will be referred to as a server 102, which will not be described in detail in the subsequent embodiments.
Referring to fig. 2, which is a flowchart of a data processing method according to an exemplary embodiment of the present application, the method is applied to a server for illustration, and the method may include the following steps:
s201, performing super-division processing on the sample image by using the initial super-division model to obtain a predicted super-division image.
In the embodiment of the present application, the super processing refers to image super resolution processing, and by performing super processing on a low resolution image (LR image), a higher resolution high resolution image (SR image) is generated. The server may perform a super-resolution process on the sample image (e.g., a low resolution image) using the initial super-resolution model to obtain a predicted super-resolution image (e.g., a high resolution image).
In one embodiment, the initial superscore model is generated based on a deep neural network, such as a convolutional neural network (Convolutional Neural Network, CNN), a generate antagonism network (Generative Adversarial Network, GAN), or the like. The initial superdivision model may contain multiple network layers or processing modules for extracting image features and accurately reconstructing high resolution image details. In the embodiment of the application, the initial superdivision model can be a superdivision model which is not subjected to model training and is constructed in advance, or a superdivision model which is subjected to model training but has lower model precision, and a large amount of training data (such as a sample image and a corresponding reference superdivision image) is used for training the initial superdivision model, so that the trained initial superdivision model has the capability of carrying out detail recovery on a low-resolution image and increasing resolution, and the model precision and the superdivision effect of the superdivision model are gradually improved, thereby being capable of generating a high-quality superresolution image.
In one embodiment, after the server inputs the sample image into the initial super-division model, the initial super-division model may perform feature extraction processing on the sample image to obtain a low-frequency feature map; then, carrying out feature enhancement processing on the low-frequency feature map to obtain an enhanced feature map; then carrying out feature reconstruction processing on the enhanced feature map to obtain a reconstructed feature map; then, carrying out up-sampling treatment on the sample image to obtain an up-sampling image; and finally, performing feature fusion processing on one or more of the low-frequency feature map, the enhanced feature map and the up-sampling image to obtain a predicted super-resolution image. It should be noted that, the specific steps and the sequence of the above-mentioned superdivision process may be different according to different algorithms and structures of the superdivision model, and meanwhile, the steps of the above-mentioned feature extraction process, feature enhancement process, upsampling process and the like may be optional, which is not limited in the embodiment of the present application.
In an embodiment, the server may further perform a preprocessing operation on the original image, and the preprocessing operation may include: one or more of normalization, scaling, clipping, etc. In addition, the server may perform post-processing operation on an image obtained after the sample image is subjected to the super-processing operation, and use the image obtained after the post-processing operation as a predicted super-resolution image, so as to further improve the quality of the predicted super-resolution image (such as improving details and definition of the predicted super-resolution image), where the post-processing operation includes: one or more of denoising, sharpening, color correction, and the like.
In one embodiment, the initial superscore model includes a feature coarse processing network, a feature fine processing network, and a feature fusion network. The main function of the feature rough machining network is to extract low-dimensional feature information from a low-resolution image through rough feature reconstruction, capture global structures and rough textures to control the overall shape and content of a super-resolution result.
The feature fine processing network has the main function of improving high-frequency details and textures of the image through fine-grained feature reconstruction, and restoring detail information of the original image as much as possible.
The feature fusion network has the main functions of improving feature expression capability in scale and semanteme through deconvolution, up-sampling and feature fusion, and generating a final hyperspectral image.
S202, performing model parameter adjustment on the initial superminute model according to the predicted superminute image and the reference superminute image corresponding to the sample image to obtain a first superminute model.
In the embodiment of the application, the sample image can be a low-resolution image, the reference super-resolution image can be a high-resolution image which corresponds to the sample image and is used as a reference standard, and the initial super-resolution model is utilized to perform super-processing on the sample image, so that the expected prediction super-resolution image is as close to the reference super-resolution image as possible. The server can perform model training (including model parameter adjustment) on the initial superminute model according to the predicted superminute image and the reference superminute image corresponding to the sample image, and finally the first superminute model is obtained. By the method, the super-division effect of the first super-division model obtained through training is guaranteed, namely the quality of the super-division image obtained through the first super-division model is improved.
In an embodiment, the server may perform difference data calculation (such as loss function calculation) on the predicted super-resolution image (SR image) and the reference super-resolution image (HR image) to obtain target difference data (such as a target loss function), and then perform model tuning on the initial super-resolution model by using the reference super-resolution image corresponding to the target difference data to obtain the first super-resolution model. The Loss function may be mean square error (Mean Squared Error, MSE), cross Entropy (CE), KL Divergence (Kullback-Leibler Divergence, KL diversity), average absolute error (Mean Absolute Error, MAE), contrast Loss (contrast Loss), perceptual Loss (residual Loss), and the like, which are not limited in the embodiment of the present application.
S203, performing re-parameterization on the to-be-processed module in the first super-division model to generate a second super-division model, wherein the second super-division model is used for generating a target super-division image of the to-be-processed image.
In the embodiment of the application, the re-parameterization is a method for re-representing the weight of the network model parameters, and the weight of the network model is re-parameterized by carrying out new linear transformation on the weight of a specific layer and the output of the two network models before and after the re-parameterization is kept unchanged. By carrying out the re-parameterization processing on the first superminute model, a second superminute model with smaller parameter quantity can be used for representing the first superminute model with more parameter quantity, the second superminute model is applied to terminal equipment (such as mobile equipment and embedded equipment) to carry out the image superminute task, the calculation complexity of the second superminute model is reduced on the premise of ensuring the same processing effect as the first superminute model, and the processing performance of the second superminute model on the terminal equipment is improved. Meanwhile, the second hyper-score model obtained by re-parameterization has higher model training efficiency and higher model generalization capability, and can effectively avoid the phenomenon of over-fitting. Compared with the method for configuring the superdivision model with a simple structure on the terminal equipment, the method provided by the embodiment of the application enables the superdivision model with larger original model capacity to be configured on the terminal equipment, improves the model capacity of the superdivision model on the terminal equipment, improves the quality of the superdivision image obtained by the superdivision model, and improves the utilization rate of computing resources on the terminal equipment.
In an embodiment, the module to be processed is a part or all of a feature rough machining network and a feature fine machining network after model parameter adjustment. The module to be processed in the first superdivision model can be a module with a complex structure, and the second superdivision model with light weight can be obtained by carrying out heavy parameterization on the module to be processed in the first superdivision model. Because the second super-resolution model is obtained by performing the re-parameterization treatment on the first super-resolution model, the processing capacity and the effect of the first super-resolution model and the second super-resolution model are the same (the super-resolution images output by the two models aiming at the same low-resolution image are the same). While "model capacity" refers to the ability of a model to fit various functions, as determined by the model prior to the re-parameterization process (e.g., the first hyper-model), a model with more learnable parameters generally means a model with greater capacity to fit more complex functional relationships, but at the same time requires more computing resources and memory to operate. The embodiment of the application can simplify the complex module to be processed in the first super-division model into a simpler form by adopting the method of re-parameterization processing, and generate the second super-division model, thereby reducing the parameter quantity of the model, ensuring the model capacity of the second super-division model, enabling the second super-division model to be used on terminal equipment with limited processing capacity and improving the quality of super-division images obtained by the second super-division model.
In an embodiment, the number of parameters of the second superdivision model is smaller than the number of parameters of the first superdivision model. When the to-be-processed module in the first superdivision model is subjected to the re-parameterization processing, the generated parameter number of the second superdivision model is smaller than that of the first superdivision model, and in this case, the re-parameterization processing can reconstruct and simplify the weight matrix of the neural network in the superdivision model (such as the first superdivision model) in a matrix decomposition, orthogonalization, pruning and other modes, so that the parameter number of the superdivision model is reduced, and the complexity of the model is reduced.
In an embodiment, the parameter quantity of the second superdivision model is equal to the parameter quantity of the first superdivision model. When the to-be-processed module in the first superdivision model is subjected to the re-parameterization processing, the parameter quantity of the generated second superdivision model is equal to the parameter quantity of the first superdivision model, in this case, the re-parameterization processing can optimize the weight matrix of the neural network in the superdivision model (such as the first superdivision model) in the modes of structured pruning, quantization, low-rank decomposition and the like, and under the condition that the parameter quantity of the superdivision model is unchanged, the processing precision of the superdivision model can be improved, the calculated quantity can be reduced, and the robustness of the superdivision model can be improved.
Based on the embodiment, the application has the following beneficial effects: according to the embodiment of the application, an initial super-division model is utilized to perform super-division processing on a sample image to obtain a predicted super-division image; then performing model parameter adjustment on the initial superdivision model according to the predicted superdivision image and the reference superdivision image to obtain a first superdivision model, so that the first superdivision model has higher model precision and better superdivision effect; and performing re-parameterization processing on the module to be processed in the first superdivision model to generate a second superdivision model, so that the second superdivision model with smaller parameter quantity can be used for representing the first superdivision model with larger parameter quantity, and the second superdivision model is applied to terminal equipment to perform image superdivision tasks, thereby ensuring that the same processing effect as that of the first superdivision model. Compared with the method for configuring the superdivision model with a simple structure on the terminal equipment, the method provided by the embodiment of the application enables the terminal equipment to configure the superdivision model with larger original model capacity, improves the model capacity of the superdivision model on the terminal equipment, improves the quality of the superdivision image obtained by the superdivision model, and improves the utilization rate of computing resources on the terminal equipment.
Referring to fig. 3, which is a flowchart of a data processing method according to an exemplary embodiment of the present application, the method is applied to a server for illustration, and the method may include the following steps:
and S301, performing feature reconstruction processing on the sample image by using a feature rough machining network in the initial super-division model to obtain a first feature map.
In one embodiment, as shown in FIG. 4A, the sample image is denoted LR, the feature roughing network is denoted Stem network, and the first feature map is denoted F stem . LR may represent a size of [ N, C, W, H ]]Wherein N represents trainingIn the embodiment of the present application, let c=1, denote that the image of the Y channel of the sample image is subjected to super-processing (the processing method of the U, V channel will be described in the following embodiments), W denotes the width of the input picture, and H denotes the height of the input picture.
Firstly, a sample image (LR) performs feature reconstruction processing (such as feature extraction) on a single-channel sample image through a feature rough processing network (Stem network) to obtain a first feature map F stem Thereby realizing the expansion of the channel number, such as F stem The shape of (C) is [ N, C ] 0 ，W，H]I.e. the number of channels extends from C to C 0 。
In one embodiment, the feature roughing network comprises a feature roughing module. In the training stage, a MobileDense Block A module is adopted for realizing the Stem module, a MobileDense Block A module is subjected to a re-parameterization process (for example, a re-parameterization scheme DBB is adopted) in the reasoning stage, and is equivalently converted into a convolution layer (for example, 3 x 3conv in fig. 4A) with a convolution kernel size of 3, and an input channel and an output channel of the convolution layer are consistent with those of the MobileDense Block A module.
In one embodiment, the feature roughing module comprises a first convolution layer comprising a plurality of convolution kernels of a first size, a second convolution layer comprising a plurality of convolution kernels of a second size, and a summing module. Illustratively, the structure of the feature roughing module is shown in fig. 4B, and the first convolution layer is formed by m Blocks in fig. 4B, where the first convolution layer includes a plurality of convolution kernels of a first size, such as m Conv having a size of 3*3; the second convolution layer is made up of n Blocks in FIG. 4B, including a plurality of convolution kernels of a second size, such as n Conv of size 1*1; the summing module is as in FIG. 4BThe module corresponding to the symbol. Based on this, the above step S301 may be implemented according to the following steps:
(a1) And respectively carrying out characteristic reconstruction processing on the sample image by using a first convolution layer and a second convolution layer which are included in a characteristic rough machining module in the characteristic rough machining network to obtain first output data and second output data.
(a2) And adding and processing the first output data, the second output data and the sample image through an adding module to obtain a first characteristic diagram.
In the steps (a 1) - (a 2), the server performs feature reconstruction processing on the sample image by using a first convolution layer included in the feature rough processing module to obtain first output data (e.g. m shapes are [ N, C ] 0 ，W，H]Feature map F of (1) a0 ) And performing feature reconstruction processing on the sample image by using a second convolution layer included in the feature rough processing module to obtain second output data (e.g. N shapes are [ N, C) 0 ，W，H]Feature map F of (1) a1 ) The method comprises the steps of carrying out a first treatment on the surface of the Then, the first output data, the second output data and the sample image (the sample image is the input data of the feature rough machining module) are added and processed by a adding module included in the feature rough machining module to obtain a first feature map F stem (the first feature map is output data of the feature roughing module), the first feature map F stem The shape of (C) is [ N, C ] 0 ，W，H]。
The first convolution layer includes a plurality of convolution kernels of a first size (e.g., conv of m 3*3) for extracting spatial information of an input sample image (shape is [ N,1, w, h ]), and by applying the plurality of convolution kernels of the first size, a wider range of receptive fields can be captured, so that the obtained feature map can contain richer image detail information and structure information. The second convolution layer includes a plurality of convolution kernels of a second size (e.g., conv of n 1*1) for increasing the number of channels of the feature map, and Conv of 1*1 is referred to as a point-by-point convolution (pointwise convolution) which allows for linear combination and nonlinear transformation between channels, and through the point-by-point convolution, more channel interactions and feature transformations can be introduced to enhance feature expression. The adding module is used for adding operation, and the adding operation is favorable for fusing different types of characteristic information, so that the diversity and the expression capacity of the characteristics are further improved. And adding the characteristic image obtained by the first convolution layer, the characteristic image obtained by the second convolution layer and the sample image element by element through an adding module, so that output data of the characteristic rough machining module, namely the first characteristic image, can be obtained.
Based on the related description in step S301, the structure and the processing procedure of the feature rough machining module are as follows: the characteristic rough machining module comprises a first convolution layer, a second convolution layer and a summation module, wherein the first convolution layer comprises a plurality of convolution kernels of a first size, the second convolution layer comprises a plurality of convolution kernels of a second size, input data of the characteristic rough machining module are processed through the first convolution layer and the second convolution layer respectively to obtain first output data and second output data, and the summation module is used for adding and processing the first output data, the second output data and the input data to obtain output data of the characteristic rough machining module.
S302, performing feature reconstruction processing on the first feature map by using a feature fine processing network in the initial super-division model to obtain a second feature map.
In one embodiment, as shown in fig. 4A, the feature finishing network includes one or more feature finishing modules, each feature finishing module including a first finishing module and a second finishing module, an output of the first finishing module being connected to an input of the second finishing module, the first finishing module including a first finishing sub-module and a second finishing sub-module, the second finishing module including a first finishing sub-module and a third finishing sub-module. The characteristic fine machining network is marked as a mobileDense network, the characteristic fine machining module is marked as a MobileDense Stage module, the first fine machining module is marked as MobileDense Basic Block, the second fine machining module is marked as MobileDense ConvFNN Block, the first machining sub-module is marked as MobileDense Block B, the second machining sub-module is marked as MobileDense Block C, the third machining sub-module is marked as ConvFFN, the first fine machining module is formed by cascading the first machining sub-module and the second machining sub-module (corresponding activation layer activation can be added after the first machining sub-module and after the second machining sub-module respectively), and the second fine machining module is formed by cascading the first machining sub-module and the third machining sub-module. In addition, the sample image is denoted as LR, the feature preform network is denoted as Stem network, and the first feature map is denoted as F stem The second feature map is denoted as F dense 。
First characteristic diagram F stem Processing feature extraction through a feature fine processing network (one or more feature fine processing modules in turn) to obtain a second feature map F dense Second characteristic diagram F dense Shape and first feature map F of (1) stem The shape of (C) is consistent and is [ N, C ] 0 ，W，H]。
In the embodiment of the present application, the step S302 will be described by taking the case that the feature fine processing network includes one feature fine processing module as an example, and when the feature fine processing network includes a plurality of feature fine processing modules, the processing flow that the feature fine processing network includes one feature fine processing module may be referred to, which is not described in detail in the embodiment of the present application. Based on this, the above step S302 may be implemented according to the following steps:
(b1) And performing feature reconstruction processing on the first feature map by using a first processing sub-module in the first feature fine processing network to obtain first intermediate data.
Wherein the server uses the first processing submodule MobileDense Block B to generate a first feature map F stem And performing characteristic reconstruction processing to obtain first intermediate data.
In one embodiment, the first processing submodule includes a third convolution layer including a third size of depth convolution kernels, a fourth convolution layer including a plurality of fourth size of depth convolution kernels, a fifth convolution layer including a plurality of fifth size of depth convolution kernels, and a summation module. Illustratively, the first processing sub-module is configured as shown in fig. 4C, and the third convolution layer is formed by k×k DW Conv in fig. 4C, and includes a depth convolution kernel of a third size, such as 1 DW Conv having a size of k×k (e.g., K may be 7); the fourth convolution layer is composed of m Blocks in fig. 4C, and includes a plurality of depth convolution kernels of a fourth size, such as m DW Conv of size 3*3; the fifth convolution layer is composed of n Blocks in fig. 4C, and includes a plurality of depth convolution kernels of a fifth size, such as n DW Conv of size 1*1; the summing module is as in FIG. 4C The module corresponding to the symbol. Based on this, the above step (b 1) may be implemented according to the following steps:
(b11) And respectively carrying out characteristic reconstruction processing on the first characteristic map by utilizing a third convolution layer, a fourth convolution layer and a fifth convolution layer in the first processing submodule to obtain third output data, fourth output data and fifth output data.
(b12) And adding the third output data, the fourth output data, the fifth output data and the first feature map through an adding module to obtain first intermediate data.
In the steps (b 11) - (b 12), the server performs feature reconstruction processing on the first feature map by using a third convolution layer included in the first processing submodule to obtain third output data (e.g. 1 shape is [ N, C) 0 ，W，H]Feature map F of (1) b0 ) And performing feature reconstruction processing on the first feature map by using a fourth convolution layer included in the first processing submodule to obtain fourth output data (e.g. m shapes are [ N, C) 0 ，W，H]Feature map F of (1) b1 ) And performing feature reconstruction processing on the first feature map by using a fifth convolution layer included in the first processing submodule to obtain fifth output data (e.g. N shapes are N, C 0 ，W，H]Feature map F of (1) b2 ) The method comprises the steps of carrying out a first treatment on the surface of the Then, the third output data, the fourth output data, the fifth output data and the first feature map (the first feature map is the input data of the first processing submodule) are subjected to addition processing by an addition module included in the first processing submodule to obtain first intermediate data (the first intermediate data is the output data of the first processing submodule), and the shape of the first intermediate data is [ N, C 0 ，W，H]。
After the first machining submodule outputs the first intermediate data, the first intermediate data may be activated by a nonlinear Activation function, for example, RELU, which may also be the Activation module in fig. 4A. The nonlinear activation function is used for carrying out nonlinear transformation on the input, so that the neural network can fit any complex nonlinear mapping relation, the neural network can better process nonlinear modes such as noise, image edge detection and the like, meanwhile, the activation function can also limit the output range of a model, the problem of gradient explosion or disappearance is avoided, and the robustness and stability of the neural network are improved. The nonlinear activation functions include Sigmoid, reLU, tanh, leakyReLU, and each activation function has its own advantages and disadvantages and usage scenarios, which are not limited by the embodiments of the present application. Based on this, when implementing step (b 1), the server may perform feature reconstruction processing on the first feature map by using the first processing sub-module in the first feature fine processing network to obtain output data of the first processing sub-module, then perform a nonlinear Activation function on the output data of the first processing sub-module (for example, input the output data of the first processing sub-module into the Activation module in fig. 4A), and then use the data processed by the nonlinear Activation function as first intermediate data, and meanwhile, refer to the above structure and refer to the structure in fig. 4A by using the second super-division model obtained by the re-parameterization processing.
In one embodiment, the third convolution layer includes a depth convolution kernel of a third size (e.g., 1 k×k DW Conv), and a larger depth convolution kernel (e.g., k=7 depthwise convolution kernel) may be used by the third convolution layer to further increase the receptive field size and capture broader context information, which may help to extract more comprehensive semantic features. The fourth convolution layer comprises a plurality of depth convolution kernels (such as DW Conv of m 3*3) with a fourth size, and the nonlinear expression capability and the receptive field size of the network can be increased through multiple 3x3 depthwise convolutions, so that the extraction of spatial information and local features in the input feature image is facilitated. The fifth convolution layer comprises a plurality of depth convolution kernels (such as DW Conv of n 1*1) with a fifth size, and the number of channels can be reduced through a plurality of 1x1 depthwise convolutions, so that the computational complexity is reduced, and nonlinear feature mapping is introduced, thereby helping to learn higher-level feature representation and enhancing the expression capability of the network. The adding module is used for adding operation, and the adding operation is favorable for fusing different types of characteristic information, so that the diversity and the expression capacity of the characteristics are further improved. And adding the characteristic diagram obtained by the third convolution layer, the characteristic diagram obtained by the fourth convolution layer, the characteristic diagram obtained by the fifth convolution layer and the first characteristic diagram element by element through an adding module, so as to obtain output data of the first processing sub-module, namely first intermediate data.
In one embodiment, the first processing sub-module (MobileDense Block B) is subjected to a re-parameterization (e.g., using a re-parameterization scheme DBB) in the inference stage, and equivalently transformed into a deep convolutional layer (e.g., K DW Conv in fig. 4A) with a convolutional kernel size of K, where the input channels and output channels of the convolutional layer are consistent with the MobileDense Block B module.
(b2) And performing feature reconstruction processing on the first intermediate data by using a second processing sub-module in the first feature fine processing network to obtain an intermediate feature map.
The server performs feature reconstruction processing on the first intermediate data by using the second processing sub-module MobileDense Block C, so as to obtain an intermediate feature map.
In one embodiment, the second processing submodule includes a sixth convolution layer including a plurality of convolution kernels of a sixth size. Illustratively, the second processing sub-module is structured as shown in fig. 4D, and the sixth convolution layer is formed by n Blocks in fig. 4D, and includes a plurality of convolution kernels of a sixth size, such as n Conv of size 1*1; the summing module is as in FIG. 4DThe module corresponding to the symbol. Based on this, the above step (b 2) may be implemented according to the following steps:
(b21) And performing characteristic reconstruction processing on the first intermediate data by using a sixth convolution layer in the second processing submodule to obtain sixth output data.
(b22) And adding the sixth output data and the first intermediate data through an adding module to obtain an intermediate feature map.
In the steps (b 21) - (b 22), the server uses the sixth convolution layer included in the second processing submodule to process the first intermediate data (shape is [ N, C) 0 ，W，H]) Performing characteristic reconstruction processing to obtain sixth output data (N shapes are N, C 0 ，W，H]Feature map F of (1) c0 ) The method comprises the steps of carrying out a first treatment on the surface of the Then, the sixth output data and the first intermediate data (the first intermediate data is the input data of the second processing sub-module) are added and processed by a adding module included in the second processing sub-module, so as to obtain an intermediate feature map (the intermediate feature map is the output data of the second processing sub-module), wherein the shape of the intermediate feature map is [ N, C ] 0 ，W，H]。
After the second processing sub-module outputs the intermediate feature map, it may be activated by a nonlinear Activation function, such as RELU, which may also be the Activation module in fig. 4A. Based on this, when implementing step (b 2), the server may perform feature reconstruction processing on the first intermediate data by using the second processing sub-module in the first feature fine processing network, to obtain output data of the second processing sub-module, then perform a nonlinear Activation function on the output data of the second processing sub-module (for example, input the output data of the second processing sub-module into the Activation module in fig. 4A), and then use the data processed by the nonlinear Activation function as an intermediate feature map.
In one embodiment, the sixth convolution layer includes a plurality of convolution kernels of a sixth size (e.g., conv of n 1*1) to enhance the representational capacity of the feature map by changing the number of channels without changing the size of the feature map. The convolution kernel of 1*1 is a lightweight convolution operation that combines the features on each channel linearly and can learn the correlation between the channels. The adding module is used for adding operation, and the adding operation is favorable for fusing different types of characteristic information, so that the diversity and the expression capacity of the characteristics are further improved. The adding operation can be realized through Skip connection, and by fusing the sixth output data and the first intermediate data, the network can be ensured to still keep low-level detail information in the deep feature extraction process, and the problem of gradient disappearance or information loss is avoided. And adding the feature map obtained by the sixth convolution layer and the first intermediate data element by element through an adding module to obtain output data of the second processing submodule, namely the intermediate feature map.
In one embodiment, the second processing sub-module (MobileDense Block C) is subjected to a re-parameterization (e.g., using a re-parameterization scheme DBB) during the inference phase, and equivalently transformed into a convolutional layer (e.g., 1 x 1conv in fig. 4D) with a convolutional kernel size of 1*1, where the input channels and output channels of the convolutional layer are consistent with those of the MobileDense Block C module.
(b3) And performing feature reconstruction processing on the intermediate feature map by using a first processing submodule in the second feature fine processing network to obtain second intermediate data.
In the specific implementation manner of step (b 3), please refer to the related description of step (b 1), and the embodiment of the present application is not repeated.
In an embodiment, the first processing submodule MobileDense Block B in the first feature refinement network may have different re-parameterization treatments performed by the two first processing submodules in the reasoning stage compared to the first processing submodule MobileDense Block B in the second feature refinement network, for example, the first processing submodule in the first feature refinement network is re-parameterized to be 7×7dwconv (where K is 7), and the first processing submodule in the second feature refinement network is re-parameterized to be 3×3dwconv (where K is 3), so that the model processing effect is improved by increasing the size of a part of convolution kernels (such as 7×7dwconv) on the premise of ensuring that the number of model parameters is small.
(b4) And performing feature reconstruction processing on the second intermediate data by using a third processing sub-module in the second feature fine processing network to obtain a second feature map.
And the server performs feature reconstruction processing on the second intermediate data by using a third processing submodule ConvFFN to obtain a second feature map. By applying ConvFFN, more flexible and complex information updating can be realized. Unlike the conventional FFN, convFFN can realize information exchange among voxels, and can improve the accuracy of the second feature map, so that the superdivision effect of the second superdivision model obtained based on the training of the second feature map is improved.
In an embodiment, the third processing module includes a first feature extraction module, a second feature extraction module, and a summation module, the first feature extraction module includes a seventh featureThe device comprises a convolution layer, a channel transformation layer and an activation layer, wherein the seventh convolution layer comprises a depth convolution kernel of a seventh size, the second feature extraction module comprises an eighth convolution layer, and the eighth convolution layer comprises a convolution kernel of the eighth size. As shown in fig. 4E, the structure of the third processing submodule is exemplary, and the first feature extraction module is formed by 7×7dw Conv, 1×1conv, GELU (activation function), and 1×1conv in fig. 4D; the seventh convolution layer may refer to 7×7dw Conv above; the channel conversion layer may refer to a first 1×1conv and a second 1×1conv, where the first 1×1conv is used for expanding the channel number, for example, the input channel number is C 0 The number of output channels is C 0 * ffn _ratio, ffn _ratio is the channel transform parameter (ffn _ratio may take an integer greater than 1, e.g. 4), and the second 1×1Conv is used for compression of the channel number, e.g. the input channel number is C 0 * ffn _ratio, number of output channels C 0 Through the channel conversion layer, more flexible and complex information updating is realized; the activation layer may be the gel, through which information exchange between voxels is achieved; the summing module is as in FIG. 4EThe module corresponding to the symbol. Based on this, the above step (b 4) may be implemented according to the following steps:
(b41) And respectively carrying out feature reconstruction processing on the second intermediate data by using the first feature extraction module and the second feature extraction module in the third processing sub-module to obtain seventh output data and eighth output data.
(b42) And adding the seventh output data and the eighth output data through an adding module to obtain a second characteristic diagram.
In the steps (b 41) - (b 42), the second intermediate data is input data of the third processing submodule, and the server uses the first feature extraction module included in the third processing submodule to extract the second intermediate data (shape is [ N, C) 0 ，W，H]) Performing characteristic reconstruction processing to obtain seventh output data (e.g. 1 shape is [ N, C ] 0 ，W，H]Feature map F of (1) d0 ) And utilizing a second feature extraction module pair comprised by the third machining sub-moduleThe second intermediate data is subjected to characteristic reconstruction processing to obtain eighth output data (e.g. 1 shape is [ N, C) 0 ，W，H]Feature map F of (1) d1 ) The method comprises the steps of carrying out a first treatment on the surface of the Then, the seventh output data and the eighth output data are added and processed through an adding module included in the third processing sub-module to obtain a second feature map (the second feature map is the output data of the third processing sub-module), and the shape of the second feature map is [ N, C ] 0 ，W，H]。
In one embodiment, in the training phase, since the third processing submodule ConvFFN includes a GELU, which is itself a nonlinear function activation function, the GELU is not re-parameterized, so that the model design is simplified, and the training and reasoning process of the model is more efficient.
Based on the related description in step S302, the structure and the processing procedure of the first processing sub-module are as follows: the first processing submodule comprises a third convolution layer, a fourth convolution layer, a fifth convolution layer and a summation module, wherein the third convolution layer comprises a depth convolution kernel of a third size, the fourth convolution layer comprises a plurality of depth convolution kernels of a fourth size, the fifth convolution layer comprises a plurality of depth convolution kernels of a fifth size, input data of the first processing submodule are processed through the third convolution layer, the fourth convolution layer and the fifth convolution layer respectively to obtain third output data, fourth output data and fifth output data, and the summation module is used for adding and processing the third output data, the fourth output data, the fifth output data and the input data to obtain output data of the first processing submodule.
Based on the related description in step S302, the structure and the processing procedure of the second processing sub-module are as follows: the second processing submodule comprises a sixth convolution layer and an addition module, the sixth convolution layer comprises a plurality of convolution kernels with a sixth size, input data of the second processing submodule are processed through the sixth convolution layer to obtain sixth output data, and the addition module is used for adding and processing the sixth output data and the input data to obtain output data of the second processing submodule.
Based on the related description in step S302, the structure and the processing procedure of the third processing sub-module are as follows: the third processing module comprises a first feature extraction module, a second feature extraction module and a summation module, wherein the first feature extraction module comprises a seventh convolution layer, a channel transformation layer and an activation layer, the seventh convolution layer comprises a depth convolution kernel of a seventh size, the second feature extraction module comprises an eighth convolution layer, the eighth convolution layer comprises a convolution kernel of an eighth size, input data of the third processing module are respectively processed through the first feature extraction module and the second feature extraction module to obtain seventh output data and eighth output data, and the summation module is used for adding and processing the seventh output data and the eighth output data to obtain output data of the third processing module.
S303, performing deconvolution processing on the second feature map by using a feature fusion network in the initial superscore model to obtain a third feature map, performing up-sampling processing on the sample image to obtain a fourth feature map, and performing feature fusion processing on the third feature map and the fourth feature map to obtain a predicted superscore image.
In the embodiment of the application, the deconvolution process can restore the detail loss caused by compression or downsampling, so that a clearer and more detailed image can be obtained; the up-sampling process may amplify the low resolution image to a higher resolution, and the server may effectively increase the resolution of the corresponding image by up-sampling the sample image and deconvoluting the second feature map. And the third feature image and the fourth feature image are fused by utilizing a feature fusion network, so that the quality of the image can be improved, and the fused predicted super-resolution image can better reserve and enhance the features such as details, textures, edges and the like of the input image, so that the predicted super-resolution image is more natural and real in vision.
In an embodiment, the feature fusion network includes a deconvolution module, an up-sampling module, and a summation module, where the deconvolution module is used to perform deconvolution processing, the up-sampling module is used to perform up-sampling processing, and the summation module is used to perform feature fusion processing.
Wherein the deconvolution module is marked as a Deconv module, the up-sampling module is marked as an UpSample module, and the second feature map F dense The deconvolution module is used for deconvolution to obtain a third characteristic diagram (the third characteristic diagram is marked as F) deconv The shape is [ N,1,2W,2H ]]) The method comprises the steps of carrying out a first treatment on the surface of the The sample image LR is subjected to up-sampling processing by an up-sampling module to obtain a fourth feature map (the fourth feature map is Fup, and shape is [ N,1,2W,2H ]]) The method comprises the steps of carrying out a first treatment on the surface of the Then, the third feature map and the fourth feature map are subjected to feature fusion processing (e.g., addition operation) to obtain a predicted super-resolution image (the predicted super-resolution image is denoted as SR, and shape is [ N,1,2W, 2H)]). The upsampling process may adopt Bilinear interpolation (Bilinear), deconvolution, inverse pooling, etc.
The steps S301 to S303 describe in detail the step of obtaining the predicted super-resolution image (i.e., step S201) by performing the super-resolution processing on the sample image using the initial super-resolution model. The following describes in detail the method of step S304-S306, which is to perform model tuning on the initial superdivision model according to the predicted superdivision image and the reference superdivision image corresponding to the sample image, to obtain the first superdivision model (i.e. step S202):
S304, acquiring a reference superminute image corresponding to the sample image.
In one embodiment, the sample image is obtained by performing a reduced resolution process on the reference super-resolution image. The server first acquires a reference super-resolution image (which is a high-resolution image), and then performs a resolution reduction process (such as a sampling process) on the reference super-resolution image to obtain a sample image (which is a low-resolution image). The server takes the sample image as input data of the initial superdivision model to carry out superdivision processing to obtain a predicted superdivision image, takes the reference superdivision image as supervision data, and hopes that the predicted superdivision image is consistent with the reference superdivision image in effect. The method can ensure the accuracy of the supervision data when the initial superscore model is trained, further ensure the model training effect and improve the superscore effect of the first superscore model obtained by training.
S305, determining target difference data between the predicted superdivision image and the reference superdivision image.
S306, adjusting model parameters of the initial superdivision model according to the target difference data, and determining the initial superdivision model with the adjusted model parameters as a first superdivision model.
In the steps S305-S306, the server may perform difference data calculation (such as loss function calculation) on the predicted super-resolution image (SR image) and the reference super-resolution image (HR image) to obtain target difference data (such as a target loss function), and then perform model parameter adjustment on the initial super-resolution model by using the reference super-resolution image corresponding to the target difference data to obtain the first super-resolution model. The Loss function may be mean square error (Mean Squared Error, MSE), cross Entropy (CE), KL Divergence (Kullback-Leibler Divergence, KL diversity), average absolute error (Mean Absolute Error, MAE), contrast Loss (contrast Loss), perceptual Loss (residual Loss), and the like, which are not limited in the embodiment of the present application.
The following describes in detail the method of step (i.e. step S203) of generating the second superdivision model by performing the re-parameterization on the module to be processed in the first superdivision model through steps S307-S308:
s307, performing re-parameterization on the module to be processed in the first superdivision model to obtain a re-parameterized module.
S308, generating a second superdivision model according to the module which does not carry out the re-parameterization processing and the re-parameterization module in the first superdivision model.
In the above steps S307-S308, the first superdivision model is subjected to the re-parameterization, so that the second superdivision model with smaller parameter quantity can be used to represent the first superdivision model with larger parameter quantity, and the second superdivision model is applied to the terminal device (such as the mobile device and the embedded device) to perform the image superdivision task, so that the computation complexity of the second superdivision model is reduced and the processing performance of the second superdivision model on the terminal device is improved on the premise of ensuring that the second superdivision model has the same processing effect as the first superdivision model. Meanwhile, the second hyper-score model obtained by re-parameterization has higher model training efficiency and higher model generalization capability, and can effectively avoid the phenomenon of over-fitting. Compared with the method for configuring the superdivision model with a simple structure on the terminal equipment, the method provided by the embodiment of the application enables the superdivision model with larger original model capacity to be configured on the terminal equipment, improves the model capacity of the superdivision model on the terminal equipment, improves the quality of the superdivision image obtained by the superdivision model, and improves the utilization rate of computing resources on the terminal equipment.
In an embodiment, the module to be processed is a feature rough machining module after model parameter adjustment, a first machining sub-module included in the first fine machining module, and a second machining sub-module and a first machining sub-module included in the second fine machining module.
The implementation of the feature rough machining module adopts a MobileDense Block A module, the feature rough machining module is subjected to re-parameterization treatment (for example, adopts a re-parameterization scheme DBB) in an inference stage, and is equivalently converted into a convolution layer (3 x 3conv in fig. 4B) with a convolution kernel size of 3, and an input channel and an output channel of the convolution layer are consistent with the feature rough machining module, so that the same treatment effect as that of the first superdivision model is ensured.
The first processing sub-module is realized by adopting a MobileDense Block B module, the first processing sub-module is subjected to heavy parameterization treatment (for example, heavy parameter scheme DBB) in an inference stage and is equivalently converted into a deep convolution layer (for example, K DW Conv in fig. 4C) with a convolution kernel size of K, and an input channel and an output channel of the convolution layer are kept consistent with the first processing sub-module, so that the same treatment effect as the first processing sub-module is ensured.
The second processing sub-module is realized by adopting a MobileDense Block C module, and the second processing sub-module is subjected to heavy parameterization treatment (for example, heavy parameter scheme DBB) in an inference stage and is equivalently converted into a convolution layer (1 x 1Conv in fig. 4D) with a convolution kernel size of 1*1, and an input channel and an output channel of the convolution layer are consistent with those of the second processing sub-module, so that the same treatment effect as that of the second processing sub-module is ensured.
In an embodiment, the re-parameterization module includes a depth convolution layer, where the depth convolution layer is used to reduce the parameter amount required by feature processing, and the depth convolution layer includes a depth convolution kernel, where the depth convolution kernel is obtained by performing re-parameterization on an original convolution kernel in the module to be processed, and the size of the depth convolution kernel is greater than that of the original convolution kernel.
Where, the depth convolution kernel may refer to k×k DW Conv, unlike the conventional convolution kernel (e.g., k×k Conv), the depth convolution kernel performs convolution computation on each input channel independently, rather than in whole. Specifically, DW conv maintains the number of channels of the input feature map unchanged, and performs a convolution operation on each channel of the input feature map by using a convolution kernel equal to the number of channels, which means that each input channel has a corresponding convolution kernel for extracting features of the channel, and then combines the feature maps obtained for each channel to form an output feature map. Therefore, the depth convolution kernel can effectively reduce the parameter quantity and the calculation amount of the model, improve the processing efficiency of the model, and is particularly suitable for scenes with limited resources such as mobile equipment and the like.
The deep convolution kernel is obtained by performing a re-parameterization treatment on an original convolution kernel in the to-be-treated module, taking a first processing sub-module (MobileDense Block B) in the first characteristic fine processing network as an example, the first processing sub-module can be regarded as the original convolution kernel, and a module (k×k DW Conv) obtained by performing the re-parameterization treatment on the first processing sub-module can be regarded as the deep convolution kernel. Assuming that the original convolution kernel has a size of 3×3×8×8, 3*3 denotes the height and width of the convolution kernel, i.e., the size of the convolution kernel is 3×3;8 x 8 denotes the number of input channels, which refers to the number of channels of the input feature map of the convolutional layer, and the number of output channels, which refers to the number of channels of the output feature map of the convolutional layer. Then the computational complexity of the original convolution kernel can be considered as 3×3×8×8=576 units.
The depth convolution kernel is a separable convolution, and the number of parameters can be reduced by the block convolution, so that the size can be set to k×k×8. When the size of the depth convolution kernel is greater than that of the original convolution kernel, taking k=7 as an example, the computational complexity corresponding to the depth convolution kernel can be regarded as 7×7× 8=392 units. It can be found that, compared to the original convolution kernel, the depth convolution kernel has a larger size (7×7>3×3), but the computation complexity of the depth convolution kernel is lower (392 < 576), so that the receptive field of the second super-division model is improved, so that the second super-division model can capture the context information and texture features in the input image more comprehensively, thereby improving the effect of super-resolution reconstruction. Compared with the method for configuring the superdivision model with a simple structure on the terminal equipment, the method provided by the embodiment of the application enables the superdivision model with larger original model capacity to be configured on the terminal equipment, improves the model capacity of the superdivision model on the terminal equipment, improves the quality of the superdivision image obtained by the superdivision model, and improves the utilization rate of computing resources on the terminal equipment.
In an embodiment, in the super-resolution task, the Y, U, V three channels of the sample image may be processed separately or may be processed in combination. Generally, only the luminance channel (Y channel) needs to be super-resolution processed, because color information is not sensitive to human eyes, and the resolution of the chrominance channel (U, V channel) is lower than that of the luminance channel, so after the super-resolution processing of the Y channel is performed, the up-sampling processing of the U, V channel can be performed according to the dependency relationship between the U, V channel and the Y channel, so as to ensure the color quality of the image.
Illustratively, the server may up-sample the component images of U, V channels of the sample image to obtain a super-resolution image of the same resolution as the component images of Y channels, and then generate a final super-resolution image from the super-resolution images of Y, U, V three channels. For example, the super-division images of the Y, U, V three channels are converted into three images in an RGB format, then the three images in the RGB format are added and processed to obtain a fusion image, and then the fusion image is converted into the super-division image in a YUV format again to be used as a prediction super-division image.
It should be noted that, in the embodiment of the present application, the re-parameterization process is exemplified by a DBB scheme, where the DBB scheme provides 6 equivalent transforms, including: BN fusion, branch addition, convolution series connection, depth connection, average pooling and multi-scale operation, through the equivalent transformation, the parameter number of the super-division model can be reduced, and the precision of the super-division model can be improved. In addition, the re-parameterization process may also use a re-parameterization technique of Reparameterization Trick, local Reparameterization Trick, implicit Reparameterization, etc., which is not described in detail in the embodiment of the present application.
Referring to fig. 5, which is a flowchart of a data processing method according to an exemplary embodiment of the present application, the method is applied to a terminal device for illustration, and the method may include the following steps:
s501, acquiring an image to be processed.
S502, performing superprocessing on the image to be processed by using the second superdivision model to obtain a target superdivision image.
In the embodiment of the application, the terminal equipment can configure the second superdivision model on the equipment, and then superprocessing the image to be processed by using the second superdivision model to obtain the target superdivision image, thereby realizing the image superdivision task. Because the second superdivision model is obtained by carrying out the re-parameterization processing based on the first superdivision model, the image superdivision effect and the capability of the second superdivision model are the same, namely the superdivision images obtained by respectively carrying out the superdivision processing on the same image are the same, and meanwhile, compared with the first superdivision model, the parameter number of the second superdivision model is smaller, and the model capacity of the superdivision model on the terminal equipment is improved and the quality of the superdivision image obtained by the superdivision model is improved by carrying out the image superdivision task on the terminal equipment by utilizing the second superdivision model. Compared with the method for configuring the superdivision model with a simple structure on the terminal equipment, the method provided by the embodiment of the application enables the superdivision model with larger original model capacity to be configured on the terminal equipment, and improves the image superdivision effect and the utilization rate of computing resources on the terminal equipment.
The second superdivision model is generated by carrying out re-parameterization treatment on a to-be-treated module in the first superdivision model, the first superdivision model is obtained by carrying out model training on an initial superdivision model by utilizing a sample image, the initial superdivision model comprises a characteristic rough machining network, a characteristic fine machining network and a characteristic fusion network, and the to-be-treated module is a part of or all of the characteristic rough machining network and the characteristic fine machining network after model parameter adjustment during model training on the initial superdivision model; the second superdivision model has a smaller number of parameters than the first superdivision model.
The specific implementation of steps S501-S502 refers to the descriptions of steps S201-S203 and steps S301-S308 in the foregoing embodiments, and are not repeated here.
Referring to fig. 6, a schematic block diagram of a data processing apparatus according to an embodiment of the present application is shown. In one embodiment, the data processing apparatus may specifically include:
the superprocessing module 601 is configured to perform superprocessing on the sample image by using an initial superprocessing model to obtain a predicted superprocessing image;
the training module 602 is configured to perform model tuning on the initial superscore model according to the predicted superscore image and a reference superscore image corresponding to the sample image, so as to obtain a first superscore model;
The re-parameterization module 603 is configured to perform re-parameterization on the to-be-processed module in the first super-division model, and generate a second super-division model, where the second super-division model is used to generate a target super-division image of the to-be-processed image;
the initial super-division model comprises a characteristic rough machining network, a characteristic fine machining network and a characteristic fusion network, and the to-be-processed module is a part of or all of the characteristic rough machining network and the characteristic fine machining network after the model is subjected to parameter adjustment; the second superdivision model has a smaller number of parameters than the first superdivision model.
Optionally, the above-mentioned super-processing module 601 is specifically configured to, when configured to perform super-division processing on a sample image by using an initial super-division model to obtain a predicted super-division image:
performing feature reconstruction processing on the sample image by using the feature rough machining network in the initial superdivision model to obtain a first feature map;
performing feature reconstruction processing on the first feature map by using the feature fine processing network in the initial super-division model to obtain a second feature map;
and performing deconvolution processing on the second feature map by using the feature fusion network in the initial superscore model to obtain a third feature map, performing up-sampling processing on the sample image to obtain a fourth feature map, and performing feature fusion processing on the third feature map and the fourth feature map to obtain a predicted superscore image.
Optionally, the feature rough machining network includes a feature rough machining module, the feature fine machining network includes one or more feature fine machining modules, each feature fine machining module includes a first fine machining module and a second fine machining module, an output of the first fine machining module is connected with an input of the second fine machining module, the first fine machining module includes a first machining sub-module and a second machining sub-module, the second fine machining module includes the first machining sub-module and a third machining sub-module, the feature fusion network includes a deconvolution module, an up-sampling module, and a summation module, the deconvolution module is used for performing the deconvolution process, the up-sampling module is used for performing the up-sampling process, and the summation module is used for performing the feature fusion process; the to-be-processed module is a characteristic rough machining module after the model is adjusted, a first machining sub-module included in the first fine machining module, a second machining sub-module and a first machining sub-module included in the second fine machining module.
Optionally, the feature rough machining module includes a first convolution layer, a second convolution layer, and a summation module, where the first convolution layer includes a plurality of convolution kernels of a first size, and the second convolution layer includes a plurality of convolution kernels of a second size;
The above-mentioned super-processing module 601 is specifically configured to, when performing feature reconstruction processing on a sample image by using the above-mentioned feature rough processing network in the initial super-division model to obtain a first feature map:
respectively carrying out characteristic reconstruction processing on a sample image by using the first convolution layer and the second convolution layer which are included by the characteristic rough machining module in the characteristic rough machining network to obtain first output data and second output data;
and adding the first output data, the second output data and the sample image through the adding module to obtain a first characteristic diagram.
Optionally, the feature fine processing network includes a feature fine processing module; the super-processing module 601 is configured to perform feature reconstruction processing on the first feature map by using the feature refinement network in the initial super-division model to obtain a second feature map, where the second feature map is specifically configured to:
performing feature reconstruction processing on the first feature map by using the first processing submodule in the first feature fine processing network to obtain first intermediate data;
performing feature reconstruction processing on the first intermediate data by using the second processing submodule in the first feature fine processing network to obtain an intermediate feature map;
Performing feature reconstruction processing on the intermediate feature map by using the first processing submodule in the second feature fine processing network to obtain second intermediate data;
and performing feature reconstruction processing on the second intermediate data by using the third processing sub-module in the second feature fine processing network to obtain a second feature map.
Optionally, the first processing submodule includes a third convolution layer, a fourth convolution layer, a fifth convolution layer and a summation module, where the third convolution layer includes a depth convolution kernel of a third size, the fourth convolution layer includes a plurality of depth convolution kernels of a fourth size, and the fifth convolution layer includes a plurality of depth convolution kernels of a fifth size;
the super-processing module 601 is configured to, when performing feature reconstruction processing on the first feature map by using the first processing sub-module in the first feature fine processing network, obtain first intermediate data, specifically configured to:
respectively carrying out characteristic reconstruction processing on the first characteristic map by using the third convolution layer, the fourth convolution layer and the fifth convolution layer in the first processing submodule to obtain third output data, fourth output data and fifth output data;
And adding the third output data, the fourth output data, the fifth output data and the first feature map through the adding module to obtain first intermediate data.
Optionally, the second processing submodule includes a sixth convolution layer and a summation module, where the sixth convolution layer includes a plurality of convolution kernels with a sixth size;
the super-processing module 601 is configured to, when performing feature reconstruction processing on the first intermediate data by using the second processing sub-module in the first feature fine processing network to obtain an intermediate feature map, specifically:
performing feature reconstruction processing on the first intermediate data by using the sixth convolution layer in the second processing submodule to obtain sixth output data;
and adding the sixth output data and the first intermediate data through the adding module to obtain an intermediate feature map.
Optionally, the third processing module includes a first feature extraction module, a second feature extraction module, and a summation module, where the first feature extraction module includes a seventh convolution layer, a channel transform layer, and an activation layer, the seventh convolution layer includes a depth convolution kernel of a seventh size, the second feature extraction module includes an eighth convolution layer, and the eighth convolution layer includes a convolution kernel of an eighth size;
The super-processing module 601 is configured to, when performing feature reconstruction processing on the second intermediate data by using the third processing sub-module in the second feature fine processing network to obtain a second feature map, specifically:
respectively carrying out feature reconstruction processing on the second intermediate data by using the first feature extraction module and the second feature extraction module in the third processing sub-module to obtain seventh output data and eighth output data;
and adding and processing the seventh output data and the eighth output data through the adding module to obtain a second characteristic diagram.
Optionally, when the training module 602 is configured to perform model tuning on the initial superscore model according to the predicted superscore image and the reference superscore image corresponding to the sample image to obtain a first superscore model, the training module is specifically configured to:
obtaining a reference superdivision image corresponding to the sample image, wherein the sample image is obtained by performing resolution reduction processing on the reference superdivision image;
determining target difference data between the predicted superdivision image and the reference superdivision image;
and adjusting the model parameters of the initial superdivision model according to the target difference data, and determining the initial superdivision model with the adjusted model parameters as a first superdivision model.
Optionally, the re-parameterizing module 603 is configured to, when performing re-parameterization processing on the module to be processed in the first superdivision model to generate a second superdivision model, specifically:
carrying out re-parameterization treatment on the module to be treated in the first superdivision model to obtain a re-parameterized module;
generating a second superdivision model according to the module which does not carry out the re-parameterization processing in the first superdivision model and the re-parameterization module;
the re-parameterization module comprises a depth convolution layer, the depth convolution layer is used for reducing parameter quantity required by feature processing, the depth convolution layer comprises a depth convolution kernel, the depth convolution kernel is obtained by performing re-parameterization on an original convolution kernel in the module to be processed, and the size of the depth convolution kernel is larger than that of the original convolution kernel.
In another embodiment, the data processing apparatus may specifically include:
the super processing module 601 is configured to acquire an image to be processed;
the above-mentioned superprocessing module 601 is further configured to perform superprocessing on the above-mentioned image to be processed by using a second superdivision model, so as to obtain a target superdivision image;
the second superdivision model is generated by carrying out re-parameterization on a to-be-processed module in a first superdivision model, the first superdivision model is obtained by carrying out model training on an initial superdivision model by utilizing a sample image, the initial superdivision model comprises a characteristic rough machining network, a characteristic fine machining network and a characteristic fusion network, and the to-be-processed module is a part of or all of the modules in the characteristic rough machining network and the characteristic fine machining network after model adjustment during model training on the initial superdivision model; the second superdivision model has a smaller number of parameters than the first superdivision model.
It should be noted that, the functions of each functional module of the data processing apparatus according to the embodiments of the present application may be specifically implemented according to the method in the embodiments of the method, and the specific implementation process may refer to the related description of the embodiments of the method, which is not repeated herein.
Referring to fig. 7, a schematic block diagram of a computer device according to an embodiment of the present application is shown. The computer device in the present embodiment as shown in the drawings may include: a processor 701, a storage device 702, and a communication interface 703. Data interaction may take place between the processor 701, the storage device 702 and the communication interface 703.
The storage 702 may include volatile memory (RAM), such as random-access memory (RAM); the storage 702 may also include a non-volatile memory (non-volatile memory), such as a flash memory (flash memory), a Solid State Drive (SSD), etc.; the storage 702 may also include a combination of the types of memory described above.
The processor 701 may be a central processing unit (central processing unit, CPU). In one embodiment, the processor 701 may also be a graphics processor (Graphics Processing Unit, GPU). The processor 701 described above may also be a combination of a CPU and a GPU. In one embodiment, the storage device 702 is configured to store program instructions, and the processor 701 may call the program instructions to perform the following operations:
Performing superdivision processing on the sample image by using the initial superdivision model to obtain a predicted superdivision image;
performing model parameter adjustment on the initial superdivision model according to the predicted superdivision image and a reference superdivision image corresponding to the sample image to obtain a first superdivision model;
performing re-parameterization on the to-be-processed module in the first super-division model to generate a second super-division model, wherein the second super-division model is used for generating a target super-division image of the to-be-processed image;
the initial super-division model comprises a characteristic rough machining network, a characteristic fine machining network and a characteristic fusion network, and the to-be-processed module is a part of or all of the characteristic rough machining network and the characteristic fine machining network after the model is subjected to parameter adjustment; the second superdivision model has a smaller number of parameters than the first superdivision model.
Optionally, the processor 701 is specifically configured to, when configured to perform a super-division process on the sample image by using the initial super-division model, obtain a predicted super-division image:
performing feature reconstruction processing on the sample image by using the feature rough machining network in the initial superdivision model to obtain a first feature map;
performing feature reconstruction processing on the first feature map by using the feature fine processing network in the initial super-division model to obtain a second feature map;
And performing deconvolution processing on the second feature map by using the feature fusion network in the initial superscore model to obtain a third feature map, performing up-sampling processing on the sample image to obtain a fourth feature map, and performing feature fusion processing on the third feature map and the fourth feature map to obtain a predicted superscore image.
Optionally, the feature rough machining network includes a feature rough machining module, the feature fine machining network includes one or more feature fine machining modules, each feature fine machining module includes a first fine machining module and a second fine machining module, an output of the first fine machining module is connected with an input of the second fine machining module, the first fine machining module includes a first machining sub-module and a second machining sub-module, the second fine machining module includes the first machining sub-module and a third machining sub-module, the feature fusion network includes a deconvolution module, an up-sampling module, and a summation module, the deconvolution module is used for performing the deconvolution process, the up-sampling module is used for performing the up-sampling process, and the summation module is used for performing the feature fusion process; the to-be-processed module is a characteristic rough machining module after the model is adjusted, a first machining sub-module included in the first fine machining module, a second machining sub-module and a first machining sub-module included in the second fine machining module.
Optionally, the feature rough machining module includes a first convolution layer, a second convolution layer, and a summation module, where the first convolution layer includes a plurality of convolution kernels of a first size, and the second convolution layer includes a plurality of convolution kernels of a second size;
the processor 701 is configured to, when performing feature reconstruction processing on a sample image by using the feature rough processing network in the initial super-division model, obtain a first feature map, specifically:
respectively carrying out characteristic reconstruction processing on a sample image by using the first convolution layer and the second convolution layer which are included by the characteristic rough machining module in the characteristic rough machining network to obtain first output data and second output data;
and adding the first output data, the second output data and the sample image through the adding module to obtain a first characteristic diagram.
Optionally, the feature fine processing network includes a feature fine processing module; the processor 701 is specifically configured to, when performing a feature reconstruction process on the first feature map by using the feature refinement network in the initial hyperspectral model to obtain a second feature map:
Performing feature reconstruction processing on the first feature map by using the first processing submodule in the first feature fine processing network to obtain first intermediate data;
performing feature reconstruction processing on the first intermediate data by using the second processing submodule in the first feature fine processing network to obtain an intermediate feature map;
performing feature reconstruction processing on the intermediate feature map by using the first processing submodule in the second feature fine processing network to obtain second intermediate data;
and performing feature reconstruction processing on the second intermediate data by using the third processing sub-module in the second feature fine processing network to obtain a second feature map.
Optionally, the first processing submodule includes a third convolution layer, a fourth convolution layer, a fifth convolution layer and a summation module, where the third convolution layer includes a depth convolution kernel of a third size, the fourth convolution layer includes a plurality of depth convolution kernels of a fourth size, and the fifth convolution layer includes a plurality of depth convolution kernels of a fifth size;
the processor 701 is configured to, when performing a feature reconstruction process on the first feature map by using the first processing submodule in the first feature refinement network to obtain first intermediate data, specifically:
Respectively carrying out characteristic reconstruction processing on the first characteristic map by using the third convolution layer, the fourth convolution layer and the fifth convolution layer in the first processing submodule to obtain third output data, fourth output data and fifth output data;
and adding the third output data, the fourth output data, the fifth output data and the first feature map through the adding module to obtain first intermediate data.
Optionally, the second processing submodule includes a sixth convolution layer and a summation module, where the sixth convolution layer includes a plurality of convolution kernels with a sixth size;
the processor 701 is configured to, when performing feature reconstruction processing on the first intermediate data by using the second processing sub-module in the first feature fine processing network to obtain an intermediate feature map, specifically:
performing feature reconstruction processing on the first intermediate data by using the sixth convolution layer in the second processing submodule to obtain sixth output data;
and adding the sixth output data and the first intermediate data through the adding module to obtain an intermediate feature map.
Optionally, the third processing module includes a first feature extraction module, a second feature extraction module, and a summation module, where the first feature extraction module includes a seventh convolution layer, a channel transform layer, and an activation layer, the seventh convolution layer includes a depth convolution kernel of a seventh size, the second feature extraction module includes an eighth convolution layer, and the eighth convolution layer includes a convolution kernel of an eighth size;
the processor 701 is specifically configured to, when performing feature reconstruction processing on the second intermediate data by using the third processing sub-module in the second feature fine processing network to obtain a second feature map:
respectively carrying out feature reconstruction processing on the second intermediate data by using the first feature extraction module and the second feature extraction module in the third processing sub-module to obtain seventh output data and eighth output data;
and adding and processing the seventh output data and the eighth output data through the adding module to obtain a second characteristic diagram.
Optionally, when the processor 701 is configured to perform model referencing on the initial superdivision model according to the predicted superdivision image and the reference superdivision image corresponding to the sample image to obtain a first superdivision model, the processor is specifically configured to:
Obtaining a reference superdivision image corresponding to the sample image, wherein the sample image is obtained by performing resolution reduction processing on the reference superdivision image;
determining target difference data between the predicted superdivision image and the reference superdivision image;
and adjusting the model parameters of the initial superdivision model according to the target difference data, and determining the initial superdivision model with the adjusted model parameters as a first superdivision model.
Optionally, when the processor 701 is configured to perform a re-parameterization process on a module to be processed in the first superdivision model, the processor is specifically configured to:
carrying out re-parameterization treatment on the module to be treated in the first superdivision model to obtain a re-parameterized module;
generating a second superdivision model according to the module which does not carry out the re-parameterization processing in the first superdivision model and the re-parameterization module;
the re-parameterization module comprises a depth convolution layer, the depth convolution layer is used for reducing parameter quantity required by feature processing, the depth convolution layer comprises a depth convolution kernel, the depth convolution kernel is obtained by performing re-parameterization on an original convolution kernel in the module to be processed, and the size of the depth convolution kernel is larger than that of the original convolution kernel.
In another embodiment, the storage device 702 is configured to store program instructions, and the processor 701 may call the program instructions to perform the following operations:
acquiring an image to be processed;
performing superprocessing on the image to be processed by using a second superdivision model to obtain a target superdivision image;
the second superdivision model is generated by carrying out re-parameterization on a to-be-processed module in a first superdivision model, the first superdivision model is obtained by carrying out model training on an initial superdivision model by utilizing a sample image, the initial superdivision model comprises a characteristic rough machining network, a characteristic fine machining network and a characteristic fusion network, and the to-be-processed module is a part of or all of the modules in the characteristic rough machining network and the characteristic fine machining network after model adjustment during model training on the initial superdivision model; the second superdivision model has a smaller number of parameters than the first superdivision model.
In specific implementation, the processor 701, the storage device 702 and the communication interface 703 described in the embodiments of the present application may perform the implementation described in the related embodiments of the data processing method provided in fig. 2, fig. 3 or fig. 5, and may also perform the implementation described in the related embodiments of the data processing device provided in fig. 6, which are not described herein again.
In the several embodiments provided in the present application, it should be understood that the disclosed method, apparatus and system may be implemented in other manners. For example, the device embodiments described above are merely illustrative; for example, the division of the units is only one logic function division, and other division modes can be adopted in actual implementation; for example, multiple units or components may be combined or may be integrated into another system, or some features may be omitted, or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be an indirect coupling or communication connection via some interfaces, devices or units, which may be in electrical, mechanical or other form.
Furthermore, it should be noted here that: the embodiment of the present application further provides a computer readable storage medium, in which a computer program executed by the aforementioned data processing apparatus is stored, and the computer program includes program instructions, when executed by a processor, can execute the method in the corresponding embodiment of fig. 2, 3 or 5, and therefore, a detailed description will not be given here. In addition, the description of the beneficial effects of the same method is omitted. For technical details not disclosed in the embodiments of the computer-readable storage medium according to the present application, please refer to the description of the method embodiments of the present application. As an example, the program instructions may be deployed on one computer device or executed on multiple computer devices at one site or, alternatively, distributed across multiple sites and interconnected by a communication network, where the multiple computer devices distributed across multiple sites and interconnected by the communication network may constitute a blockchain system.
According to one aspect of the present application, a computer program product is provided that includes computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer device can execute the method in the embodiment corresponding to fig. 2, 3 or 5, and therefore, a detailed description will not be given here.
Those skilled in the art will appreciate that implementing all or part of the above-described embodiment methods may be accomplished by way of a computer program for instructing relevant hardware, where the program may be stored on a computer readable storage medium, and where the program, when executed, may comprise the embodiment flow of the above-described methods. The storage medium may be a magnetic disk, an optical disk, a Read-Only Memory (ROM), a random-access Memory (Random Access Memory, RAM), or the like.
The above disclosure is only a few examples of the present application, and it is not intended to limit the scope of the present application, but it is understood by those skilled in the art that all or a part of the above embodiments may be implemented and equivalents thereof may be modified according to the scope of the present application.
Claims (15)
1. A method of data processing, the method comprising:
performing superdivision processing on the sample image by using the initial superdivision model to obtain a predicted superdivision image;
performing model parameter adjustment on the initial superminute model according to the predicted superminute image and a reference superminute image corresponding to the sample image to obtain a first superminute model;
performing re-parameterization on a module to be processed in the first super-division model to generate a second super-division model, wherein the second super-division model is used for generating a target super-division image of an image to be processed;
the initial superdivision model comprises a characteristic rough machining network, a characteristic fine machining network and a characteristic fusion network, and the to-be-processed module is a part of or all of the characteristic rough machining network and the characteristic fine machining network after the model is subjected to parameter adjustment; the second superdivision model has a smaller number of parameters than the first superdivision model.
2. The method of claim 1, wherein the performing the super-division on the sample image using the initial super-division model to obtain the predicted super-division image comprises:
performing feature reconstruction processing on the sample image by using the feature rough machining network in the initial superdivision model to obtain a first feature map;
Performing feature reconstruction processing on the first feature map by using the feature fine processing network in the initial superdivision model to obtain a second feature map;
and performing deconvolution processing on the second feature map by using the feature fusion network in the initial superscore model to obtain a third feature map, performing up-sampling processing on the sample image to obtain a fourth feature map, and performing feature fusion processing on the third feature map and the fourth feature map to obtain a predicted superscore image.
3. The method of claim 2, wherein the feature roughing network comprises a feature roughing module, the feature roughing network comprising one or more feature finishing modules, each feature finishing module comprising a first finishing module and a second finishing module, an output of the first finishing module being connected to an input of the second finishing module, the first finishing module comprising a first finishing sub-module and a second finishing sub-module, the second finishing module comprising the first finishing sub-module and a third finishing sub-module, the feature fusion network comprising a deconvolution module, an upsampling module, and a summing module, the deconvolution module for performing the deconvolution process, the upsampling module for performing the upsampling process, and the summing module for performing the feature fusion process; the to-be-processed module is a characteristic rough machining module after model parameter adjustment, a first machining sub-module included in the first fine machining module, a second machining sub-module and a first machining sub-module included in the second fine machining module.
4. The method of claim 3, wherein the feature roughing module comprises a first convolution layer comprising a plurality of convolution kernels of a first size, a second convolution layer comprising a plurality of convolution kernels of a second size, and a summing module;
the feature reconstruction processing is performed on the sample image by using the feature rough machining network in the initial superdivision model to obtain a first feature map, which comprises the following steps:
respectively carrying out characteristic reconstruction processing on a sample image by using the first convolution layer and the second convolution layer which are included by the characteristic rough machining module in the characteristic rough machining network to obtain first output data and second output data;
and adding and processing the first output data, the second output data and the sample image through the adding module to obtain a first characteristic diagram.
5. A method according to claim 3, wherein the feature finishing network comprises a feature finishing module; and performing feature reconstruction processing on the first feature map by using the feature fine processing network in the initial super-division model to obtain a second feature map, wherein the feature reconstruction processing comprises the following steps:
Performing feature reconstruction processing on the first feature map by using the first processing submodule in the first feature fine processing network to obtain first intermediate data;
performing feature reconstruction processing on the first intermediate data by using the second processing sub-module in the first feature fine processing network to obtain an intermediate feature map;
performing feature reconstruction processing on the intermediate feature map by using the first processing sub-module in the second feature fine processing network to obtain second intermediate data;
and performing feature reconstruction processing on the second intermediate data by using the third processing sub-module in the second feature fine processing network to obtain a second feature map.
6. The method of claim 5, wherein the first processing submodule includes a third convolution layer including a third size of depth convolution kernels, a fourth convolution layer including a plurality of fourth size of depth convolution kernels, a fifth convolution layer including a plurality of fifth size of depth convolution kernels, and a summation module;
the performing feature reconstruction processing on the first feature map by using the first processing sub-module in the first feature fine processing network to obtain first intermediate data includes:
Respectively carrying out characteristic reconstruction processing on the first characteristic map by using the third convolution layer, the fourth convolution layer and the fifth convolution layer in the first processing submodule to obtain third output data, fourth output data and fifth output data;
and adding the third output data, the fourth output data, the fifth output data and the first feature map through the adding module to obtain first intermediate data.
7. The method of claim 5, wherein the second processing submodule includes a sixth convolution layer and a summation module, the sixth convolution layer including a plurality of convolution kernels of a sixth size;
the feature reconstruction processing is performed on the first intermediate data by using the second processing sub-module in the first feature fine processing network to obtain an intermediate feature map, including:
performing characteristic reconstruction processing on the first intermediate data by using the sixth convolution layer in the second processing submodule to obtain sixth output data;
and adding and processing the sixth output data and the first intermediate data through the adding module to obtain an intermediate feature map.
8. The method of claim 5, wherein the third processing module comprises a first feature extraction module, a second feature extraction module, and a summation module, the first feature extraction module comprising a seventh convolution layer, a channel transform layer, and an activation layer, the seventh convolution layer comprising a seventh-sized depth convolution kernel, the second feature extraction module comprising an eighth convolution layer, the eighth convolution layer comprising an eighth-sized convolution kernel;
the performing feature reconstruction processing on the second intermediate data by using the third processing sub-module in the second feature fine processing network to obtain a second feature map, including:
respectively carrying out feature reconstruction processing on the second intermediate data by using the first feature extraction module and the second feature extraction module in the third processing sub-module to obtain seventh output data and eighth output data;
and adding and processing the seventh output data and the eighth output data through the adding module to obtain a second characteristic diagram.
9. The method according to any one of claims 1-8, wherein performing model referencing on the initial superscore model according to the predicted superscore image and a reference superscore image corresponding to the sample image to obtain a first superscore model includes:
Obtaining a reference superdivision image corresponding to the sample image, wherein the sample image is obtained by performing resolution reduction processing on the reference superdivision image;
determining target difference data between the predicted superdivision image and the reference superdivision image;
and adjusting the model parameters of the initial superdivision model according to the target difference data, and determining the initial superdivision model with the adjusted model parameters as a first superdivision model.
10. The method according to any one of claims 1-8, wherein the performing a re-parameterization on the module to be processed in the first hyper-score model to generate a second hyper-score model includes:
performing re-parameterization on the module to be processed in the first superdivision model to obtain a re-parameterized module;
generating a second superdivision model according to the module which does not carry out the re-parameterization processing in the first superdivision model and the re-parameterization module;
the re-parameterization module comprises a depth convolution layer, wherein the depth convolution layer is used for reducing parameter quantity required by feature processing, the depth convolution layer comprises a depth convolution kernel, the depth convolution kernel is obtained by carrying out re-parameterization on an original convolution kernel in the module to be processed, and the size of the depth convolution kernel is larger than that of the original convolution kernel.
11. A method of data processing, the method comprising:
acquiring an image to be processed;
performing superprocessing on the image to be processed by using a second superdivision model to obtain a target superdivision image;
the second superdivision model is generated by carrying out re-parameterization on a to-be-processed module in a first superdivision model, the first superdivision model is obtained by carrying out model training on an initial superdivision model by utilizing a sample image, the initial superdivision model comprises a characteristic rough machining network, a characteristic fine machining network and a characteristic fusion network, and the to-be-processed module is a part of or all of the modules in the characteristic rough machining network and the characteristic fine machining network after model adjustment when the initial superdivision model is subjected to model training; the second superdivision model has a smaller number of parameters than the first superdivision model.
12. A data processing apparatus, characterized in that the apparatus comprises means for implementing the data processing method according to any of claims 1-10 or means for implementing the data processing method according to claim 11.
13. A computer device, comprising: processor, storage means and communication interface, said processor, said communication interface and said storage means being interconnected, wherein said storage means stores executable program code, said processor being adapted to invoke said executable program code for implementing a data processing method according to any of claims 1-10 or for implementing a data processing method according to claim 11.
14. A computer readable storage medium, characterized in that the computer readable storage medium stores a computer program comprising program instructions for execution by a processor for implementing the data processing method according to any one of claims 1-10 or for implementing the data processing method according to claim 11.
15. A computer program product, characterized in that the computer program product comprises a computer program or computer instructions for implementing the data processing method according to any of claims 1-10 or for implementing the data processing method according to claim 11 when executed by a processor.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310811145.9A CN116977169A (en) | 2023-07-04 | 2023-07-04 | Data processing method, apparatus, device, readable storage medium, and program product |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310811145.9A CN116977169A (en) | 2023-07-04 | 2023-07-04 | Data processing method, apparatus, device, readable storage medium, and program product |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116977169A true CN116977169A (en) | 2023-10-31 |
Family
ID=88477614
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310811145.9A Pending CN116977169A (en) | 2023-07-04 | 2023-07-04 | Data processing method, apparatus, device, readable storage medium, and program product |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116977169A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117649577A (en) * | 2024-01-30 | 2024-03-05 | 深圳金三立视频科技股份有限公司 | Training method of target detection model, and anti-external-damage monitoring and early warning method and device |
-
2023
- 2023-07-04 CN CN202310811145.9A patent/CN116977169A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117649577A (en) * | 2024-01-30 | 2024-03-05 | 深圳金三立视频科技股份有限公司 | Training method of target detection model, and anti-external-damage monitoring and early warning method and device |
| CN117649577B (en) * | 2024-01-30 | 2024-05-24 | 深圳金三立视频科技股份有限公司 | Training method of target detection model, and anti-external-damage monitoring and early warning method and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| WO2021018163A1 (en) | Neural network search method and apparatus | |
| KR20210074360A (en) | Image processing method, device and apparatus, and storage medium | |
| WO2022022288A1 (en) | Image processing method and apparatus | |
| CN112950471A (en) | Video super-resolution processing method and device, super-resolution reconstruction model and medium | |
| US11570477B2 (en) | Data preprocessing and data augmentation in frequency domain | |
| CN112906721B (en) | Image processing method, device, equipment and computer readable storage medium | |
| CN113344794B (en) | Image processing method and device, computer equipment and storage medium | |
| CN116958534A (en) | Image processing method, training method of image processing model and related device | |
| CN113066018A (en) | Image enhancement method and related device | |
| CN116977169A (en) | Data processing method, apparatus, device, readable storage medium, and program product | |
| Löhdefink et al. | On low-bitrate image compression for distributed automotive perception: Higher peak snr does not mean better semantic segmentation | |
| KR20220008135A (en) | Method and apparatus for image super resolution | |
| CN116977531A (en) | Three-dimensional texture image generation method, three-dimensional texture image generation device, computer equipment and storage medium | |
| CN115170388A (en) | Character line draft generation method, device, equipment and medium | |
| CN115131196A (en) | Image processing method, system, storage medium and terminal equipment | |
| CN114761968B (en) | Method, system and storage medium for frequency domain static channel filtering | |
| CN115409697A (en) | Image processing method and related device | |
| CN117593187A (en) | Remote sensing image super-resolution reconstruction method based on meta-learning and transducer | |
| TWI826160B (en) | Image encoding and decoding method and apparatus | |
| Yu et al. | A review of single image super-resolution reconstruction based on deep learning | |
| WO2023010981A1 (en) | Encoding and decoding methods and apparatus | |
| CN111861877A (en) | Method and apparatus for video hyper-resolution | |
| CN116703719A (en) | Face super-resolution reconstruction device and method based on face 3D priori information | |
| CN114586056A (en) | Image processing method and device, equipment, video processing method and storage medium | |
| CN118096961B (en) | Image processing method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication |