WO2022143921A1

WO2022143921A1 - Image reconstruction method, and related apparatus and system

Info

Publication number: WO2022143921A1
Application number: PCT/CN2021/143179
Authority: WO
Inventors: 吕飞帆; 张运超
Original assignee: 华为技术有限公司
Priority date: 2020-12-31
Filing date: 2021-12-30
Publication date: 2022-07-07
Also published as: CN114697543B; CN114697543A

Abstract

Disclosed is an image reconstruction method. An electronic device can extract image features of specified target content from a photographed low-definition image, and can match, on the basis of a high-definition texture dictionary library on a cloud server that is of the same category as the specified target content, high-definition texture features similar to the image features of the specified target content. The electronic device can fuse the high-definition texture features with the low-definition image to obtain a high-definition image. In this way, super-resolution reconstruction can be performed on a specific target in a targeted manner, and the computational burden on an electronic device during a super-resolution reconstruction process is simplified.

Description

An image reconstruction method, related device and system

This application claims the priority of the Chinese patent application filed on December 31, 2020 with the application number 202011639566.0 and titled "An Image Reconstruction Method, Related Apparatus and System", the entire contents of which are incorporated by reference in in this application.

technical field

The present application relates to the field of computer vision, and in particular, to an image reconstruction method, related apparatus and system.

Background technique

Since the development of smartphones, taking pictures has become one of its most important features. Both the basic imaging device and the imaging algorithm have undergone great development and changes, which have promoted the transformation of mobile phone photography again and again, and improved the user's photography experience. However, behind the changes again and again, the cost of the device is getting higher and higher, and the imaging quality is getting closer and closer to the bottleneck. The shooting of long-distance scenes is an important part of mobile phone photography, and the quality of the telephoto seriously affects the user's evaluation of the level of mobile phone photography.

In order to obtain higher quality telephoto quality, at this stage, electronic devices such as mobile phones mostly use a combination of optical zoom and digital zoom to improve the quality of telephoto imaging. For the digital zoom stage, deep learning-based image reconstruction methods are usually used on electronic devices. For example, a large convolutional neural network (CNN) is trained with a large number of low-definition image samples and high-definition image samples of different shooting scenes. The CNN model can be used to super-reconstruct the input image of any category. . In order to ensure the effect of super-resolution reconstruction, the CNN model needs to take into account many scenarios, but the CNN model is generally large, and there may be a risk of insufficient performance when running the CNN model on electronic devices such as mobile phones, making it difficult to quickly super-resolution reconstruction after taking pictures Produce high-quality pictures.

SUMMARY OF THE INVENTION

The present application provides an image reconstruction method, related device and system, which realizes identifying and cropping out specific target content from a photographed photo, and then performing super-resolution reconstruction processing on the cropped image to obtain a high-definition image. The quality of the image captured to a specific target in the photo.

In a first aspect, the present application provides an image reconstruction method, comprising: an electronic device receiving a first input; the electronic device, in response to the first input, acquiring a low-definition image captured by the camera; the electronic device detecting the low-definition image The image includes the first specified target content, and the category of the first specified target content is the first category; the electronic device extracts the first image feature of the first specified target content in the low-definition image; the electronic device An image feature and the first category are sent to the cloud server; wherein, the first image feature is used by the cloud server to match the shooting target category of the first category from the high-definition texture dictionary library of multiple different shooting target categories stored in the cloud server. a first high-definition texture dictionary library; the first image feature is used by the cloud server to match a first high-definition texture feature whose similarity with the first image feature is greater than a preset value from the first high-definition texture dictionary library; the electronic The device receives the first high-definition texture feature sent by the cloud server; the electronic device fuses the first high-definition texture feature into the low-definition image to obtain a high-definition image, and the resolution of the first specified target content in the high-definition image is greater than The resolution of the first specified target content in the low-definition image.

With the image reconstruction method provided by the present application, the electronic device can extract the image features of the specified target content from the captured low-definition image, and match the image features of the specified target content based on the high-definition texture dictionary library of the same category as the specified target content on the cloud server. High-definition texture features similar to the image features of the specified target content are obtained. The electronic device can fuse the high-definition texture features into a low-definition image to obtain a high-definition image. In this way, the super-resolution reconstruction can be carried out for a specific target, which simplifies the calculation amount on the electronic device in the super-resolution reconstruction process.

In a possible implementation manner, the electronic device extracts the first image feature of the first specified target content in the low-definition image, which specifically includes: the electronic device recognizes and crops out the first specified target content in the low-definition image. A first low-definition cropped image is obtained in the first area where the target content is located, and the first low-definition cropped image includes the first designated target content; the electronic device extracts a first image feature of the first low-definition cropped image.

In a possible implementation manner, the electronic device fuses the first high-definition texture feature into the low-definition image to obtain a high-definition image, which specifically includes: the electronic device fuses the first high-definition texture feature into the first low-definition image. In the clear cropped image, a first high-definition cropped image is obtained; the electronic device replaces the first high-definition cropped image with the first low-definition cropped image, and pastes it back to the first area in the low-definition image to obtain the high-definition image.

In a possible implementation manner, after the electronic device fuses the first high-definition texture feature into the low-definition image to obtain a high-definition image, the method further includes: the electronic device saves the high-definition image.

In a possible implementation manner, before the electronic device receives the first input, the method further includes: the electronic device displays a shooting preview interface, and the shooting preview interface displays a preview image stream captured in real time by a shooting key and a camera, The first input is an input for the shooting key.

In a possible implementation manner, a first control is also displayed on the shooting preview interface; before the electronic device receives the first input, the method further includes: the electronic device receives a second input for the first control; In response to the second input, the electronic device initiates a super-resolution reconstruction mode.

In a possible implementation manner, before the electronic device receives the first input, the method further includes: the electronic device displays a picture browsing interface, where the low-definition image and the second control are displayed on the picture browsing interface, the first An input is an input for the second control.

In a possible implementation manner, the preview image stream includes a first preview image; before the electronic device receives the first input, the method further includes: the electronic device detects that the first preview image includes a second specified image target content, the category of the second specified target content is a second category, and the second category is the same as or different from the first category; the electronic device extracts the second image feature in the first preview image; the electronic device extracts the second image feature from the first preview image; The second image feature and the second category are sent to the cloud server; wherein, the second image feature is used by the cloud server to match the shooting target category as the second shooting target category from the stored high-definition texture dictionary library of multiple different shooting target categories The second high-definition texture dictionary library of the category; the second image feature is used by the cloud server to match the second high-definition texture feature with the second image feature similarity greater than a preset value from the second high-definition texture dictionary library; The electronic device receives the second high-definition texture feature sent by the cloud server; the electronic device stores the second high-definition texture feature in a cache queue.

In a possible implementation manner, the electronic device extracts the second image feature of the second specified target content in the first preview image, which specifically includes: the electronic device recognizes and crops out the second image feature in the low-definition image. The second area where the target content is located is specified, and a second low-definition cropped image is obtained; the electronic device extracts a second image feature of the second low-definition cropped image.

In a possible implementation manner, after the electronic device detects that the first preview image includes the second specified target content, the method further includes: the electronic device displays category information on the shooting preview interface, the category information The category used to represent the specified target content in the preview image stream is the second category.

In a possible implementation manner, after the electronic device stores the second high-definition texture feature in a cache queue, the method further includes: acquiring, by the electronic device, a second preview image collected by a camera; the electronic device, the electronic device It is detected that the second preview image includes a third specified target content, the category of the third specified target content is a third category, and the third category is the same as or different from the second category; the electronic device identifies and cuts out the low Clear the third area where the third specified target content is located in the image, and obtain a third low-definition cropped image; the electronic device extracts a third image feature of the third low-definition cropped image; the electronic device determines whether the cache queue is in the A high-definition texture feature whose similarity with the third image feature is greater than a preset value is stored; if a high-definition texture feature whose similarity with the third image feature is greater than a preset value is stored in the cache queue, the electronic device uses the The high-definition texture feature whose similarity with the third image feature is greater than the preset value in the cache queue is fused into the third low-definition cropped image to obtain a third high-definition cropped image; the electronic device replaces the third high-definition cropped image with the third high-definition cropped image. The third low-definition cropped image is pasted back to the third area in the second preview image to obtain a high-definition preview image; the electronic device displays the high-definition preview image on the shooting preview interface.

In a possible implementation manner, if no high-definition texture feature whose similarity with the third image feature is greater than a preset value is stored in the cache queue, the method further includes: the electronic device combines the third image feature with the third image feature. The third category is sent to the cloud server; wherein, the third image feature is used by the cloud server to match a third high-definition texture whose shooting target category is the third category from the stored high-definition texture dictionary library of multiple different shooting target categories dictionary library; the third image feature is used by the cloud server to match the third high-definition texture feature with the third image feature from the third high-definition texture dictionary library whose similarity is greater than a preset value; the electronic device receives the cloud server The third high-definition texture feature sent; the electronic device stores the third high-definition texture feature in the cache queue, and fuses the third high-definition texture feature into the third low-definition cropped image to obtain the third high-definition texture feature Crop the image.

In a possible implementation manner, before the electronic device sends the first image feature and the first category to the cloud server, the method further includes: the electronic device determines whether the cache queue is stored with the first image feature and the first category. A high-definition texture feature whose similarity of image features is greater than a preset value; the electronic device sends the first image feature and the first category to the cloud server, which specifically includes: if the cache queue is not stored with the first image feature The high-definition texture feature whose similarity is greater than the preset value, the electronic device sends the first image feature and the first category to the cloud server.

In a possible implementation manner, if a high-definition texture feature whose similarity with the first image feature is greater than a preset value is stored in the cache queue, the method further includes: the electronic device associates the cache queue with the first image feature. A high-definition texture feature whose similarity of image features is greater than a preset value is fused into the low-definition image to obtain the high-definition image.

In a second aspect, the present application provides an electronic device including one or more processors and one or more memories. The one or more memories are coupled to the one or more processors for storing computer program code, the computer program code comprising computer instructions that, when executed by the one or more processors, cause the electronic device to perform The image reconstruction method in any possible implementation manner of any one of the above aspects.

In a third aspect, an embodiment of the present application provides a computer storage medium, including computer instructions, when the computer instructions are executed on an electronic device, the electronic device is made to execute the image reconstruction method in any possible implementation manner of any one of the above aspects .

In a fourth aspect, the embodiments of the present application provide a computer program product that, when the computer program product runs on a computer, enables the computer to execute the image reconstruction method in any possible implementation manner of any one of the foregoing aspects.

In a fifth aspect, an embodiment of the present application provides an image reconstruction system, including an electronic device and a cloud server, where a plurality of high-definition texture dictionary libraries of different shooting target categories are stored on the cloud server; wherein, the electronic device is used for receiving a first input; the electronic device is further configured to acquire a low-definition image captured by the camera in response to the first input; the electronic device is further configured to detect that the low-definition image includes the first specified target content, and the first specified target content is included in the first input. A category of the specified target content is the first category; the electronic device is further configured to extract the first image feature of the first specified target content in the low-definition image; the electronic device is also configured to extract the first image feature of the first specified target content and the first category are sent to the cloud server; the cloud server is used to match the first high-definition texture dictionary library whose shooting target category is the first category from the stored high-definition texture dictionary libraries of the plurality of different shooting target categories; The cloud server is further configured to match, from the first high-definition texture dictionary library, a first high-definition texture feature whose similarity with the first image feature is greater than a preset value; the cloud server is further configured to match the first high-definition texture feature with the first image feature. The texture feature is sent to the electronic device; the electronic device is further configured to fuse the first high-definition texture feature into the low-definition image to obtain a high-definition image, wherein the resolution of the first specified target content in the high-definition image is greater than The resolution of the first specified target content in the low-definition image.

In a possible implementation manner, the cloud server is further configured to acquire a plurality of high-quality images and the shooting target category of each high-quality image; the cloud server is further configured to extract the shooting target category of the high-quality image. The high-definition texture features are stored in the high-definition texture dictionary library corresponding to the shooting target category of the high-quality image.

In a possible implementation manner, the electronic device is further configured to send the identification category set of the target identification model on the electronic device to the cloud server; the cloud server is further configured to determine the relationship between the identification category set and the cloud server. When the category sets of the high-definition texture dictionary libraries of multiple different shooting target categories are different, send model update information to the electronic device; the electronic device is also used to update the target recognition model on the electronic device based on the model update information; wherein , after the update, the recognition category set of the target recognition model on the electronic device is the same as the category set of the high-definition texture dictionary libraries of the multiple different shooting target categories.

Wherein, the electronic device is further configured to execute the image reconstruction method in any possible implementation manner of any one of the foregoing aspects.

In a sixth aspect, an embodiment of the present application provides a chip system, including: one or more processors and one or more memories; the one or more memories are used to store computer program codes, and the computer program codes include computer instructions, when When one or more processors execute the computer instructions, the image reconstruction system causes the image reconstruction method in any possible implementation manner of any one of the foregoing aspects.

The one or more processors may include one or more of an application processor, an image signal processor, a digital signal processor, a neural network processor, and a graphics processor.

Description of drawings

FIG. 1 is a schematic structural diagram of a communication system according to an embodiment of the present application;

2A is a schematic diagram of a hardware structure of an electronic device provided by an embodiment of the application;

FIG. 2B is a schematic diagram of a software structure of an electronic device provided by an embodiment of the present application;

FIG. 2C is a schematic diagram of a hardware structure of a cloud server according to an embodiment of the present application;

3A-3E are schematic diagrams of a group of interfaces provided by an embodiment of the present application;

4A-4B are schematic diagrams of another set of interfaces provided by an embodiment of the present application;

5A-5B are schematic diagrams of another set of interfaces provided by an embodiment of the present application;

6A-6E are another set of interface schematic diagrams provided by the embodiments of the present application;

7 is a schematic flowchart of an image reconstruction method provided by an embodiment of the present application;

8A is a schematic diagram of a low-definition image provided by an embodiment of the present application;

8B is a schematic diagram of a specified target content mask image provided by an embodiment of the present application;

8C is a schematic diagram of a cropping frame in a low-definition image provided by an embodiment of the present application;

8D is a schematic diagram of a first low-definition cropped image provided by an embodiment of the present application;

8E is a schematic diagram of a first high-definition cropped image provided by an embodiment of the application;

8F is a schematic diagram of a high-definition image provided by an embodiment of the present application;

9 is a schematic flowchart of an image reconstruction method provided by another embodiment of the present application;

10 is a schematic flowchart of an image reconstruction method provided by another embodiment of the present application;

11 is a schematic flowchart of an image reconstruction method provided by another embodiment of the present application;

FIG. 12 is a schematic structural diagram of an image reconstruction system provided by an embodiment of the present application.

Detailed ways

The technical solutions in the embodiments of the present application will be described clearly and in detail below with reference to the accompanying drawings. Wherein, in the description of the embodiments of the present application, unless otherwise specified, “/” means or, for example, A/B can mean A or B; “and/or” in the text is only a description of an associated object The association relationship indicates that there can be three kinds of relationships, for example, A and/or B can indicate that A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of this application , "plurality" means two or more than two.

Hereinafter, the terms "first" and "second" are only used for descriptive purposes, and should not be construed as implying or implying relative importance or implying the number of indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present application, unless otherwise specified, the "multiple" The meaning is two or more.

The embodiment of the present application provides an image reconstruction method. The electronic device can extract the image features of the specified target content from the captured low-definition image, and based on the high-definition texture dictionary library of the same category as the specified target content on the cloud server , and match high-definition texture features similar to the image features of the specified target content. The electronic device can fuse the high-definition texture features into a low-definition image to obtain a high-definition image. In this way, the super-resolution reconstruction can be carried out for a specific target, which simplifies the calculation amount on the electronic device in the super-resolution reconstruction process.

The following introduces a communication system 10 involved in the embodiments of the present application.

Please refer to FIG. 1. FIG. 1 shows a schematic structural diagram of a communication system 10 according to an embodiment of the application. The communication system 10 may include a terminal 100 and a cloud server 200 . The terminal 100 may be a mobile phone, a tablet computer, a desktop computer, a laptop computer, a handheld computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a cellular phone, a personal digital assistant ( personal digital assistant, PDA), augmented reality (AR) \ virtual reality (virtual reality, VR) devices, etc. The specific type of the terminal 100 is not particularly limited in this embodiment of the present application.

The terminal 100 can be connected to the cloud server 200 through a 2G network, a 3G network, a 4G network, a 5G network, a wireless local area network (wireless local area network, WLAN), and the like. The terminal 100 may send the image feature/image data and the category information of the specified target content in the image feature/image data to the cloud server 200 . The cloud server 200 may send high-definition texture features and the like to the terminal 100 .

The cloud server 200 can establish connections with multiple terminals 100, and can independently process processing tasks requested by the multiple terminals 100. Wherein, the cloud server 200 can distinguish the terminals by the account (for example, a Huawei account) logged in by the user on the terminal.

FIG. 2A shows a schematic structural diagram of the electronic device 100 .

The embodiment will be described in detail below by taking the electronic device 100 as an example. It should be understood that the electronic device 100 shown in FIG. 2A is merely an example, and the electronic device 100 may have more or fewer components than those shown in FIG. 2A, two or more components may be combined, or Different component configurations are possible. The various components shown in Figure 2A may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and/or application specific integrated circuits.

The electronic device 100 may include: a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2. Mobile communication module 150, wireless communication module 160, audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, sensor module 180, buttons 190, motor 191, indicator 192, camera 193, display screen 194, And a subscriber identification module (subscriber identification module, SIM) card interface 195 and so on. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, and ambient light. Sensor 180L, bone conduction sensor 180M, etc.

It can be understood that, the structures illustrated in the embodiments of the present invention do not constitute a specific limitation on the electronic device 100 . In other embodiments of the present application, the electronic device 100 may include more or less components than shown, or combine some components, or separate some components, or arrange different components. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (application processor, AP), a modem processor, a graphics processor (graphics processing unit, GPU), an image signal processor (image signal processor, ISP), controller, memory, video codec, digital signal processor (digital signal processor, DSP), baseband processor, and/or neural-network processing unit (NPU) Wait. Wherein, different processing units may be independent devices, or may be integrated in one or more processors.

The controller may be the nerve center and command center of the electronic device 100 . The controller can generate an operation control signal according to the instruction operation code and timing signal, and complete the control of fetching and executing instructions.

A memory may also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in processor 110 is cache memory. This memory may hold instructions or data that have just been used or recycled by the processor 110 . If the processor 110 needs to use the instruction or data again, it can be called directly from the memory. Repeated accesses are avoided and the latency of the processor 110 is reduced, thereby increasing the efficiency of the system.

In some embodiments, the processor 110 may include one or more interfaces. The interface may include an integrated circuit (inter-integrated circuit, I2C) interface, an integrated circuit built-in audio (inter-integrated circuit sound, I2S) interface, a pulse code modulation (pulse code modulation, PCM) interface, a universal asynchronous transceiver (universal asynchronous transmitter) receiver/transmitter, UART) interface, mobile industry processor interface (MIPI), general-purpose input/output (GPIO) interface, subscriber identity module (SIM) interface, and / or universal serial bus (universal serial bus, USB) interface, etc.

The I2C interface is a bidirectional synchronous serial bus that includes a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may contain multiple sets of I2C buses. The processor 110 can be respectively coupled to the touch sensor 180K, the charger, the flash, the camera 193 and the like through different I2C bus interfaces. For example, the processor 110 may couple the touch sensor 180K through the I2C interface, so that the processor 110 and the touch sensor 180K communicate with each other through the I2C bus interface, so as to realize the touch function of the electronic device 100 .

The I2S interface can be used for audio communication. In some embodiments, the processor 110 may contain multiple sets of I2S buses. The processor 110 may be coupled with the audio module 170 through an I2S bus to implement communication between the processor 110 and the audio module 170 . In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 through the I2S interface, so as to realize the function of answering calls through a Bluetooth headset.

The PCM interface can also be used for audio communications, sampling, quantizing and encoding analog signals. In some embodiments, the audio module 170 and the wireless communication module 160 may be coupled through a PCM bus interface. In some embodiments, the audio module 170 can also transmit audio signals to the wireless communication module 160 through the PCM interface, so as to realize the function of answering calls through the Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

The UART interface is a universal serial data bus used for asynchronous communication. The bus may be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, a UART interface is typically used to connect the processor 110 with the wireless communication module 160 . For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 through the UART interface to implement the Bluetooth function. In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 through the UART interface, so as to realize the function of playing music through the Bluetooth headset.

The MIPI interface can be used to connect the processor 110 with peripheral devices such as the display screen 194 and the camera 193 . MIPI interfaces include camera serial interface (CSI), display serial interface (DSI), etc. In some embodiments, the processor 110 communicates with the camera 193 through a CSI interface, so as to realize the photographing function of the electronic device 100 . The processor 110 communicates with the display screen 194 through the DSI interface to implement the display function of the electronic device 100 .

The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface may be used to connect the processor 110 with the camera 193, the display screen 194, the wireless communication module 160, the audio module 170, the sensor module 180, and the like. The GPIO interface can also be configured as I2C interface, I2S interface, UART interface, MIPI interface, etc.

The USB interface 130 is an interface that conforms to the USB standard specification, and may specifically be a Mini USB interface, a Micro USB interface, a USB Type C interface, and the like. The USB interface 130 can be used to connect a charger to charge the electronic device 100, and can also be used to transmit data between the electronic device 100 and peripheral devices. It can also be used to connect headphones to play audio through the headphones. The interface can also be used to connect other electronic devices, such as AR devices.

It can be understood that the interface connection relationship between the modules illustrated in the embodiment of the present invention is only a schematic illustration, and does not constitute a structural limitation of the electronic device 100 . In other embodiments of the present application, the electronic device 100 may also adopt different interface connection manners in the foregoing embodiments, or a combination of multiple interface connection manners.

The charging management module 140 is used to receive charging input from the charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive charging input from the wired charger through the USB interface 130 . In some wireless charging embodiments, the charging management module 140 may receive wireless charging input through a wireless charging coil of the electronic device 100 . While the charging management module 140 charges the battery 142 , it can also supply power to the electronic device through the power management module 141 .

The power management module 141 is used for connecting the battery 142 , the charging management module 140 and the processor 110 . The power management module 141 receives input from the battery 142 and/or the charging management module 140 and supplies power to the processor 110 , the internal memory 121 , the external memory, the display screen 194 , the camera 193 , and the wireless communication module 160 . The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle times, battery health status (leakage, impedance). In some other embodiments, the power management module 141 may also be provided in the processor 110 . In other embodiments, the power management module 141 and the charging management module 140 may also be provided in the same device.

The wireless communication function of the electronic device 100 may be implemented by the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modulation and demodulation processor, the baseband processor, and the like.

Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 may be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, the antenna 1 can be multiplexed as a diversity antenna of the wireless local area network. In other embodiments, the antenna may be used in conjunction with a tuning switch.

The mobile communication module 150 may provide wireless communication solutions including 2G/3G/4G/5G etc. applied on the electronic device 100 . The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA) and the like. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, filter and amplify the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and then turn it into an electromagnetic wave for radiation through the antenna 1 . In some embodiments, at least part of the functional modules of the mobile communication module 150 may be provided in the processor 110 . In some embodiments, at least part of the functional modules of the mobile communication module 150 may be provided in the same device as at least part of the modules of the processor 110 .

The modem processor may include a modulator and a demodulator. Wherein, the modulator is used to modulate the low frequency baseband signal to be sent into a medium and high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low frequency baseband signal. Then the demodulator transmits the demodulated low-frequency baseband signal to the baseband processor for processing. The low frequency baseband signal is processed by the baseband processor and passed to the application processor. The application processor outputs sound signals through audio devices (not limited to the speaker 170A, the receiver 170B, etc.), or displays images or videos through the display screen 194 . In some embodiments, the modem processor may be a stand-alone device. In other embodiments, the modem processor may be independent of the processor 110, and may be provided in the same device as the mobile communication module 150 or other functional modules.

The wireless communication module 160 can provide applications on the electronic device 100 including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellites Wireless communication solutions such as global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared technology (IR). The wireless communication module 160 may be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2 , frequency modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110 . The wireless communication module 160 can also receive the signal to be sent from the processor 110 , perform frequency modulation on it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2 .

In some embodiments, the antenna 1 of the electronic device 100 is coupled with the mobile communication module 150, and the antenna 2 is coupled with the wireless communication module 160, so that the electronic device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), broadband Code Division Multiple Access (WCDMA), Time Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC , FM, and/or IR technology, etc. The GNSS may include global positioning system (global positioning system, GPS), global navigation satellite system (global navigation satellite system, GLONASS), Beidou navigation satellite system (beidou navigation satellite system, BDS), quasi-zenith satellite system (quasi -zenith satellite system, QZSS) and/or satellite based augmentation systems (SBAS).

The electronic device 100 implements a display function through a GPU, a display screen 194, an application processor, and the like. The GPU is a microprocessor for image processing, and is connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or alter display information.

Display screen 194 is used to display images, videos, and the like. Display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (active-matrix organic light). emitting diode, AMOLED), flexible light-emitting diode (flex light-emitting diode, FLED), Miniled, MicroLed, Micro-oLed, quantum dot light-emitting diode (quantum dot light emitting diodes, QLED) and so on. In some embodiments, the electronic device 100 may include one or N display screens 194 , where N is a positive integer greater than one.

The electronic device 100 may implement a shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, an application processor, and the like.

The ISP is used to process the data fed back by the camera 193 . For example, when taking a photo, the shutter is opened, the light is transmitted to the camera photosensitive element through the lens, the light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing, and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on image noise, brightness, and skin tone. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, the ISP may be provided in the camera 193 .

Camera 193 is used to capture still images or video. The object is projected through the lens to generate an optical image onto the photosensitive element. The photosensitive element may be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then transmits the electrical signal to the ISP to convert it into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. DSP converts digital image signals into standard RGB, YUV and other formats of image signals. In some embodiments, the electronic device 100 may include 1 or N cameras 193 , where N is a positive integer greater than 1.

A digital signal processor is used to process digital signals, in addition to processing digital image signals, it can also process other digital signals. For example, when the electronic device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy and so on.

Video codecs are used to compress or decompress digital video. The electronic device 100 may support one or more video codecs. In this way, the electronic device 100 can play or record videos of various encoding formats, such as: Moving Picture Experts Group (moving picture experts group, MPEG) 1, MPEG2, MPEG3, MPEG4 and so on.

The NPU is a neural-network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transfer mode between neurons in the human brain, it can quickly process the input information and can continuously learn by itself. Applications such as intelligent cognition of the electronic device 100 can be implemented through the NPU, such as image recognition, face recognition, speech recognition, text understanding, and the like.

The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100 . The external memory card communicates with the processor 110 through the external memory interface 120 to realize the data storage function. For example to save files like music, video etc in external memory card.

Internal memory 121 may be used to store computer executable program code, which includes instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by executing the instructions stored in the internal memory 121 . The internal memory 121 may include a storage program area and a storage data area. The storage program area can store an operating system, an application program required for at least one function (such as a sound playback function, an image playback function, etc.), and the like. The storage data area may store data (such as audio data, phone book, etc.) created during the use of the electronic device 100 and the like. In addition, the internal memory 121 may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, universal flash storage (UFS), and the like.

The electronic device 100 may implement audio functions through an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, an application processor, and the like. Such as music playback, recording, etc.

The audio module 170 is used for converting digital audio information into analog audio signal output, and also for converting analog audio input into digital audio signal. Audio module 170 may also be used to encode and decode audio signals. In some embodiments, the audio module 170 may be provided in the processor 110 , or some functional modules of the audio module 170 may be provided in the processor 110 .

Speaker 170A, also referred to as a "speaker", is used to convert audio electrical signals into sound signals. The electronic device 100 can listen to music through the speaker 170A, or listen to a hands-free call.

The receiver 170B, also referred to as "earpiece", is used to convert audio electrical signals into sound signals. When the electronic device 100 answers a call or a voice message, the voice can be answered by placing the receiver 170B close to the human ear.

The microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can make a sound by approaching the microphone 170C through a human mouth, and input the sound signal into the microphone 170C. The electronic device 100 may be provided with at least one microphone 170C. In other embodiments, the electronic device 100 may be provided with two microphones 170C, which can implement a noise reduction function in addition to collecting sound signals. In other embodiments, the electronic device 100 may further be provided with three, four or more microphones 170C to collect sound signals, reduce noise, identify sound sources, and implement directional recording functions.

The earphone jack 170D is used to connect wired earphones. The earphone interface 170D may be the USB interface 130, or may be a 3.5mm open mobile terminal platform (OMTP) standard interface, a cellular telecommunications industry association of the USA (CTIA) standard interface.

The pressure sensor 180A is used to sense pressure signals, and can convert the pressure signals into electrical signals. In some embodiments, the pressure sensor 180A may be provided on the display screen 194 . There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, and the like. The capacitive pressure sensor may be comprised of at least two parallel plates of conductive material. When a force is applied to the pressure sensor 180A, the capacitance between the electrodes changes. The electronic device 100 determines the intensity of the pressure according to the change in capacitance. When a touch operation acts on the display screen 194, the electronic device 100 detects the intensity of the touch operation according to the pressure sensor 180A. The electronic device 100 may also calculate the touched position according to the detection signal of the pressure sensor 180A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities may correspond to different operation instructions. For example, when a touch operation whose intensity is less than the first pressure threshold acts on the short message application icon, the instruction for viewing the short message is executed. When a touch operation with a touch operation intensity greater than or equal to the first pressure threshold acts on the short message application icon, the instruction to create a new short message is executed.

The gyro sensor 180B may be used to determine the motion attitude of the electronic device 100 . In some embodiments, the angular velocity of electronic device 100 about three axes (ie, x, y, and z axes) may be determined by gyro sensor 180B. The gyro sensor 180B can be used for image stabilization. Exemplarily, when the shutter is pressed, the gyro sensor 180B detects the angle at which the electronic device 100 shakes, calculates the distance that the lens module needs to compensate for according to the angle, and allows the lens to counteract the shake of the electronic device 100 through reverse motion to achieve anti-shake. The gyro sensor 180B can also be used for navigation and somatosensory game scenarios.

The air pressure sensor 180C is used to measure air pressure. In some embodiments, the electronic device 100 calculates the altitude through the air pressure value measured by the air pressure sensor 180C to assist in positioning and navigation.

The magnetic sensor 180D includes a Hall sensor. The electronic device 100 can detect the opening and closing of the flip holster using the magnetic sensor 180D. In some embodiments, when the electronic device 100 is a flip machine, the electronic device 100 can detect the opening and closing of the flip according to the magnetic sensor 180D. Further, according to the detected opening and closing state of the leather case or the opening and closing state of the flip cover, characteristics such as automatic unlocking of the flip cover are set.

The acceleration sensor 180E can detect the magnitude of the acceleration of the electronic device 100 in various directions (generally three axes). The magnitude and direction of gravity can be detected when the electronic device 100 is stationary. It can also be used to identify the posture of electronic devices, and can be used in applications such as horizontal and vertical screen switching, pedometers, etc.

Distance sensor 180F for measuring distance. The electronic device 100 can measure the distance through infrared or laser. In some embodiments, when shooting a scene, the electronic device 100 can use the distance sensor 180F to measure the distance to achieve fast focusing.

Proximity light sensor 180G may include, for example, light emitting diodes (LEDs) and light detectors, such as photodiodes. The light emitting diodes may be infrared light emitting diodes. The electronic device 100 emits infrared light to the outside through the light emitting diode. Electronic device 100 uses photodiodes to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the electronic device 100 . When insufficient reflected light is detected, the electronic device 100 may determine that there is no object near the electronic device 100 . The electronic device 100 can use the proximity light sensor 180G to detect that the user holds the electronic device 100 close to the ear to talk, so as to automatically turn off the screen to save power. Proximity light sensor 180G can also be used in holster mode, pocket mode automatically unlocks and locks the screen.

The ambient light sensor 180L is used to sense ambient light brightness. The electronic device 100 can adaptively adjust the brightness of the display screen 194 according to the perceived ambient light brightness. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures. The ambient light sensor 180L can also cooperate with the proximity light sensor 180G to detect whether the electronic device 100 is in a pocket, so as to prevent accidental touch.

The fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can use the collected fingerprint characteristics to realize fingerprint unlocking, accessing application locks, taking pictures with fingerprints, answering incoming calls with fingerprints, and the like.

The temperature sensor 180J is used to detect the temperature. In some embodiments, the electronic device 100 uses the temperature detected by the temperature sensor 180J to execute a temperature processing strategy. For example, when the temperature reported by the temperature sensor 180J exceeds a threshold value, the electronic device 100 reduces the performance of the processor located near the temperature sensor 180J in order to reduce power consumption and implement thermal protection. In other embodiments, when the temperature is lower than another threshold, the electronic device 100 heats the battery 142 to avoid abnormal shutdown of the electronic device 100 caused by the low temperature. In some other embodiments, when the temperature is lower than another threshold, the electronic device 100 boosts the output voltage of the battery 142 to avoid abnormal shutdown caused by low temperature.

Touch sensor 180K, also called "touch panel". The touch sensor 180K may be disposed on the display screen 194 , and the touch sensor 180K and the display screen 194 form a touch screen, also called a “touch screen”. The touch sensor 180K is used to detect a touch operation on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to touch operations may be provided through display screen 194 . In other embodiments, the touch sensor 180K may also be disposed on the surface of the electronic device 100 , which is different from the location where the display screen 194 is located.

The bone conduction sensor 180M can acquire vibration signals. In some embodiments, the bone conduction sensor 180M can acquire the vibration signal of the vibrating bone mass of the human voice. The bone conduction sensor 180M can also contact the pulse of the human body and receive the blood pressure beating signal. In some embodiments, the bone conduction sensor 180M can also be disposed in the earphone, combined with the bone conduction earphone. The audio module 170 can analyze the voice signal based on the vibration signal of the vocal vibration bone block obtained by the bone conduction sensor 180M, so as to realize the voice function. The application processor can analyze the heart rate information based on the blood pressure beat signal obtained by the bone conduction sensor 180M, and realize the function of heart rate detection.

The keys 190 include a power-on key, a volume key, and the like. Keys 190 may be mechanical keys. It can also be a touch key. The electronic device 100 may receive key inputs and generate key signal inputs related to user settings and function control of the electronic device 100 .

Motor 191 can generate vibrating cues. The motor 191 can be used for vibrating alerts for incoming calls, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, playing audio, etc.) can correspond to different vibration feedback effects. The motor 191 can also correspond to different vibration feedback effects for touch operations on different areas of the display screen 194 . Different application scenarios (for example: time reminder, receiving information, alarm clock, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

The indicator 192 can be an indicator light, which can be used to indicate the charging state, the change of the power, and can also be used to indicate a message, a missed call, a notification, and the like.

The SIM card interface 195 is used to connect a SIM card. The SIM card can be contacted and separated from the electronic device 100 by inserting into the SIM card interface 195 or pulling out from the SIM card interface 195 . The electronic device 100 may support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM card, Micro SIM card, SIM card and so on. Multiple cards can be inserted into the same SIM card interface 195 at the same time. The types of the plurality of cards may be the same or different. The SIM card interface 195 can also be compatible with different types of SIM cards. The SIM card interface 195 is also compatible with external memory cards. The electronic device 100 interacts with the network through the SIM card to implement functions such as call and data communication. In some embodiments, the electronic device 100 employs an eSIM, ie: an embedded SIM card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100 .

The software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a microkernel architecture, a microservice architecture, or a cloud architecture. The embodiment of the present invention takes an Android system with a layered architecture as an example to illustrate the software structure of the electronic device 100 as an example.

FIG. 2B is a block diagram of the software structure of the electronic device 100 according to the embodiment of the present invention.

The layered architecture divides the software into several layers, and each layer has a clear role and division of labor. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, which are, from top to bottom, an application layer, an application framework layer, an Android runtime (Android runtime) and a system library, and a kernel layer.

The application layer can include a series of application packages.

As shown in Fig. 2B, the application package may include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message, etc.

The application framework layer provides an application programming interface (application programming interface, API) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions.

As shown in Figure 2B, the application framework layer may include a window manager, a content provider, a view system, a telephony manager, a resource manager, a notification manager, and the like.

A window manager is used to manage window programs. The window manager can get the size of the display screen, determine whether there is a status bar, lock the screen, take screenshots, etc.

Content providers are used to store and retrieve data and make these data accessible to applications. The data may include video, images, audio, calls made and received, browsing history and bookmarks, phone book, etc.

The view system includes visual controls, such as controls for displaying text, controls for displaying pictures, and so on. View systems can be used to build applications. A display interface can consist of one or more views. For example, the display interface including the short message notification icon may include a view for displaying text and a view for displaying pictures.

The phone manager is used to provide the communication function of the electronic device 100 . For example, the management of call status (including connecting, hanging up, etc.).

The resource manager provides various resources for the application, such as localization strings, icons, pictures, layout files, video files and so on.

The notification manager enables applications to display notification information in the status bar, which can be used to convey notification-type messages, and can disappear automatically after a brief pause without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also display notifications in the status bar at the top of the system in the form of graphs or scroll bar text, such as notifications of applications running in the background, and notifications on the screen in the form of dialog windows. For example, text information is prompted in the status bar, a prompt sound is issued, the electronic device vibrates, and the indicator light flashes.

Android Runtime includes core libraries and a virtual machine. Android runtime is responsible for scheduling and management of the Android system.

The core library consists of two parts: one is the function functions that the java language needs to call, and the other is the core library of Android.

The application layer and the application framework layer run in virtual machines. The virtual machine executes the java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, safety and exception management, and garbage collection.

A system library can include multiple functional modules. For example: surface manager (surface manager), media library (Media Libraries), 3D graphics processing library (eg: OpenGL ES), 2D graphics engine (eg: SGL), etc.

The Surface Manager is used to manage the display subsystem and provides a fusion of 2D and 3D layers for multiple applications.

The media library supports playback and recording of a variety of commonly used audio and video formats, as well as still image files. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

2D graphics engine is a drawing engine for 2D drawing.

The kernel layer is the layer between hardware and software. The kernel layer contains at least display drivers, camera drivers, audio drivers, and sensor drivers.

In the following, the workflow of the software and hardware of the electronic device 100 is exemplarily described in conjunction with the capturing and photographing scene.

When the touch sensor 180K receives a touch operation, a corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes touch operations into raw input events (including touch coordinates, timestamps of touch operations, etc.). Raw input events are stored at the kernel layer. The application framework layer obtains the original input event from the kernel layer, and identifies the control corresponding to the input event. Taking the touch operation as a touch click operation, and the control corresponding to the click operation is the control of the camera application icon, for example, the camera application calls the interface of the application framework layer to start the camera application, and then starts the camera driver by calling the kernel layer. The camera 193 captures still images or video.

The following introduces a schematic structural diagram of the cloud server 200 involved in the embodiments of the present application.

FIG. 2C is a schematic structural diagram of a cloud server 200 provided by an embodiment of the present application.

As shown in FIG. 2C, the cloud server 200 includes one or more processors 201, a communication interface 202, and a memory 203. The processor 201, the communication interface 202, and the memory 203 may be connected through a bus or other means. The bus 204 is connected as an example. in:

The processor 201 may be constituted by one or more general-purpose processors, such as a CPU. The processor 201 may be used to execute relevant program codes of the device control method.

Communication interface 202 may be a wired interface (eg, an Ethernet interface) or a wireless interface (eg, a cellular network interface or using a wireless local area network interface) for communicating with other nodes. In this embodiment of the present application, the communication interface 202 may specifically be used to communicate with the electronic device 100 .

The memory 203 may include volatile memory, such as RAM; the memory may also include non-volatile memory, such as ROM, flash memory, HDD or solid-state drive SSD; memory 203A may also include a combination of the types of memories described above. The memory 203 may be used to store a set of program codes, so that the processor 201 can call the program codes stored in the memory 203 to implement the implementation method on the server of the embodiments of the present application.

It should be noted that the server shown in FIG. 2C is only an implementation manner of the embodiment of the present application. In practical applications, the server may further include more or less components, which is not limited here.

The following describes an image reconstruction method provided by an embodiment of the present application in combination with an application scenario.

In some application scenarios, when the user uses the electronic device 100 to take a picture, the user can open the camera application to take a picture. When the electronic device 100 receives the user's photographing operation, the electronic device 100 obtains the low-definition image captured by the camera, identifies and extracts the image features of the specified target content in the low-definition image (for example, the Ferris wheel building), and based on the The high-definition texture dictionary library of the same category of the specified target content matches high-definition texture features similar to the image features of the specified target content. The electronic device 100 can fuse the high-definition texture feature into a low-definition image to obtain a high-definition image. In this way, the imaging quality of the specified target content can be improved.

Exemplarily, as shown in FIG. 3A , the electronic device 100 may display an interface 310 with a home screen, the interface 310 displays a page on which application icons are placed, and the page includes a plurality of application icons (eg, weather application icons, Stock application icon, calculator application icon, settings application icon, mail application icon, Alipay application icon, Facebook application icon, browser application icon, gallery application icon 312, music application icon, video application icon, WeChat application icon, etc.) . A page indicator is also displayed below the multiple application icons to indicate the positional relationship between the currently displayed page and other pages. Below the page indicator are a plurality of tray icons (eg, a dialing application icon, an information application icon, a contact application icon, and a camera application icon 313 ), and the tray icons remain displayed when switching pages. In some embodiments, the above-mentioned page may also include multiple application icons and page indicators. The page indicator may not be a part of the page, but exists alone. The above-mentioned picture icon is also optional, which is not limited in this embodiment of the present application. A status bar 311 is displayed in the upper part of the interface 310, and the status bar 311 may include: one or more signal strength indicators of mobile communication signals (also called cellular signals), battery status indicators, and time indicators , Wi-Fi signal indicator, and more.

The electronic device 100 may receive an input operation (eg, single click) of the user acting on the camera application icon 513, and in response to the input operation, the electronic device 100 may display a shooting preview interface 520 as shown in FIG. 5B .

As shown in FIG. 3B , the shooting preview interface 320 can display a control 321 for echoing the captured image, a shooting control 322, a camera switching control 323, a preview screen captured by the camera 324, a setting control 325, a zoom ratio control 326, one or Multiple capture mode controls (eg, "Night Mode" control 327A, "Portrait Mode" control 327B, "Normal Photo Mode" control 327C, "Video Mode" control 327D, "Pro Mode" control 327E, More Mode Controls 327F, "Large Aperture Mode" controls 327H, etc.). The captured image echo control 321 may be used to display the captured image. The shooting control 322 is used to trigger saving of the image captured by the camera. The camera switching control 323 can be used to switch cameras for taking pictures. The setting control 325 can be used to set the camera function. The zoom ratio control 326 can be used to set the zoom ratio of the camera. The shooting mode control can be used to trigger the opening of the image processing flow corresponding to the shooting mode. For example, a "night scene mode" control 327A can be used to trigger an increase in brightness and color richness, etc. in the captured image. A "portrait mode" control 372B may be used to trigger blurring of the background of people in the captured image, among other things. As shown in FIG. 3B , the shooting mode currently selected by the user is the "normal shooting mode".

The electronic device 100 may receive a user's input operation (eg, click) on the photographing control 322 in the photographing preview interface 320 , and in response to the input, the electronic device 100 may acquire a low-definition image (eg, a preview image 324 ) captured by the camera. The electronic device 100 can identify and extract the image features of the specified target content in the low-definition image, and based on the high-definition texture dictionary library of the same category as the specified target content on the cloud server, match the high-definition image features similar to the specified target content. texture features. The electronic device 100 can fuse the high-definition texture feature into a low-definition image to obtain a high-definition image.

Optionally, after the electronic device 100 recognizes the category of the specified target content in the low-definition image, it can display category information 329 in the above-mentioned shooting interface 320, and the category information 329 can be used to indicate the category of the specified target content in the low-definition image, for example , the category information 329 may be the text of "Ferris wheel".

Specifically, for the super-resolution reconstruction process of the low-definition image, reference may be made to the subsequent embodiments of the present application, and details are not described herein again.

In a possible implementation manner, when the electronic device 100 uses the high-definition texture dictionary library on the cloud server 200 to perform super-resolution reconstruction processing on the captured low-definition image, the electronic device 100 may display a processing progress prompt or countdown, It is used to prompt the user that the current electronic device 100 performs super-resolution reconstruction processing on the captured low-definition image by means of the high-definition texture dictionary library on the cloud server 200 .

Exemplarily, as shown in FIG. 3C , the electronic device 100 may display a progress prompt window 341 on the shooting preview interface 320 for prompting the progress of super-resolution reconstruction of the shot low-definition image. For example, the progress prompt window 341 displays a text prompt "The photo is being optimized through the cloud, the progress is 10%". A cancel control 342 may also be displayed in the progress prompt window 341, which may be used to trigger canceling the super-score reconstruction of the captured image. For example, when the electronic device 100 receives a user's input operation (eg, a single click) for the cancel control 342, the electronic device 100 may stop performing super-resolution reconstruction processing on the captured image. In this way, when the progress is slow or when the user temporarily does not want to over-score the captured image, the over-score reconstruction of the captured image can be canceled immediately.

In a possible implementation manner, after the electronic device 100 completes the enhancement of the captured image by using the processing capability of the cloud server 200, the electronic device 100 may display a completion prompt, and save the high-definition image after super-resolution reconstruction locally. The completion prompt may be used to prompt the user that the super-resolution reconstruction of the captured image has been completed.

Exemplarily, as shown in FIG. 3D , the electronic device 100 displays a completion prompt 343 after completing the super-score reconstruction of the captured image by means of the high-definition texture dictionary library on the cloud server 200 . For example, the completion prompt 343 may be a text prompt "The photo over-scoring process has been completed, and can be viewed in the gallery".

The electronic device 100 may receive a user's input operation (eg, click) on the captured image echo control 321, and in response to the input operation, the electronic device 100 may display a photo browsing interface 550 as shown in FIG. 3E .

As shown in FIG. 3E , the photo browsing interface 350 may include a high-definition image 351 after super-resolution reconstruction, image-related information 353 , a menu 354 , and a gallery control 355 . Wherein, the image-related information 353 may include the shooting time of the high-definition image 351, shooting weather, geographic location information, and the like. For example, the shooting time can be "December 3, 2019 8:00AM", the shooting weather can be "cloudy", the shooting location can be "Shanghai Disney", and so on. The menu 354 may include a share button, a favorite button, an edit button, a delete button, and more buttons. A share button may be used to trigger sharing of the high-definition image 351 . The favorite button can be used to trigger to save the high-definition image 351 to the picture favorite folder. The edit button can be used to trigger editing functions such as rotation, trimming, adding filters, and blurring of the high-definition image 351. The delete button can be used to trigger deletion of the high-definition image 351 . The More button can be used to trigger opening more functions related to this HD image 351 . The gallery control 355 can be used to trigger the electronic device 100 to open the gallery application.

In some embodiments, when the user uses the electronic device 100 to take a picture, the user can open the camera application to take a picture. When the electronic device 100 displays the shooting preview interface, the shooting preview interface may include a super-resolution reconstruction control and a shooting control, and the super-resolution reconstruction control can be used to trigger the electronic device 100 to perform super-resolution reconstruction on the captured low-definition image. After receiving the user's input for the image reconstruction control, the electronic device 100 can enable the super-resolution reconstruction function. After the super-resolution reconstruction function is enabled, when the electronic device 100 receives the user's photographing operation, the electronic device 100 obtains a low-definition image captured by the camera. The electronic device 100 can identify and extract the image features of the specified target content in the low-definition image, and based on the high-definition texture dictionary library of the same category as the specified target content on the cloud server, match the high-definition image features similar to the specified target content. texture features. The electronic device 100 can fuse the high-definition texture feature into a low-definition image to obtain a high-definition image. In this way, the imaging quality of the specified target content can be improved.

Exemplarily, the electronic device 100 may receive an input operation (such as a click) by the user acting on the camera application icon 313 in the interface 310 shown in FIG. 3A through the earphone, and in response to the input operation, the electronic device 100 may display as shown in FIG. 4A . The shooting preview interface 420 shown.

As shown in FIG. 4A , the shooting preview interface 420 may include a captured image echo control 421 , a shooting control 422 , a camera conversion control 423 , a camera-captured preview screen 424 , a super-resolution reconstruction control 428 , a setting control, and a zoom ratio. controls, one or more capture mode controls (eg, "Night Mode" control 427A, "Portrait Mode" control 427B, "Normal Photo Mode" control 427C, "Video Mode" control 427D, "Pro Mode" control 427E, more mode control 427F, "large aperture mode" control 427H, etc.). The super-resolution reconstruction control 428 may be used to trigger the electronic device 100 to perform super-resolution reconstruction on the captured low-definition image. For the text description of the captured image echo control 421 , the shooting control 422 , the camera switching control 423 , the setting control, the zoom magnification control, and one or more shooting mode controls, reference may be made to the aforementioned embodiment shown in FIG. 3B , which is not repeated here. Repeat.

The electronic device 100 may receive an input operation (eg, a single click) of the user with respect to the super-score reconstruction control 428, and in response to the input operation, the electronic device 100 may turn on the image reconstruction function. After the image reconstruction function is enabled, the electronic device 100 may receive an input operation (eg, a single click) from the user on the shooting control 422 in the shooting preview interface 420, and in response to the input, the electronic device 100 may acquire a low-definition image captured by the camera (eg preview screen 424). The electronic device 100 can identify and extract the image features of the specified target content in the low-definition image, and based on the high-definition texture dictionary library of the same category as the specified target content on the cloud server, match the high-definition image features similar to the specified target content. texture features. The electronic device 100 can fuse the high-definition texture feature into a low-definition image to obtain a high-definition image. The electronic device 100 can save the high-definition image locally.

In this embodiment of the present application, the above-mentioned super-score reconstruction control 428 may be referred to as a first control. The input to the super-score reconstruction control 428 may be referred to as the second input.

Optionally, after the electronic device 100 recognizes the category of the specified target content in the low-definition image, it can display category information 429 in the above-mentioned shooting interface 420, and the category information 429 can be used to indicate the category of the specified target content in the low-definition image, for example. , the category information 429 can be the text of "Ferris wheel".

In some embodiments, when the user uses the electronic device 100 to take pictures, the user can take pictures in different photographing modes in the electronic device 100 . When the user feels that the resolution of the preview image displayed in the shooting preview interface is low, the user can select the "super-resolution reconstruction mode" in the camera application to shoot. In the "super-resolution reconstruction mode", when the electronic device 100 receives a user's photographing operation, the electronic device 100 obtains a low-definition image captured by the camera. The electronic device 100 can identify and extract the image features of the specified target content in the low-definition image, and based on the high-definition texture dictionary library of the same category as the specified target content on the cloud server, match the high-definition image features similar to the specified target content. texture features. The electronic device 100 can fuse the high-definition texture feature into a low-definition image to obtain a high-definition image. In this way, the imaging quality of the specified target content can be improved.

Exemplarily, the electronic device 100 may receive an input operation (such as a click) that the user acts on the camera application icon 313 in the interface 310 shown in FIG. 3A , and in response to the input operation, the electronic device 100 may display the image shown in FIG. Shooting preview interface 520 .

As shown in FIG. 5A , the shooting preview interface 520 may include a captured image echo control 521 , a shooting control 522 , a camera conversion control 523 , a camera-captured preview screen 524 , a setting control, a zoom ratio control, one or more Capture mode controls (eg, "Night Mode" control 527A, "Portrait Mode" control 527B, "Normal Photo Mode" control 527C, "Super Reconstruction Mode" control 527G, "Video Mode" control 527D, "Pro Mode" control 527E , More Mode Controls 527F, etc.). The “super-resolution reconstruction mode” control 527G can be used to trigger the electronic device 100 to perform super-resolution reconstruction on the captured low-definition image. For the text description of the captured image echo control 521 , the shooting control 522 , the camera switching control 523 , the setting control, the zoom magnification control, and one or more shooting mode controls, reference may be made to the aforementioned embodiment shown in FIG. 3B , which is not repeated here. Repeat. As shown in FIG. 5A , the shooting mode currently selected by the user is the "normal shooting mode".

The electronic device 100 may receive an input operation (eg, a single click) that the user acts on the “super-resolution reconstruction mode” control 527G, and in response to the input operation, as shown in FIG. 5B , the electronic device 100 may adjust the currently selected shooting mode to “ Hyperdivision Reconstruction Mode".

In this embodiment of the present application, the above-mentioned "super-resolution reconstruction mode" control 527G may be referred to as a first control. The input to the "super-resolution reconstruction mode" control 527G may be referred to as the second input.

After the shooting mode of the electronic device 100 is adjusted to the "super-resolution reconstruction mode", the electronic device 100 may receive an input operation (eg, click) by the user on the shooting control 522 in the shooting preview interface 520, and in response to the input, the electronic device 100 may acquire a low-definition image (eg, a preview image 524 ) captured by the camera. The electronic device 100 can identify and extract the image features of the specified target content in the low-definition image, and based on the high-definition texture dictionary library of the same category as the specified target content on the cloud server, match the high-definition image features similar to the specified target content. texture features. The electronic device 100 can fuse the high-definition texture feature into a low-definition image to obtain a high-definition image. The electronic device 100 can save the high-definition image locally.

Optionally, after the electronic device 100 identifies the category of the specified target content in the low-definition image, it can display category information 529 in the above-mentioned shooting interface 520, and the category information 529 can be used to indicate the category of the specified target content in the low-definition image, for example , the category information 529 can be the text of "Ferris wheel".

In some application scenarios, the electronic device 100 may store many pictures locally, and these pictures may be captured by the electronic device 100, or sent from other devices, or downloaded from the network. Among them, some of the pictures have low resolution due to the limitation of the capabilities of the shooting equipment. The electronic device 100 may acquire the low-definition image in the gallery in response to the user's input. The electronic device 100 can identify and extract the image features of the specified target content in the low-definition image, and based on the high-definition texture dictionary library of the same category as the specified target content on the cloud server, match the high-definition image features similar to the specified target content. texture features. The electronic device 100 can fuse the high-definition texture feature into a low-definition image, obtain a high-definition image, and store it in a gallery. In this way, the imaging quality of the specified target content can be improved.

Exemplarily, as shown in FIG. 6A , the electronic device 100 may receive an input operation (eg, click) that the user acts on the gallery application icon 312 in the interface 310 , and in response to the input operation, the electronic device 100 may display as shown in FIG. 6B . 610 of the gallery application interface.

As shown in FIG. 6B, the gallery application interface 610 can display thumbnail images including one or more pictures. The electronic device 100 may mark the one or more pictures that include the specified target content and have not undergone super-score reconstruction processing. For example, the electronic device 100 may display a mark 612 on the thumbnail 611, where the mark 612 may be used to prompt the user that the picture corresponding to the thumbnail 611 is waiting for the super-score reconstruction process. Optionally, the electronic device 100 may mark the original picture that has undergone the super-resolution reconstruction process and the high-definition picture after the super-resolution reconstruction process in the one or more pictures.

The electronic device 100 may receive an input operation (eg, single click) of the user acting on the thumbnail 611, and in response to the input operation, the electronic device 100 may display a picture browsing interface 620 as shown in FIG. 6C.

As shown in FIG. 6C , the picture browsing interface 620 may display a picture 621 including the corresponding thumbnail 611 , a super-score reconstruction control 624 , picture-related information 622 , a menu 623 , and the like. The picture-related information 622 may include one or more of the shooting time of the picture 621, shooting weather, geographic location information, and the like. For example, the shooting time can be "8:00AM on December 1, 2019", the shooting weather can be "cloudy", the shooting location can be "Shanghai·The Bund", and so on. The menu 623 may include a share button, a favorite button, an edit button, a delete button, and more buttons. The super-resolution reconstruction control 624 can be used to trigger the electronic device 100 to perform super-resolution reconstruction processing on the picture 621 . The specific process of performing the super-score reconstruction processing on the picture 621 may refer to the subsequent embodiments, which will not be repeated here.

The electronic device 100 may receive an input operation (eg, click) of the user for the super-score reconstruction control 624, and in response to the input operation, the electronic device 100 may identify and extract the image features of the specified target content in the picture 621, and based on the cloud The high-definition texture dictionary library of the same category as the specified target content on the server matches high-definition texture features similar to the image features of the specified target content. The electronic device 100 may fuse the high-definition texture feature into the picture 621 to obtain a high-definition picture 631 , and the resolution of the high-definition picture 631 is higher than that of the picture 621 .

In this embodiment of the present application, the above-mentioned super-score reconstruction control 624 may be referred to as a second control.

As shown in FIG. 6D , after obtaining the high-definition picture 631 , the electronic device 100 may display a picture browsing interface 630 . The picture browsing interface 630 may include a high-definition picture 631, picture-related information 632, a menu 633, an effect comparison control 635, a mark 634, a save control 636, and the like. The picture-related information 632 may include one or more of the shooting time of the high-definition picture 631, shooting weather, geographic location information, and the like. For example, the shooting time can be "8:00AM on December 1, 2019", the shooting weather can be "cloudy", the shooting location can be "Shanghai·The Bund", and so on. The menu 633 may include one or more of a share button, a favorite button, an edit button, a delete button, a more button, and the like. The effect comparison control 634 can be used to trigger the electronic device 100 to display the above-mentioned picture 621 at the position of the high-definition picture 631 . The mark 634 may be used to prompt the user that the high-definition picture 631 is a high-definition image processed by super-resolution reconstruction. The save control 636 can be used to trigger the electronic device 100 to save the high-definition picture 634 to the gallery.

The electronic device 100 may receive the user's input operation (eg, click) on the effect comparison control 635, and in response to the input operation, as shown in FIG. 6E, the electronic device 100 may display the above-mentioned picture browsing interface 620, which includes There is picture 621 above. The electronic device 100 may receive the user's input operation (eg, click) on the effect comparison control 635 again, and in response to the input operation, the electronic device 100 may display the picture browsing interface 630 shown in FIG. 6D .

The following describes an image reconstruction method provided by an embodiment of the present application.

FIG. 7 exemplarily shows a schematic flowchart of an image reconstruction method provided by an embodiment of the present application.

As shown in Figure 7, the method may include the following steps:

S701. The cloud server 200 acquires multiple high-quality images and a shooting target category of each high-quality image.

The photographing target categories may include human faces, buildings, animal cats, animal dogs, animal birds, and the like.

Possibly, in order to subdivide the types of shooting objects, the shooting object category may include multiple sub-categories. For example, for a face photographing target, sub-categories of skin color, age group, gender, race, etc. may be included. For well-known architectural targets, subcategories of perspective, season, lighting environment, etc. can be included.

For different shooting targets, the cloud server 200 can use different feature extraction networks to extract high-definition texture features in high-quality images. For example, to extract facial features in high-quality images, factors such as age, skin color, facial features, and gender need to be taken into account. The cloud server 200 can use a face recognition model as a feature extractor; if extracting plant features, it is necessary to consider the color, shape, texture and other characteristics of plants, and a plant classification network can be used as a feature extractor.

S702. The cloud server 200 extracts the texture features of the shooting target in the high-quality image, and establishes a plurality of high-definition texture dictionary libraries of different shooting target categories.

For each shooting target category, the cloud server 200 can extract high-definition texture features of the shooting target category from a plurality of high-quality images under the shooting target category through the feature extractor corresponding to the shooting target category. The cloud server 200 can perform clustering processing on the high-definition texture features under the shooting target category to obtain the most representative high-definition texture features. The cloud server 200 may store the most representative high-definition texture features under the same shooting target category in the corresponding high-definition texture dictionary library under the shooting target category.

Exemplarily, for example, for multiple high-quality face images whose target type is a human face, the cloud server 200 may perform the following operations for each high-quality face image: the cloud server 200 may extract a high-quality face image. Features at different scales. Then, the cloud server 200 can detect the landmark points of the face in the high-quality image of the face, and crop and resample the left eye, right eye, nose, mouth and other facial features at each scale, so that these The features of the facial features reach a fixed size. Then, the cloud server 200 may generate K clusters for each facial feature through the K-means algorithm, and obtain high-definition texture features of each facial feature. The cloud server 200 can extract and store the high-definition texture features of each facial feature into the high-definition texture dictionary library corresponding to the face category.

Among them, the multiple high-definition texture dictionary libraries of different shooting target categories established on the cloud server 200 can be as shown in Table 1 below:

Table 1

拍摄目标类别Shooting target category	高清纹理字典库HD Texture Dictionary Library
人脸human face	高清纹理字典库1HD Texture Dictionary Library 1
建筑architecture	高清纹理字典库2HD Texture Dictionary Library 2
绿植green plants	高清纹理字典库3HD Texture Dictionary Library 3
动物老鹰animal eagle	高清纹理字典库4HD Texture Dictionary Library 4
……	……

As can be seen from the above Table 1, the cloud server 200 may establish a high-definition texture dictionary library 1, a high-definition texture dictionary library 2, a high-definition texture dictionary library 3, a high-definition texture dictionary library 4, and so on. Among them, the shooting target category of HD texture dictionary library 1 is human face, the shooting target category of HD texture dictionary library 2 is buildings, the shooting target category of HD texture dictionary library 3 is green plants, and the shooting target category of HD texture dictionary library 4 is animals eagle. The above-mentioned Table 1 is only used to exemplify the present application, and should not be construed as a limitation.

In some embodiments, in order to subdivide the types of objects, the category of objects may include multiple sub-categories. For example, for a face photographing target, one or more sub-categories of skin color, age group, gender, race, etc. may be included. For well-known architectural targets, subcategories of perspective, season, lighting environment, etc. can be included. Among them, the multiple high-definition texture dictionary libraries of different shooting target categories established on the cloud server 200 can be shown in Table 2 below:

Table 2

As can be seen from the above Table 1, the cloud server 200 can establish a high-definition texture dictionary library 1, a high-definition texture dictionary library 2, a high-definition texture dictionary library 3, a high-definition texture dictionary library 4, a high-definition texture dictionary library 5, and a high-definition texture dictionary library. 6. HD texture dictionary library 7, HD texture dictionary library 8, HD texture dictionary library 9, HD texture dictionary library 10, HD texture dictionary library 11, HD texture dictionary library 12, HD texture dictionary library 13, HD texture dictionary library 14, HD texture dictionary library 15, HD texture dictionary library 16, HD texture dictionary library 17, HD texture dictionary library 18, etc. Among them, the shooting target category of the high-definition texture dictionary library 1 is a young male face with yellow skin. The shooting target category of HD Texture Dictionary Library 2 is the face of a young woman with yellow skin. The target category of HD Texture Dictionary Library 3 is a middle-aged male face with yellow skin. The target category of HD Texture Dictionary Library 4 is a middle-aged female face with yellow skin. The shooting target category of HD Texture Dictionary Library 5 is the face of an elderly male with yellow skin. The shooting target category of HD Texture Dictionary Library 6 is the face of an elderly woman with yellow skin. The shooting target category of HD Texture Dictionary Library 7 is the face of a young male with black skin. The shooting target category of HD Texture Dictionary Library 8 is the face of a young woman with black skin. The shooting target category of HD Texture Dictionary Library 9 is a middle-aged male face with black skin. The target category of HD Texture Dictionary Library 10 is middle-aged female faces with dark skin. The shooting target category of HD Texture Dictionary Library 11 is the face of an elderly male with black skin. The shooting target category of HD Texture Dictionary Library 12 is the face of an elderly woman with black skin. The shooting target category of HD Texture Dictionary Library 13 is the face of a young male with white skin. The shooting target category of HD Texture Dictionary Library 14 is the face of a young woman with white skin. The target category of HD Texture Dictionary Library 15 is a middle-aged male face with white skin. The shooting target category of HD Texture Dictionary Library 16 is the face of a middle-aged woman with white skin. The shooting target category of HD Texture Dictionary Library 17 is the face of an elderly male with white skin. The shooting target category of HD Texture Dictionary Library 18 is the face of an elderly woman with white skin. The above-mentioned Table 2 is only used to exemplify the present application, and should not be construed as a limitation.

S703. The electronic device 100 receives the first input.

S704, the electronic device 100 acquires a low-definition image in response to the first input.

Exemplarily, the first input may be the input for the shooting control 322 in the shooting preview interface 320 shown in FIG. 3B , the input for the shooting control 422 in the shooting preview interface 420 shown in FIG. 4B , or the input for the shooting control 422 in the shooting preview interface 420 shown in FIG. 4B . The input of the shooting control 522 in the shooting preview interface 520 shown in FIG. 5B above. In response to the first input, the electronic device 100 may acquire a low-definition image captured by the camera.

In a possible implementation manner, the first input may also be an input for the super-score reconstruction control 624 in the picture browsing interface 620 shown in FIG. 6C. In response to the first input, the electronic device 100 may determine the picture 621 displayed in the picture browsing interface 620 shown in FIG. 6C as a low-definition image.

S705, the electronic device 100 determines whether the low-definition image includes the specified target content.

Specifically, identification algorithm models corresponding to different shooting target categories are preset on the electronic device 100 . The electronic device 100 can detect whether the specified target content is included in the low-definition image through the recognition algorithm models corresponding to these different shooting target categories. Wherein, the category of the specified target content may include any one of image contents such as human faces, buildings, green plants, animal cats, animal dogs, and animal birds.

In a possible implementation manner, in order to subdivide the type of the shooting target, the type of the specified target content may include multiple sub-types. For example, for a face photographing target, subtypes such as skin color, age group, gender, race, etc. may be included. For well-known architectural targets, subtypes of perspective, season, lighting environment, etc. can be included.

In a possible implementation manner, the set of shooting target categories that can be identified by the multiple recognition algorithm models preset on the electronic device 100 is the same as the set of categories of the high-definition texture dictionary stored on the cloud server 200. For example, the set of photographing target categories that can be identified by the multiple identification algorithm models on the electronic device 100 may include human faces, buildings, and the like. The set of shooting target categories of the high-definition texture dictionary library stored on the cloud server 200 may also include human faces, buildings, and the like.

In a possible implementation manner, when the cloud server 200 detects that the category set of the high-definition texture dictionary library is different from the identification category set of the target recognition model in the electronic device 100, the cloud server 200 may send the target recognition to the electronic device 100 Model update information. The electronic device 100 may update the target recognition algorithm model according to the update information. In the updated target recognition algorithm model, the set of photographing target categories that can be identified is the same as the set of categories of the high-definition texture dictionary stored on the cloud server 200 .

In a possible implementation manner, a plurality of recognition algorithm models on the electronic device 100 can recognize the category of the shooting target. The multiple high-definition texture dictionary libraries on the cloud server 200 also have multiple sub-categories under the category of each shooting target. For example, the set of photographing target categories that can be identified by the multiple identification algorithm models of the electronic device 100 may include human faces, buildings, and the like. The categories of the multiple high-definition texture dictionary libraries stored on the cloud server 200 may include human faces, buildings, and the like. The face category may further include a yellow-skinned human face, a dark-skinned human face, a white-skinned human face, and the like. After the electronic device 100 sends the face category and image features to the cloud server 200, the cloud server 200 can first filter out the high-definition texture dictionary library corresponding to the face of the yellow-skinned race, the high-definition texture dictionary library corresponding to the face of the dark-skinned race, A high-definition texture dictionary library corresponding to a white-skinned human face. Then, the cloud server 200 can identify the race and skin color in the image features uploaded by the electronic device 100, and based on the race skin color, determine the same race as the image features from multiple high-definition texture dictionary libraries under the category of human faces. A library of HD texture dictionaries for skin tones.

In this embodiment of the present application, the specified target content in the low-definition image may be referred to as the first specified target content.

If the low-definition image includes the specified target content, the electronic device 100 may execute S706, and the electronic device 100 determines that the category of the specified target content in the low-definition image is the first category and the first region where the specified target content is located.

S707. The electronic device 100 may crop out the first region where the specified target content is located from the low-definition image, to obtain a first low-definition cropped image.

Exemplarily, the low-definition image may be as shown in FIG. 8A . The electronic device 100 generates a binary mask image of the low-definition image after identifying the first area where the specified target content (eg, eagle) of the low-definition image is located.

Wherein, as shown in FIG. 8B , in the binary mask image, the first area where the specified target content is located in the low-definition image may be displayed as white, and other locations may be displayed as black. The electronic device 100 may determine the coordinates of the cropping frame in the low-definition image based on the binary mask image, where the position of the cropping frame includes the first region. The position of the cropping frame in the low-definition image can be as shown in FIG. 8C .

The electronic device 100 may, based on the cropping frame, crop out the first region where the specified target content is located from the low-definition image, to obtain a first low-definition cropped image. The first low-definition cropped image may be as shown in FIG. 8D .

S708. The electronic device 100 may extract the first image feature of the low-definition cropped image.

The first image feature may include a feature vector or a feature map of information such as contour information, color information, and texture information of the specified target content in the low-definition cropped image.

S709 , the electronic device 100 may send the first image feature and the first category of the first low-definition cropped image to the cloud server 200 .

S710. The cloud server 200 may determine that the shooting target category is the first high-definition texture dictionary library of the first category.

S711. The cloud server 200 may match a first high-definition texture feature similar to the first image feature from the first high-definition texture dictionary library.

The first high-definition texture feature may include a feature vector or a feature map of information such as high-definition outline information, high-definition color information, and high-definition texture information corresponding to the specified target content.

Specifically, the cloud server 200 may match a first high-definition texture feature whose content similarity with the first image feature is greater than a preset value from the first high-definition texture dictionary library.

S712 , the cloud server 200 may send the first high-definition texture feature to the electronic device 100 .

S713. The electronic device 100 may fuse the first high-definition texture feature into the first low-definition cropped image to obtain a first high-definition cropped image.

S714. The electronic device 100 replaces the first high-definition cropped image with the first low-definition cropped image, and pastes it back to the first area in the low-definition image to obtain a high-definition image.

Exemplarily, the first high-definition cropped image may be as shown in FIG. 8E . Wherein, the resolution of the first high-definition cropped image is greater than the resolution of the first low-definition cropped image. The high-definition image may be as shown in Figure 8F.

For another example, when the specified target category included in the low-definition image is a human face, the first may refer to the region where the human face is located in the low-definition image. The electronic device 100 can extract face key points from the low-definition image. The electronic device 100 may acquire facial feature information and hair information, etc. in the low-definition image based on the key points of the face. The electronic device 100 can generate a binary mask image of the low-definition image based on the facial features information, wherein, in the binary mask image of the low-definition image, the display color of the location where the facial features are located can be set to white, and the display color of other locations can be set to white. Set to black. Therefore, the electronic device 100 can determine the coordinates of the face frame in the low-definition image based on the binary mask image. The electronic device 100 can enlarge the area of the face frame on the low-definition image. The electronic device 100 may, based on the enlarged face frame, crop out the image in the face frame to obtain the first low-definition cropped image. The electronic device 100 extracts feature information of each facial feature position from the low-definition image based on the facial feature information. Then, the electronic device 100 may send the feature information of each facial feature location and the face category to the cloud server 200 . The cloud server 200 can match the high-definition texture dictionary library corresponding to the face category. The cloud server 200 may match the high-definition texture features corresponding to the feature information of each facial feature based on the high-definition texture dictionary library corresponding to the face category. The cloud server 200 may send the high-definition texture features of each facial feature to the electronic device 100 . The electronic device 100 may fuse the corresponding high-definition texture features of each facial feature into the first low-definition cropped image to obtain the first high-definition cropped image. Then, the electronic device 100 may base the first high-definition cropped image on the binary mask image of facial feature information, and paste the first high-definition cropped image back to the face region in the low-definition image to obtain a high-definition image.

S715. The electronic device 100 may save a high-definition image.

In a possible implementation manner, the electronic device 100 may also save the above-mentioned low-definition image while saving the high-definition image.

In a possible implementation manner, the electronic device 100 can also display the high-definition image while saving the high-definition image.

In a possible implementation manner, if the low-definition image does not include the specified target content, the electronic device 100 may perform super-resolution reconstruction on the low-definition image by using a general super-resolution processing strategy to obtain a high-definition image. Among them, the general super-resolution processing strategy includes inputting low-definition images into the super-resolution network model for super-resolution reconstruction to obtain high-definition images. Among them, the super-resolution network model includes the use of super-resolution reconstruction generative adversarial networks (super-resolution generative adversarial networks, SRGAN), enhanced super-resolution reconstruction generative adversarial networks (super-resolution generative adversarial networks, ESRGAN), broad Activation of wide activation for efficient and accurate image super-resolution (WDSR) networks, etc.

In some embodiments, when the electronic device 100 can identify multiple specified target contents from the acquired low-definition image, the electronic device 100 can extract image features of the multiple specified target contents and the category of each specified target content respectively. . The categories of the multiple specified target contents may all be the same, may be different, may be the same, or may be completely different. The electronic device 100 may transmit the image feature of each specified target content and the category of each specified target content to the cloud server 200 . The cloud server 200 can match the high-definition texture dictionary library corresponding to each category of the specified target content, and match the high-definition texture feature of each specified target content based on the corresponding high-definition texture dictionary library of each category of the specified target content. The cloud server 200 can send the high-definition texture features of each specified target content to the electronic device 100, and the electronic device 100 can fuse the respective high-definition texture features of the multiple specified target contents into the respective high-definition texture features of the multiple specified target contents in the low-definition image. In the area, high-definition images are obtained. In this way, the electronic device 100 can synchronously perform super-resolution reconstruction processing on multiple designated targets in the low-definition image, thereby improving the super-resolution processing efficiency.

In some embodiments, after acquiring the low-definition image, the electronic device 100 may identify and crop out the region where the specified target content is located from the low-definition image, to obtain the low-definition cropped image and the category of the specified target content. The electronic device 100 may transmit the low-definition cropped image and the category of the designated target content to the cloud server 200 . The cloud server 200 may extract the first image feature from the low-definition cropped image. A first high-definition texture feature whose type of the shooting target is the same as the category of the specified target content and whose content similarity with the first image feature is greater than a preset value is matched. The cloud server 200 may fuse the first high-definition texture feature into the first low-definition cropped image to obtain a first high-definition cropped image. The cloud server 200 may send the first high-definition cropped image to the electronic device 100 . The electronic device 100 may paste the first high-definition cropped image back to the first region in the low-definition image where the specified target content is located, to obtain a high-definition image.

In some embodiments, the cloud server 200 may store multiple high-definition image libraries of different shooting target types, and the high-definition image library of each shooting target type includes multiple high-definition reference images under the shooting target type. After acquiring the low-definition image, the electronic device 100 can identify and crop out the region where the specified target content is located from the low-definition image, and obtain the low-definition cropped image and the category of the specified target content. The electronic device 100 may transmit the low-definition cropped image and the category of the designated target content to the cloud server 200 . Based on the low-definition cropped image and the category of the specified target content, the cloud server 200 can match a high-definition guide image with the same shooting target type as the specified target content category and with a content similarity greater than a preset value in the low-definition cropped image. The cloud server 200 may send the high-definition guide image to the electronic device 100 . The electronic device 100 may fuse the high-definition texture features in the high-definition guide image into the low-definition cropped image to obtain a high-definition cropped image. The electronic device 100 can paste the high-definition cropped image back to the low-definition image where the specified target content is located to obtain a high-definition image.

Optionally, the cloud server 200 may, based on the high-definition guide image, fuse high-definition texture features in the high-definition guide image into a low-definition cropped image to obtain a high-definition cropped image. The cloud server 200 may send the high-definition cropped image to the electronic device 100 . The electronic device 100 can paste the high-definition cropped image back to the low-definition image where the specified target content is located to obtain a high-definition image.

With the image reconstruction method provided in the embodiment of the present application, the electronic device 100 extracts the image features of the specified target content from the acquired low-definition image, and based on the high-definition texture dictionary of the same category as the specified target content on the cloud server library to match high-definition texture features similar to the image features of the specified target content. The electronic device can fuse the high-definition texture features into a low-definition image to obtain a high-definition image. In this way, the super-resolution reconstruction can be performed specifically for a specific target, which simplifies the calculation amount on the electronic device 100 during the super-resolution reconstruction process.

The following introduces an image reconstruction method provided by another embodiment of the present application.

FIG. 9 shows a schematic flowchart of an image reconstruction method provided by an embodiment of the present application. As shown in Figure 9, the method may include the following steps:

S901. The cloud server 200 acquires multiple high-quality images and a shooting target category of each high-quality image.

For specific content, reference may be made to step S701 in the foregoing embodiment shown in FIG. 7 .

S902. The cloud server 200 can extract the texture features of the shooting target in the high-quality image, and establish a plurality of high-definition texture dictionary libraries of different shooting target categories.

For specific content, reference may be made to step S702 in the foregoing embodiment shown in FIG. 7 .

S903 , the electronic device 100 displays a shooting preview interface, where a shooting key and a first preview image are displayed on the shooting preview interface.

Exemplarily, the shooting preview interface may be the shooting preview interface 320 shown in FIG. 3B, the shooting preview interface 420 shown in FIG. 4B, or the shooting preview interface 520 shown in FIG. 5B. .

S904, the electronic device 100 determines whether the first preview image contains the specified target content.

Wherein, the type of the specified target content may include any one of image content such as human faces, buildings, green plants, animal cats, animal dogs, and animal birds.

In this embodiment of the present application, the specified target content in the first preview image may be referred to as the second specified target content.

If the first preview image contains the specified target content, then execute S905, and the electronic device 100 may determine that the category of the specified target content in the first preview image is the second category and the second region where the specified target content is located.

S906 , the electronic device 100 crops out the second region where the specified target content is located from the first preview image, and obtains a second low-definition cropped image.

The electronic device 100 may generate a binary mask image of the first preview image after identifying the second region where the specified target content of the first preview image is located. The binary mask image of the first preview image can display the second area where the specified target content is located in the low-definition image as white, and display other locations as black. The electronic device 100 may determine the coordinates of the cropping frame in the first preview image based on the binary mask image of the first preview image. The position of the cropping frame in the first preview image includes the second area. The electronic device 100 may, based on the cropping frame, crop out the second area where the specified target content is located from the first preview image to obtain a second low-definition cropped image.

S907 , the electronic device 100 extracts the second image feature of the second low-definition cropped image.

S908 , the electronic device 100 sends the second image feature and the second category to the cloud server 200 .

S909 , the cloud server 200 determines that the shooting target category is the second high-definition texture dictionary library of the second category.

S910. The cloud server 200 matches a second high-definition texture feature similar to the second image feature from the second high-definition texture dictionary library.

Specifically, the cloud server 200 may match a second high-definition texture feature whose content similarity with the second image feature is greater than a preset value from the second high-definition texture dictionary library.

S911 , the cloud server 200 sends the second high-definition texture feature to the electronic device 100 .

S912, the electronic device 100 saves the second high-definition texture feature to the cache queue.

Wherein, the cache queue may be a storage area in the running memory. After acquiring the second high-definition texture feature, the electronic device 100 may save the second high-definition texture feature into a cache queue.

In some embodiments, the electronic device 100 may also acquire a second preview image captured by the camera, and the electronic device 100 may determine whether the second preview image contains the specified target content, and if so, the electronic device 100 may determine the second preview image The category of the specified target content in the image is the third category and the third area where the specified target content is located. The electronic device 100 may crop out the third area where the specified target content is located from the second preview image to obtain a third low-definition cropped image. The electronic device 100 may extract the third image feature of the third low-definition cropped image. The electronic device 100 can determine whether the similarity between the third image feature and the second high-definition texture feature stored in the cache queue is greater than a preset value. In the third low-definition cropped image, a third high-definition cropped image is obtained.

In this embodiment of the present application, the specified target content in the third preview image may be referred to as the third specified target content.

When the similarity between the third image feature and the second high-definition texture feature stored in the cache queue is less than the preset value, the electronic device 100 may send the third image feature and the third category to the cloud server 200 . The cloud server 200 determines a third high-definition texture dictionary corresponding to the third category, and matches a third high-definition texture feature similar to the third image feature from the third high-definition texture dictionary. The cloud server 200 may send the third high-definition texture feature to the electronic device 100 . After acquiring the third high-definition texture feature, the electronic device 100 may fuse the third high-definition texture feature into the third low-definition cropped image to obtain a third high-definition cropped image.

After obtaining the third high-definition cropped image, the electronic device 100 may paste the third high-definition cropped image back to the third area in the second preview image to obtain a high-definition preview image. The electronic device 100 may display a high-definition preview image on the shooting preview interface.

In this way, the resolution of the preview image during the shooting preview process of the electronic device 100 can be improved.

S913, the electronic device 100 receives the first input, and acquires a low-definition image.

S914 , the electronic device 100 determines whether the low-definition image contains the specified target content.

If yes, execute S915, the electronic device 100 determines that the category of the specified target content in the low-definition image is the first category and the first region where the specified target content is located.

S916 , the electronic device 100 crops out the first region where the specified target content is located from the low-definition image, and obtains a first low-definition cropped image.

For specific content, reference may be made to step S707 in the aforementioned embodiment shown in FIG. 7 , which will not be repeated here.

S917. The electronic device 100 may extract the first image feature of the first low-definition cropped image.

For specific content, reference may be made to step S708 in the foregoing embodiment shown in FIG. 7 , which will not be repeated here.

S918. The electronic device 100 may determine whether the similarity between the first image feature and the high-definition texture feature in the cache queue is greater than a preset value.

If the similarity between the first image feature and the high-definition texture feature in the cache queue is greater than the preset value, then perform S919, and the electronic device 100 may fuse the high-definition texture feature in the cache queue into the first low-definition cropped image to obtain the first low-definition cropped image. HD cropped images.

For example, if the second high-definition texture feature is stored in the cache queue, the electronic device 100 can determine whether the similarity between the first image feature and the second high-definition texture feature is greater than a preset value. The second high-definition texture feature is fused into the first cropped image to obtain the first high-definition cropped image.

For another example, the second high-definition texture feature and the third high-definition texture feature are stored in the cache queue, and the electronic device 100 may calculate the similarity between the second high-definition texture feature and the second high-definition texture feature respectively for the first image feature. If the similarity between the first image feature and the second high-definition texture feature is greater than the preset value, and the similarity with the third high-definition texture feature is less than or equal to the preset value, the electronic device 100 may fuse the second high-definition texture feature into the first cropping image, get the first high-definition cropped image.

If the similarity between the first image feature and the second high-definition texture feature is less than or equal to the preset value, and the similarity with the third high-definition texture feature is greater than the preset value, the electronic device 100 may fuse the third high-definition texture feature into the first cropping image, get the first high-definition cropped image.

If the similarity between the first image feature and the second high-definition texture feature is greater than the preset value, and the similarity with the third high-definition texture feature is greater than the preset value, the electronic device 100 may compare the first image feature and the second high-definition texture feature and the similarity between the first image feature and the third high-definition texture feature. When the similarity between the first image feature and the second high-definition texture feature is greater than the similarity between the first image feature and the third high-resolution texture feature, the electronic device 100 may fuse the second high-definition texture feature into the first cropped image to obtain the first image feature. One HD cropped image. When the similarity between the first image feature and the second high-definition texture feature is less than or equal to the similarity between the first image feature and the third high-resolution texture feature, the electronic device 100 may fuse the third high-definition texture feature into the first cropped image to obtain First HD cropped image.

If the similarity between the first image feature and the second high-definition texture feature is greater than the preset value, steps S920 to S924 are performed. in:

S920 , the electronic device 100 may send the first image feature and the first category to the cloud server 200 .

The first category, the second category and the third category may be the same or different.

S921. The electronic device 100 may determine that the shooting target category is the first high-definition texture dictionary library of the first category.

For specific content, reference may be made to the aforementioned step S710 shown in FIG. 7 , which will not be repeated here.

S922. The cloud server 200 may match a first high-definition texture feature similar to the first image feature from the first high-definition texture dictionary library.

S923 , the cloud server 200 may send the first high-definition texture feature to the electronic device 100 .

S924. The electronic device 100 may fuse the first high-definition texture feature into the first low-definition cropped image to obtain a first high-definition cropped image.

S925. The electronic device 100 may replace the first high-definition cropped image with the first low-definition cropped image, and paste it back to the first area in the low-definition image to obtain a high-definition image.

For specific content, reference may be made to step S714 in the foregoing embodiment shown in FIG. 7 , which will not be repeated here.

S926, the electronic device 100 can save the high-definition image.

For specific content, reference may be made to step S715 in the foregoing embodiment shown in FIG. 7 , which will not be repeated here.

With the image reconstruction method provided in the embodiment of the present application, the electronic device 100 can use the image features of the specified target content in the preview screen captured by the camera before taking a picture, to retrieve the high-definition texture dictionary of the same category as the specified target content from the cloud server library to match high-definition texture features similar to the image features of the specified target content. When taking a photo, the electronic device can fuse the high-definition texture feature into the captured low-definition image to obtain a high-definition image. In this way, high-definition texture features can be obtained in advance by using the specified target content in the preview image and merged into the low-definition image taken, which improves the resolution of the photo taken by the electronic device 100 and shortens the time in the super-resolution reconstruction process.

FIG. 10 shows a schematic flowchart of an image reconstruction method provided by an embodiment of the present application. As shown in Figure 10, the method may include the following steps:

S1001. The cloud server 200 acquires multiple high-quality images and a shooting target category of each high-quality image.

S1002. The cloud server 200 may extract the texture features of the shooting target in the high-quality image, and establish a plurality of high-definition texture dictionary libraries of different shooting target categories.

S1003. The cloud server 200 may send a dictionary library resource package to the electronic device 100, where the dictionary library resource package includes multiple high-definition texture dictionary libraries.

S1004, the electronic device receives the first input.

S1005, the electronic device 100 acquires a low-definition image in response to the first input.

S1006, the electronic device 100 determines whether the low-definition image contains the specified target content.

If the low-definition image contains the specified target content, step S1007 is executed, and the electronic device 100 determines that the category of the specified target content in the low-definition image is the first category.

S1008. The electronic device 100 may determine whether the dictionary library resource package includes the first high-definition texture dictionary library of the first category.

If the dictionary library resource package does not include the first high-definition texture dictionary library of the first category, S1009 is performed, and the electronic device 100 may determine whether to download the first high-definition texture dictionary library according to the user's operation. If downloading is required, S1010 is executed, and the electronic device 100 sends the first category to the cloud server 200 .

S1011. After receiving the first category, the cloud server 200 may determine that the shooting target category is the first high-definition texture dictionary library of the first category.

S1012 The cloud server 200 may send the first high-definition texture dictionary library to the electronic device 100 .

In a possible implementation manner, if the electronic device 100 determines not to download the first high-definition texture dictionary library according to the user's operation, the electronic device 100 can perform super-resolution reconstruction on the low-definition image by using a general super-resolution processing strategy to obtain a high-definition image . Among them, the general super-resolution processing strategy includes inputting low-definition images into the super-resolution network model for super-resolution reconstruction to obtain high-definition images. Among them, the super-resolution network model includes the use of super-resolution reconstruction generative adversarial network (SRGAN), enhanced super-resolution reconstruction generative adversarial network (ESRGAN), wide activation of efficient and accurate image super-resolution reconstruction (WDSR) network, and many more.

In a possible implementation manner, if the dictionary library resource package does not include the first high-definition texture dictionary library of the first category, the electronic device 100 may skip step S1009 and directly execute the above-mentioned step S1010.

If the dictionary library resource package includes the first high-definition texture dictionary library of the first category, the electronic device 100 may directly execute S1013.

S1013 , after acquiring the first high-definition texture dictionary library, the electronic device 100 may crop out the first region where the specified target content is located from the low-definition image, to obtain a first low-definition cropped image.

For specific content, reference may be made to step S707 in the aforementioned embodiment shown in FIG. 7 , and details are not repeated here.

S1014. The electronic device 100 may extract the first image feature of the first low-definition cropped image.

For specific content, reference may be made to step S708 in the aforementioned embodiment shown in FIG. 7 , and details are not repeated here.

S1015. The electronic device 100 may match a first high-definition texture feature similar to the first image feature from the first high-definition texture dictionary library.

For specific content, reference may be made to the processing process of the cloud server 200 in step S711 of the foregoing embodiment shown in FIG. 7 , which will not be repeated here.

S1016. The electronic device 100 may fuse the first high-definition texture feature into the first low-definition cropped image to obtain a first high-definition cropped image.

For specific content, reference may be made to step S713 in the aforementioned embodiment shown in FIG. 7 , and details are not repeated here.

S1017. The electronic device 100 may replace the first high-definition cropped image with the first low-definition cropped image, and paste it back to the first area in the low-definition image to obtain a high-definition image.

For specific content, reference may be made to step S714 in the aforementioned embodiment shown in FIG. 7 , which will not be repeated here.

S1018, the electronic device 100 saves a high-definition image.

For specific content, reference may be made to step S715 in the foregoing embodiment shown in FIG. 7 , and details are not repeated here.

In some embodiments, after acquiring the low-definition image, the electronic device 100 may identify and crop out the region where the specified target content is located from the low-definition image, to obtain the low-definition cropped image and the category of the specified target content. The electronic device 100 may transmit the low-definition cropped image and the category of the designated target content to the cloud server 200 . The cloud server 200 may extract the first image feature from the low-definition cropped image. A first high-definition texture feature whose type of the shooting target is the same as the category of the specified target content and whose content similarity with the first image feature is greater than a preset value is matched. The cloud server 200 may fuse the first high-definition texture feature into the first low-definition cropped image to obtain a first high-definition cropped image. The cloud server 200 may send the first high-definition cropped image to the electronic device 100. The electronic device 100 may paste the first high-definition cropped image back to the first region in the low-definition image where the specified target content is located, to obtain a high-definition image.

With the image reconstruction method provided by the embodiment of the present application, the electronic device 100 can download the offline dictionary library resource package from the cloud server 200 in advance. The electronic device 100 can use the image features of the specified target content in the preview screen captured by the camera before taking the picture, from the high-definition texture dictionary library of the same category as the specified target content in the offline dictionary library resource package, and match the image of the specified target content. Features similar HD texture features. When taking a photo, the electronic device can fuse the high-definition texture feature into the captured low-definition image to obtain a high-definition image. In this way, high-definition texture features can be obtained in advance by using the specified target content in the preview image and merged into the low-definition image taken, which improves the resolution of the photo taken by the electronic device 100 and shortens the time in the super-resolution reconstruction process.

FIG. 11 shows a schematic flowchart of an image reconstruction method provided by an embodiment of the present application. As shown in Figure 11, the method may include the following steps:

S1101. The electronic device 100 may store a plurality of high-definition texture dictionary libraries of different shooting target categories.

S1102. The electronic device 100 receives the first input.

S1103. The electronic device 100 acquires a low-definition image in response to the first input.

S1104. The electronic device 100 determines whether the low-definition image contains the specified target content.

If the low-definition image contains the specified target content, execute S1105, and the electronic device 100 determines that the category of the specified target content in the low-definition image is the first category.

S1106. The electronic device 100 determines, from a plurality of high-definition texture dictionary libraries of different shooting target categories, that the shooting target category is the first high-definition texture dictionary library of the first category.

S1107. The electronic device 100 may crop out the first region where the specified target content is located from the low-definition image, to obtain a first low-definition cropped image.

S1108. The electronic device 100 may extract the first image feature of the first low-definition cropped image.

S1109. The electronic device 100 matches a first high-definition texture feature similar to the first image feature from the first high-definition texture dictionary library.

For details, refer to the process in step S711 of the aforementioned embodiment shown in FIG. 7 that the cloud server 200 matches the first high-definition texture feature similar to the first image feature from the first high-definition texture dictionary library, which will not be repeated here.

S1110. The electronic device 100 may fuse the first high-definition texture feature into the first cropped image to obtain a first high-definition cropped image.

S1111. The electronic device 100 may replace the first high-definition cropped image with the first low-definition cropped image, and paste it back to the first area in the low-definition image to obtain a high-definition image.

When the low-definition image does not contain the specified target content, the electronic device 100 may perform step S1112, and the electronic device 100 processes the low-definition image through a general super-resolution processing strategy to obtain a high-definition image.

Among them, the general super-resolution processing strategy includes inputting low-definition images into the super-resolution network model for super-resolution reconstruction to obtain high-definition images. Among them, the super-resolution network model includes the use of super-resolution reconstruction generative adversarial network (SRGAN), enhanced super-resolution reconstruction generative adversarial network (ESRGAN), wide activation of efficient and accurate image super-resolution reconstruction (WDSR) network, and many more.

S1013, the electronic device 100 saves a high-definition image.

With the image reconstruction method provided by the embodiment of the present application, the electronic device 100 may store a plurality of high-definition texture dictionary libraries of different shooting target categories. The electronic device 100 can use the image features of the specified target content in the preview screen captured by the camera before taking pictures to match the high-definition texture dictionary library of the same category as the specified target content from the high-definition texture dictionary libraries of multiple different shooting target categories. High-definition texture features similar to the image features of the specified target content. When taking a picture, the electronic device can fuse the high-definition texture feature into the captured low-definition image to obtain a high-definition image. In this way, high-definition texture features can be obtained in advance by using the specified target content in the preview image and merged into the low-definition image taken, which improves the resolution of the photo taken by the electronic device 100 and shortens the time in the super-resolution reconstruction process.

The following describes an image reconstruction system 1200 involved in the embodiments of the present application.

FIG. 12 shows a schematic structural diagram of an image reconstruction system 1200 provided in an embodiment of the present application. The image reconstruction system 1200 may include the electronic device 100 and the cloud server 200 .

As shown in FIG. 12, the electronic device 100 may include a camera 1211, an image signal processor (ISP) 1212, a digital signal processor (DSP) 1213, a target recognition module 1214, a feature extraction module 1215, a feature fusion module 1216, and image stitching Module 1217. Wherein, in some embodiments, the target recognition module 1214, the feature extraction module 1215, the feature fusion module 1216 and the image stitching module 1217 may be a neural network computing processor (NPU) or a graphics processing unit (GPU). In other embodiments, the target recognition module 1214 , the feature extraction module 1215 , the feature fusion module 1216 and the image stitching module 1217 may also be software processing modules in the application processor 1215 .

The cloud server 200 may include a plurality of high-definition texture dictionary libraries of different shooting target types, a dictionary library matching module 1221, and a feature matching module 1222.

Wherein, the camera 1211 can be used to capture optical signals when starting the camera application or function, convert the optical signals into electrical signals, and send them to the image signal processor 1212 .

The image signal processor 912 can be used to convert the electrical signal sent by the camera 1211 into a digital image signal and send the digital image signal to the digital signal processor 1213 .

The digital signal processor 1213 can be used to process the digital image signal into a preview image stream of a specified image format, and the specified image format can be Raw format, YUV format, RGB format, and so on.

The target identification module 1214 can be configured to obtain the captured low-definition image from the book signal processor 1213 based on the shooting instruction triggered by the user operation, and identify the specified target content of the first category. The target identification module 1214 can also identify the first region where the specified target content is located in the low-definition image, and generate a binary mask image. The target identification module 1214 can identify and crop out the first region in the low-definition image to obtain the first low-definition cropped image.

The feature extraction module 1215 may extract first image features from the first low-definition cropped image.

The electronic device 100 may send the first category identified by the target identification module 1214 to the cloud server 200 . The electronic device 100 may send the first image feature (or the first low-definition cropped image) to the cloud server 200 . in:

The dictionary library matching module 1221 can select the first high-definition texture dictionary library of the first category from a plurality of high-definition texture dictionary libraries of different shooting target categories and send it to the feature matching module 1222.

The feature matching module 1222 may be configured to match a first high-definition texture feature similar to the first image feature from the first high-definition texture dictionary library. Optionally, after identifying the first high-definition texture features with similar first image features, the feature matching module 1222 may fuse the first high-definition texture features into the first low-definition cropped image to obtain the first high-definition cropped image.

The cloud server 200 may send the first high-definition texture feature (or the first high-definition cropped image) to the electronic device 100 . in:

The feature fusion module 1216 may fuse the first high-definition texture feature into the first low-definition cropped image to obtain the first high-definition cropped image.

The image stitching module 1217 can use the binary mask image to obtain the first area in the low-definition image where the specified target content is located, and paste the first high-definition cropped image back to the first area in the low-definition image to obtain a high-definition image.

For parts that are not described in detail, reference may be made to the foregoing embodiments, which will not be repeated here.

As mentioned above, the above embodiments are only used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand: The technical solutions described in the embodiments are modified, or some technical features thereof are equivalently replaced; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

An image reconstruction method, comprising:

the electronic device receives the first input;

The electronic device, in response to the first input, acquires a low-definition image captured by the camera;

The electronic device detects that the low-definition image includes first specified target content, and the category of the first specified target content is the first category;

The electronic device extracts the first image feature of the first specified target content in the low-definition image;

The electronic device sends the first image feature and the first category to the cloud server; wherein, the first image feature is used by the cloud server to store a plurality of high-definition texture dictionary libraries of different shooting target categories The first high-definition texture dictionary library whose shooting target category is the first category is matched in the middle; the first image feature is used by the cloud server to match the first high-definition texture dictionary library with the first image feature from the first high-definition texture dictionary library. the first high-definition texture feature whose similarity is greater than the preset value;

The electronic device receives the first high-definition texture feature sent by the cloud server;

The electronic device fuses the first high-definition texture feature into the low-definition image to obtain a high-definition image, and the resolution of the first specified target content in the high-definition image is greater than the resolution of the first specified target content in the low-definition image. - Specifies the resolution of the target content.
The method according to claim 1, wherein the electronic device extracts the first image feature of the first specified target content in the low-definition image, which specifically includes:

The electronic device identifies and crops out the first area in the low-definition image where the first specified target content is located, and obtains a first low-definition cropped image, where the first low-definition cropped image includes the first specified target content;

The electronic device extracts a first image feature of the first low-definition cropped image.
The method according to claim 2, wherein the electronic device fuses the first high-definition texture into the low-definition image to obtain a high-definition image, specifically comprising:

The electronic device fuses the first high-definition texture feature into the first low-definition cropped image to obtain a first high-definition cropped image;

The electronic device replaces the first low-definition cropped image with the first high-definition cropped image, and pastes it back to the first area in the low-definition image to obtain the high-definition image.
The method according to any one of claims 1-3, wherein after the electronic device fuses the first high-definition texture feature into the low-definition image to obtain a high-definition image, the method further comprises: :

The electronic device saves the high-definition image.
The method according to any one of claims 1-4, wherein before the electronic device receives the first input, the method further comprises:

The electronic device displays a shooting preview interface, where a shooting key and a preview image stream captured in real time by a camera are displayed on the shooting preview interface, and the first input is an input for the shooting key.
The method according to any one of claims 1-5, wherein a first control is further displayed on the shooting preview interface; before the electronic device receives the first input, the method further comprises:

the electronic device receives a second input for the first control;

In response to the second input, the electronic device initiates a super-resolution reconstruction mode.
The method according to claims 1-4, wherein before the electronic device receives the first input, the method further comprises:

The electronic device displays a picture browsing interface, the low-definition image and a second control are displayed on the picture browsing interface, and the first input is an input for the second control.
The method according to claim 5, wherein the preview image stream includes a first preview image; before the electronic device receives the first input, the method further comprises:

The electronic device detects that the first preview image includes a second specified target content, and the category of the second specified target content is a second category, and the second category is the same as or different from the first category;

The electronic device extracts the second image feature in the first preview image;

The electronic device sends the second image feature and the second category to the cloud server; wherein, the second image feature is used by the cloud server from a plurality of high-definition texture dictionary libraries of different shooting target categories stored in the cloud server The second high-definition texture dictionary library whose shooting target category is the second category is matched from the second image feature; the second image feature is used by the cloud server to match the second high-definition texture dictionary library with the second image feature from the second high-definition texture dictionary library. The second high-definition texture feature whose similarity is greater than the preset value;

receiving, by the electronic device, the second high-definition texture feature sent by the cloud server;

The electronic device stores the second high-definition texture feature in a cache queue.
The method according to claim 8, wherein the electronic device extracts the second image feature of the second specified target content in the first preview image, which specifically includes:

The electronic device identifies and crops out the second area in the low-definition image where the second specified target content is located, to obtain a second low-definition cropped image;

The electronic device extracts a second image feature of the second low-definition cropped image.
The method according to claim 8, wherein after the electronic device detects that the first preview image includes the second specified target content, the method further comprises:

The electronic device displays category information on the shooting preview interface, where the category information is used to indicate that the category of the specified target content in the preview image stream is the second category.
The method according to claim 8, wherein after the electronic device stores the second high-definition texture feature in a cache queue, the method further comprises:

The electronic device obtains the second preview image collected by the camera;

The electronic device detects that the second preview image includes a third specified target content, and the category of the third specified target content is a third category, and the third category is the same as the second category or different;

The electronic device identifies and crops out a third area in the low-definition image where the third specified target content is located, to obtain a third low-definition cropped image;

extracting, by the electronic device, a third image feature of the third low-definition cropped image;

The electronic device determines whether there is a high-definition texture feature whose similarity with the third image feature is greater than a preset value stored in the cache queue;

If a high-definition texture feature whose similarity with the third image feature is greater than a preset value is stored in the cache queue, the electronic device uses the cache queue with a similarity with the third image feature greater than a preset value. The high-definition texture feature of the value is fused into the third low-definition cropped image to obtain a third high-definition cropped image;

The electronic device replaces the third high-definition cropped image with the third low-definition cropped image, and pastes it back to the third area in the second preview image to obtain a high-definition preview image;

The electronic device displays the high-definition preview image on the shooting preview interface.
The method according to claim 11, wherein, if no high-definition texture feature whose similarity with the third image feature is greater than a preset value is stored in the cache queue, the method further comprises:

The electronic device sends the third image feature and the third category to the cloud server; wherein, the third image feature is used by the cloud server from a plurality of high-definition texture dictionary libraries of different shooting target categories stored in the cloud server The third high-definition texture dictionary library whose shooting target category is the third category is matched from the third image feature; the third image feature is used by the cloud server to match the third image feature from the third high-definition texture dictionary library. The third high-definition texture feature whose similarity is greater than the preset value;

The electronic device receives the third high-definition texture feature sent by the cloud server;

The electronic device stores the third high-definition texture feature in the cache queue, and fuses the third high-definition texture feature into the third low-definition cropped image to obtain the third high-definition cropped image.
The method according to any one of claims 8-12, wherein before the electronic device sends the first image feature and the first category to a cloud server, the method further comprises:

The electronic device determines whether a high-definition texture feature with a similarity with the first image feature greater than a preset value is stored in the cache queue;

The electronic device sends the first image feature and the first category to the cloud server, specifically including:

If no high-definition texture feature whose similarity with the first image feature is greater than a preset value is stored in the cache queue, the electronic device sends the first image feature and the first category to the cloud server.
The method according to claim 13, wherein if the cache queue stores high-definition texture features whose similarity with the first image feature is greater than a preset value, the method further comprises:

The electronic device fuses a high-definition texture feature in the cache queue with a similarity degree of the first image feature greater than a preset value into the low-definition image to obtain the high-definition image.
An electronic device, comprising: one or more processors, one or more memories, and a display screen; the one or more memories are coupled with one or more processors, and the one or more memories for storing computer program code, the computer program code comprising computer instructions which, when executed by the one or more processors, cause the electronic device to perform any one of the preceding claims 1-14. Image reconstruction method.
A readable storage medium for storing computer instructions, when the computer instructions are executed on an electronic device, the electronic device causes the electronic device to perform the image reconstruction method according to any one of the above claims 1-14.
An image reconstruction system, characterized in that it includes an electronic device and a cloud server, and the cloud server stores a plurality of high-definition texture dictionary libraries of different shooting target categories; wherein,

the electronic device for receiving a first input;

The electronic device is further configured to acquire a low-definition image captured by the camera in response to the first input;

The electronic device is further configured to detect that the low-definition image includes a first specified target content, and the category of the first specified target content is the first category;

The electronic device is further configured to extract the first image feature of the first specified target content in the low-definition image;

The electronic device is further configured to send the first image feature and the first category to the cloud server;

The cloud server is configured to match a first high-definition texture dictionary library whose shooting target category is the first category from the stored high-definition texture dictionary libraries of different shooting target categories;

The cloud server is further configured to match a first high-definition texture feature whose similarity with the first image feature is greater than a preset value from the first high-definition texture dictionary library;

The cloud server is further configured to send the first high-definition texture feature to the electronic device;

The electronic device is further configured to fuse the first high-definition texture feature into the low-definition image to obtain a high-definition image, wherein the resolution of the first specified target content in the high-definition image is greater than the resolution of the low-definition image. Clear the resolution of the first specified target content in the image.
The system according to claim 17, wherein the cloud server is further configured to acquire a plurality of high-quality images and a shooting target category of each high-quality image;

The cloud server is further configured to extract the high-definition texture features of the shooting target category in the high-quality image, and store them in the high-definition texture dictionary library corresponding to the shooting target category of the high-quality image.
The system according to claim 17 or 18, wherein the electronic device is further configured to send the set of recognition categories of the target recognition model on the electronic device to the cloud server;

The cloud server is further configured to send model update information to the electronic device when it is determined that the identification category set is different from the category set of the multiple high-definition texture dictionary libraries of different shooting target categories;

The electronic device is further configured to update the target recognition model on the electronic device based on the model update information; wherein, after the update, the recognition category set of the target recognition model on the electronic device is the same as that of the plurality of target recognition models. The category set of the HD texture dictionary library is the same for different shooting target categories.