WO2021115483A1 - 一种图像处理方法及相关装置 - Google Patents

一种图像处理方法及相关装置 Download PDF

Info

Publication number
WO2021115483A1
WO2021115483A1 PCT/CN2020/136178 CN2020136178W WO2021115483A1 WO 2021115483 A1 WO2021115483 A1 WO 2021115483A1 CN 2020136178 W CN2020136178 W CN 2020136178W WO 2021115483 A1 WO2021115483 A1 WO 2021115483A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
definition
cloud server
terminal
shooting
Prior art date
Application number
PCT/CN2020/136178
Other languages
English (en)
French (fr)
Inventor
陈刚
李江伟
罗巍
秦春林
王镇
呼静
张金雷
李政
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from CN202011446454.3A external-priority patent/CN112989092A/zh
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Priority to US17/784,342 priority Critical patent/US20230056332A1/en
Priority to EP20900091.8A priority patent/EP4060603A4/en
Publication of WO2021115483A1 publication Critical patent/WO2021115483A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/73—Deblurring; Sharpening
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/77—Retouching; Inpainting; Scratch removal
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
    • G06V10/267—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion by performing operations on regions, e.g. growing, shrinking or watersheds
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/74—Image or video pattern matching; Proximity measures in feature spaces
    • G06V10/75—Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries
    • G06V10/759—Region-based matching
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/74—Image or video pattern matching; Proximity measures in feature spaces
    • G06V10/761—Proximity, similarity or dissimilarity measures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/70—Labelling scene content, e.g. deriving syntactic or semantic representations
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10004—Still image; Photographic image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20021—Dividing image into blocks, subimages or windows

Definitions

  • This application relates to the field of computer vision, and in particular to an image processing method and related devices.
  • the terminal in order to improve the quality of photos taken by the terminal, usually after the camera obtains the captured image, the terminal generates image data through an image signal processing (ISP), and uses a preset algorithm for the image data. Optimize the processing to improve the image effect.
  • ISP image signal processing
  • the terminal uses a fixed algorithm to process the pictures, and the processed images are all in the same style, and due to the limitation of the terminal's capabilities, the processed image quality is not high.
  • different sceneries have different presentation styles. For example, historical sites such as the Forbidden City in Beijing show the style of antiquity, while urban attractions such as the Bund in Shanghai show the style of metropolis. In different environments of the same scene, the presentation style is also different.
  • the Forbidden City in summer and the Forbidden City in winter have different styles.
  • each style has different image parameters such as sharpness, color richness, contrast, white balance, detail, texture and so on.
  • the terminal has very limited means to improve the quality of captured pictures, and the imaging quality is not high.
  • This application provides an image processing method and related devices, which realize the combination of a cloud server and a terminal, make full use of high-quality image data and algorithms, and empower the terminal’s camera imaging system.
  • the terminal can break through the terminal’s own imaging quality limit, making it more common
  • the camera can also produce effects comparable to professional SLR cameras, thus, while reducing the cost of mobile phones, it greatly improves the quality of the photos taken by the terminal and the user's photo experience.
  • the present application provides an image processing method, the method includes: a cloud server obtains a low-definition image sent by a terminal and shooting information when the low-definition image is taken, the shooting information includes the low-definition image
  • the cloud server recognizes the semantic information of the low-definition image
  • the cloud server determines the number of image blocks N based on the zoom magnification of the high-definition image in the high-definition image library and the zoom magnification of the low-definition image,
  • the N is a positive integer
  • the cloud server divides the low-definition image into N image blocks based on the number of image blocks N, and marks each of the N image blocks based on the semantic information of the low-definition image
  • the type of image content in each image block the cloud server retrieves from the high-definition image library T high-definition retrieval images that match the shooting content of the first type of image block, and the T is a positive integer
  • the cloud Based on the T high-defini
  • the cloud server replaces the first type of image block in the low-definition image based on the first type of high-definition image block to obtain a high-definition output image, and sends the high-definition output image to the terminal .
  • the shooting information of the low-definition image further includes one or more of the following: geographic location information, weather information, season information, time information, shooting parameters, and shooting posture information.
  • the cloud server retrieves from the high-definition image library T high-definition retrieval images that match the shooting content of the first type of image block, which specifically includes: the cloud server is based on the The shooting information of the low-definition image, at least one first matching image is determined from the high-definition image library, wherein the degree of matching between the shooting information of the first matching image and the shooting information of the low-definition image is greater than that of the first matching image.
  • the resolution of the first matching image is greater than the resolution of the low-definition image; the cloud server separately obtains the first feature and the corresponding first feature and the corresponding image of each first matching image in the at least one first matching image.
  • the similarity between the feature and the first feature corresponding to the first image block, and determine from the at least one first matching image T pictures with the most similar shooting content to the shooting content of the first type of image block retrieve the image.
  • the cloud server is based on the difference between the first feature corresponding to each first matching image in the at least one first matching image and the first feature corresponding to the image block of the first type.
  • the degree of similarity of the at least one first matching image it is determined from the at least one first matching image that the shooting content is the most similar to the shooting content of the first type of image block.
  • the similarity between the first feature corresponding to each first matching image in the first matching image and the first feature corresponding to the image block of the first type is determined to be the same as the first feature of the first type.
  • the cloud server separately obtains the second feature corresponding to each second matching image in the at least one second matching image and the corresponding image block of the first type
  • the cloud server is based on the second feature corresponding to each second matching image in the at least one second matching image and the first type of image
  • the similarity between the second features corresponding to the blocks determines the T high-definition retrieval images that are most similar to the first type of image block in the second feature; wherein, the second feature is greater than the first feature More refined.
  • the cloud server uses the T high-definition retrieval images to repair the details in the first type of image block to obtain the first type of high-definition image block, which specifically includes: The cloud server obtains the image space coordinate transformation parameters of the first type image block and each of the T high-definition search images; the cloud server is based on the first type image block and the The image space coordinate transformation parameters of the matching feature point pairs in each high-definition retrieval image in T high-definition retrieval images, and the feature points in each high-definition retrieval image that match the first type of image block are adjusted to At the same position as in the image block of the first type, T high-definition registered images are obtained; the cloud server repairs the details of the image block of the first type based on the T high-definition registered images to obtain The first type of high-definition image block.
  • the cloud server repairs the details of the first type of image block based on the T high-definition registration images to obtain the first type of high-definition image block, which specifically includes: The cloud server determines from the T high-definition registered images based on the image space coordinate transformation parameters of the first type of image block and the matching feature point pairs in each of the T high-definition retrieval images Match the L high-definition reference images with the most similar image spatial position of the feature point pair, and L is a positive integer; the cloud server repairs the details of the first type of image block based on the L high-definition reference images to obtain all The first type of high-definition image block.
  • the cloud server replaces the first type of image block in the low-definition image based on the first type of high-definition image block to obtain a high-definition output image, which specifically includes: When only the image blocks of the first type are included in the image, the cloud server stitches the high-definition image blocks of the first type into the high-definition output image; when the low-definition image includes the image blocks of the first type and the second type In the case of image blocks of the type, the cloud server splices the high-definition image blocks of the first type and the high-definition image blocks of the second type into the high-definition output image.
  • the method further includes: the cloud server according to the low-definition image Determining the first image processing model corresponding to the environmental information; the cloud server processes the low-definition image through the first processing model.
  • the cloud server divides the low-definition image into N image blocks based on the number N of image blocks, which specifically includes: the cloud server processes the first image processing model based on the number N of image blocks.
  • the low-definition image is divided into the N image blocks.
  • the cloud server processing the low-definition image through the first processing model specifically includes: the cloud server processing the low-definition image through the first image processing model
  • the detail definition, and/or dynamic range, and/or contrast, and/or noise, and/or color are processed.
  • This application provides an image processing method that uses a high-definition image database on a cloud server to send a low-resolution image with a low zoom magnification (that is, a large field of view) shot on a terminal to the cloud server.
  • the cloud server cuts the low-definition image with low zoom magnification based on the zoom magnification of the high-definition image in the high-definition image library to obtain multiple image blocks.
  • the cloud server can search and match in the high-definition image library to obtain one or more high-definition reference images, and then through the image enhancement network based on the high-definition reference image to improve the image quality, obtain multiple high-definition image blocks .
  • the cloud server splices these multiple high-definition image blocks into an ultra-high-definition image with a larger resolution and higher definition, and returns it to the terminal.
  • the processing power of the cloud server can be used to enable the terminal to take pictures or videos with a wide field of view, as well as the rich details, textures and sharpness that can be taken at high zoom.
  • the present application provides an image processing method applied to an image processing system.
  • the image processing system includes a terminal and a cloud server.
  • the method includes: the terminal receives a first input from a user. In response to the first input, the terminal acquires the first image and the first environment information when the first image was taken. The terminal sends the first environment information to the cloud server.
  • the cloud server determines a first image processing model corresponding to the environmental information according to the first environmental information; the first image processing model is used to improve the quality of the image.
  • the cloud server sends the first image processing model to the terminal.
  • the terminal processes the first image through the first image processing model to obtain a second image.
  • This application provides an image processing method that can combine the processing capabilities of the terminal and the cloud server, make full use of the high-quality image data and image processing algorithms on the cloud server, and empower the terminal camera imaging system.
  • the cloud server can use the image uploaded by the terminal and the environmental information of the image (such as geographic location information, weather information, time information, shooting parameters, shooting posture information, etc.) to match the image processing algorithm suitable for the image and the shooting content of the image Same, high-quality image data with similar shooting environment.
  • the image processing algorithm and the high-quality image data the quality and effect of the image uploaded by the terminal can be enhanced. In this way, pictures taken in different environments can show different styles and effects.
  • This also enables the terminal to break through the terminal's own imaging quality limit, so that the ordinary camera on the terminal can also take pictures of professional SLR equipment, which greatly improves the terminal's camera quality and user's camera experience while reducing the cost of mobile phones.
  • the method before the terminal receives the first input of the user, the method further includes: the terminal displays a shooting interface, the shooting interface is used to display a preview image; the shooting interface includes shooting controls.
  • the first input is a user's operation on the shooting control, and the first image is the preview image in the shooting interface.
  • the method before the terminal receives the user's first input, the method further includes: the terminal displays a cloud enhanced interface in a gallery application; the cloud enhanced interface display includes a thumbnail corresponding to the first image Sketch and upload control; wherein, the first input is the user's operation on the upload control.
  • the obtaining of the first image and the first environment information of the first image by the terminal specifically includes: obtaining the first image from the local storage by the terminal.
  • the terminal obtains the first environment information from the exchangeable image file format Exif data of the first image. In this way, after the user selects the picture that he wants to optimize in the gallery, he can optimize and enhance it with the help of the processing power of the cloud server to improve the quality of the pictures saved on the terminal.
  • the method further includes: the cloud server sending the first image to the cloud server.
  • the cloud server determines the first high-definition image according to the first environment information. Wherein, the similarity between the second environment information of the first high-definition picture and the first environment information is within a first preset range, and the shooting content in the first high-definition image is between the shooting content in the first image The similarity of is within the second preset range.
  • the cloud server performs image registration on the first image and the first high-definition image to obtain a second high-definition image.
  • the cloud server sends the second high-definition image to the terminal.
  • the terminal performs image fusion on the second high-definition image and the second image to obtain a third image, the definition of the third image is higher than that of the second image.
  • a specific image processing model can be used to optimize the first image for a specific shooting environment, and the quality of the image can be improved.
  • the cloud server determines the first high-definition image according to the first environment information, which specifically includes: the cloud server retrieves the first high-definition image from a high-definition image library according to the first environment information. image.
  • the high-definition image library stores multiple high-definition images and second environment information corresponding to each of the multiple high-definition images.
  • the cloud server determines the first high-definition image according to the first environment information, which specifically includes: the cloud server renders the first high-definition image from the three-dimensional world model through a rendering engine according to the first environment information.
  • the first high-definition image is determined according to the first environment information, which specifically includes: the cloud server renders the first high-definition image from the three-dimensional world model through a rendering engine according to the first environment information.
  • the first high-definition image is determined according to the first environment information, which specifically includes: the cloud server renders the first high-definition image from the three-dimensional world model through a rendering engine according to the first environment information.
  • the first high-definition image is a possible implementation manner.
  • the terminal performs image fusion on the second high-definition image and the second image to obtain the third image, which specifically includes: the terminal extracts the feature information of the second high-definition image, and merges it with the first high-definition image.
  • the third image is obtained; wherein, the characteristic information includes high-resolution details and high-frequency information.
  • the terminal processes the quality of the first image through the first image processing model to obtain the second image, which specifically includes: the terminal processes the first image through the first image processing model.
  • the detail definition, and/or dynamic range, and/or contrast, and/or noise, and/or color are processed to obtain the second image.
  • the method further includes: the terminal displays the second image.
  • the method further includes: the terminal displays the third image.
  • the first environment information includes one or more of the following: geographic location information, weather information, season information, time information, shooting posture information, shooting parameters, and so on.
  • the present application provides an image processing method, which is applied to an image processing system, and the image processing system includes a terminal and a cloud server.
  • the terminal receives the user's first input.
  • the terminal acquires the first image and the first environment information of the first image.
  • the terminal sends the first image and the first environment information to the cloud server.
  • the cloud server determines the first high-definition image according to the first environment information and the first image.
  • the similarity between the environmental information of the first high-definition image and the first environmental information is within a first preset range
  • the difference between the shooting content in the first high-definition image and the shooting content in the first image The similarity is within the second preset range.
  • the cloud server performs image registration on the first high-definition image according to the first image to obtain a second high-definition image.
  • the cloud server sends the second high-definition image to the terminal.
  • the terminal performs image fusion on the second high-definition image and the first image to obtain a third image, the definition of the third image is higher than that of the first image.
  • This application provides an image processing method that can combine the processing capabilities of the terminal and the cloud server, make full use of the high-quality image data and image processing algorithms on the cloud server, and empower the terminal camera imaging system.
  • the cloud server can use the image uploaded by the terminal and the environmental information of the image (such as geographic location information, weather information, time information, shooting parameters, shooting posture information, etc.) to match the image processing algorithm suitable for the image and the shooting content of the image Same, high-quality image data with similar shooting environment.
  • the image processing algorithm and the high-quality image data the quality and effect of the image uploaded by the terminal can be enhanced. In this way, pictures taken in different environments can show different styles and effects.
  • This also enables the terminal to break through the terminal's own imaging quality limit, so that the ordinary camera on the terminal can also take pictures of professional SLR equipment, thereby greatly improving the terminal's camera quality and user's camera experience while reducing the cost of mobile phones.
  • the method before the terminal receives the first input of the user, the method further includes: the terminal displays a shooting interface, the shooting interface is used to display a preview image; the shooting interface includes shooting controls.
  • the first input is a user's operation on the shooting control, and the first image is the preview image in the shooting interface.
  • the method before the terminal receives the user's first input, the method further includes: the terminal displays a cloud enhanced interface in a gallery application; the cloud enhanced interface display includes a thumbnail corresponding to the first image Sketch and upload control; wherein, the first input is the user's operation on the upload control.
  • the obtaining of the first image and the first environment information of the first image by the terminal specifically includes: obtaining the first image from the local storage by the terminal.
  • the terminal obtains the first environment information from the exchangeable image file format Exif data of the first image. In this way, after the user selects the picture that he wants to optimize in the gallery, he can optimize and enhance it with the help of the processing power of the cloud server to improve the quality of the pictures saved on the terminal.
  • the method further includes: before the terminal performs image fusion on the second high-definition image and the first image to obtain the third image, the method further includes: the cloud server according to the first image
  • the environmental information determines the first image processing model corresponding to the first environmental information.
  • the first image processing model is used to improve the quality of the image.
  • the cloud server sends the first image processing model to the terminal.
  • the terminal processes the first image through the first image processing model.
  • the terminal performs image fusion on the second high-definition image and the first image to obtain a third image, which specifically includes: the terminal performs image processing on the second high-definition image and the first image processed by the first image processing model Fusion to obtain the third image.
  • the first image can be optimized through the high-definition pictures in the same shooting environment on the cloud server, and the details of the image can be improved.
  • the cloud server determines the first high-definition image according to the first environment information, which specifically includes: the cloud server retrieves the first high-definition image from a high-definition image library according to the first environment information. image.
  • the high-definition image library stores multiple high-definition images and second environment information corresponding to each of the multiple high-definition images.
  • the cloud server determines the first high-definition image according to the first environment information, which specifically includes: the cloud server renders the first high-definition image from the three-dimensional world model through a rendering engine according to the first environment information.
  • the first high-definition image is determined according to the first environment information, which specifically includes: the cloud server renders the first high-definition image from the three-dimensional world model through a rendering engine according to the first environment information.
  • the first high-definition image is determined according to the first environment information, which specifically includes: the cloud server renders the first high-definition image from the three-dimensional world model through a rendering engine according to the first environment information.
  • the first high-definition image is a possible implementation manner.
  • the terminal performs image fusion on the second high-definition image and the second image to obtain the third image, which specifically includes: the terminal extracts the feature information of the second high-definition image, and merges it with the first high-definition image.
  • the third image is obtained.
  • the feature information includes high-definition details.
  • the terminal processes the first image through the first image processing model, which specifically includes: the terminal uses the first image processing model to determine the details of the first image, and/or Dynamic range, and/or contrast, and/or noise, and/or color are processed.
  • the method further includes: the terminal displays the third image.
  • the first environment information includes one or more of the following: geographic location information, weather information, season information, time information, shooting posture information, shooting parameters, and so on.
  • the present application provides an image processing method, which is applied to an image processing system, and the image processing system includes a terminal and a cloud server.
  • the method includes: the terminal receives a user's first input. In response to the first input, the terminal acquires the first image and the first environment information of the first image. The terminal sends the first image and the first environment information to the cloud server.
  • the cloud server determines a first image processing model corresponding to the first environment information according to the first environment information.
  • the first image processing model is used to improve the quality of the image.
  • the cloud server processes the quality of the first image through the first image processing model to obtain a second image.
  • the cloud server sends the second image to the terminal.
  • the method further includes: the cloud server determines the first high-definition image according to the first environment information; wherein the second environment information of the first high-definition image and the first environment information are in Within the second preset range, and the similarity between the shooting content in the first high-definition image and the shooting content in the first image is within the second preset range.
  • the cloud server performs image registration on the first high-definition image according to the first image to obtain a second high-definition image.
  • the cloud server performs image fusion on the second high-definition image and the second image to obtain a third image.
  • the cloud server sends the third image to the terminal.
  • the present application provides an image processing method, which is applied to an image processing system, and the image processing system includes a terminal and a cloud server.
  • the method includes: the terminal receives a user's first input. In response to the first input, the terminal acquires the first image and the first environment information of the first image. The terminal sends the first image and the first environment information to the cloud server.
  • the cloud server determines the first high-definition image according to the first environment information.
  • the similarity between the second environment information of the first high-definition image and the first environment information is within a first preset range, and there is a difference between the shooting content in the first high-definition image and the shooting content in the first image
  • the similarity of is within the second preset range.
  • the cloud server performs image registration on the first high-definition image according to the first image to obtain a second high-definition image.
  • the cloud server performs image fusion on the second high-definition image and the first image to obtain a third image, the definition of the third image is higher than that of the first image.
  • the cloud server sends the third image to the terminal.
  • the method before the server performs image fusion on the second high-definition image and the first image to obtain the third image, the method further includes: the cloud server determines and The first image processing model corresponding to the first environment information.
  • the first image processing model is used to improve the quality of the image.
  • the cloud server processes the first image through the first image processing model.
  • an embodiment of the present application provides an image processing method, where the method may be executed by a terminal or a cloud server, and the image processing method may include: acquiring image A shot by the terminal and shooting information of the image A.
  • the terminal may send the captured image A and the shooting information of the image A to the cloud server, and the cloud server receives the image A and the shooting information of the image A.
  • At least one image B is determined according to the shooting information of the image A, wherein the degree of matching between the shooting information of each image B in the at least one image B and the shooting information of the image A is greater than a first threshold.
  • the resolution of each image B in the at least one image B is greater than the resolution of image A.
  • the at least one image B is a high-definition image
  • the image A is a low-definition image.
  • the terminal may send the shooting information of image A to the cloud server, the cloud server selects the at least one image B from the high-definition image library, and sends the at least one image B to the terminal, and the terminal receives the at least one image B from the at least one image B.
  • the reference image 1 whose shooting content is most similar to the shooting content of image A is determined in.
  • the terminal may send the shooting information of image A and image A to the cloud server, and the cloud server selects the at least one image B from the high-definition image library, and determines the shooting content and the shooting content of the image A from the at least one image B The most similar reference image 1.
  • At least one image B is determined based on the shooting information of image A, and then the reference image 1 whose shooting content is most similar to the shooting content of image A is determined from the at least one image B, that is, the shooting of image A can be passed first Information narrows the scope of retrieval, thereby improving the efficiency of image retrieval.
  • the shooting information may include one or more of the following information: shooting magnification, shooting geographic location information, or shooting weather condition information, and so on.
  • shooting magnification may be a magnification.
  • the shooting geographic location information may refer to the geographic location information where the photographing device is located when the image A or the image B is captured, and the geographic location information may be defined by global positioning system (GPS) information.
  • GPS global positioning system
  • the shooting weather condition information may refer to the weather conditions when the image A or the image B is shot, for example, sunny, cloudy, rainy, and so on.
  • the matching degree between the shooting geographic location information of each image B in the at least one image B and the shooting geographic location information of image A is greater than a first threshold, and the degree of matching is determined by the geographic location.
  • the matching of information can ensure the consistency of the shooting geographic locations of the image A and the at least one image B.
  • a reference image 1 whose shooting content is most similar to that of image A is determined.
  • the reference image 1 is not only highly similar to image A, but also the shooting angle (ie, shooting geographic location information) is similar to image A. Also highly consistent.
  • the matching degree between the shooting magnification of each image B in the at least one image B and the shooting magnification of the image A is greater than a first threshold, which can ensure that the image A and the at least one image The magnification of B is not much different.
  • the shooting information includes shooting weather condition information
  • the degree of matching between the shooting weather condition information of each image B in the at least one image B and the shooting weather condition information of image A is greater than a first threshold, for example, both If it is taken on a sunny day, or taken on a rainy day, the consistency of the transparency of the image can be guaranteed.
  • the image A is subjected to detail restoration processing to obtain the image C.
  • the resolution of the obtained image C is greater than the resolution of the image A, that is, the resolution of the image C is greater than the resolution of the image A.
  • the reference image 1 and image A can be input to the deep learning neural network, the high frequency information of the reference image 1 can be extracted through the deep learning neural network, and the details of the image A can be restored through the high frequency information, so as to obtain the sharpness Higher image C.
  • determining, from at least one image B, the reference image 1 whose shooting content is most similar to the shooting content of image A may include: separately acquiring each image B corresponding to the at least one image B The first feature of and the first feature corresponding to the image A, where each image B corresponds to a first feature.
  • the first feature corresponding to each image B in the at least one image B may be extracted through a retrieval network, and the first feature corresponding to the image A may also be extracted through the retrieval network.
  • the first feature includes but is not limited to CNN features, and correspondingly, the retrieval network may be a CNN network.
  • M is an integer greater than or equal to 1.
  • M may be 15, that is, 15 images B that are most similar to the shooting content of image A are selected from the at least one image B.
  • the similarity between the first feature corresponding to each image B and the first feature corresponding to image A may be sorted, so as to obtain M images B with the greatest similarity between image A.
  • the reference image 1 whose shooting content is most similar to the shooting content of the image A is further determined from the M images B.
  • the reference image 1 can be obtained by matching with more detailed features.
  • the retrieval range is gradually reduced to obtain the reference image 1 that is most similar to the shot content of the image A, which not only improves the retrieval efficiency, but also ensures the consistency of the retrieved reference image 1 and the shot content of the image A.
  • determining the reference image 1 whose shooting content is most similar to the shooting content of the image A from the M images B includes: separately acquiring the second corresponding to each image B in the M images B The feature corresponds to the second feature of the image A, wherein each image B corresponds to a second feature.
  • the second feature corresponding to each of the M images B may be extracted through a reordering network, and the second feature corresponding to the image A may also be extracted through the reordering network.
  • the reordering network can be a feature extraction network that is more complex than the retrieval network.
  • the second feature extracted by the reordering network can be an image feature that is more refined than the first feature extracted by the retrieval network. For example, the second feature is more complex than the first feature. Reflect the detailed characteristics of the image.
  • the similarity between the second feature corresponding to each image B and the second feature corresponding to image A may be sorted, so that the image B with the greatest similarity is used as the reference image 1.
  • the sorting result of sorting the similarity between the first feature corresponding to each image B and the first feature corresponding to image A may be the same as that of comparing the second feature corresponding to each image B with image A
  • the ranking results for ranking the similarity between the corresponding second features are different.
  • the search network is simpler than the ranking network, the search efficiency is also higher.
  • the first feature extracted by the search network is first matched, and from at least one image B, the image A roughly similar M images B are matched by the second feature extracted by the reordering network. From the M images B, the most similar reference image 1 is obtained, which can improve the accuracy of the retrieval under the premise of ensuring the retrieval efficiency. degree.
  • the method before performing detail restoration processing on image A according to reference image 1, the method further includes: determining the ratio of the shooting content of the reference image 1 to the shooting content of image A, where the ratio may be It reflects whether the shooting content contained in the retrieved reference image 1 covers the shooting content contained in the full image A.
  • the shooting content matching the shooting content of the reference image 1 is removed from the shooting content of the image A, and the removed image A is obtained.
  • the second threshold can be set according to actual needs. For example, the second threshold can be set to 80%. If the ratio of the shooting content of the reference image 1 to the shooting content of the image A is less than 80%, the reference image 1 is described. The included shooting content does not cover the shooting content included in the full image A, and the shooting content included in the image A can be further searched for shooting content other than the shooting content included in the reference image 1, that is, the removed image A.
  • the reference image 2 whose shooting content is most similar to the shooting content of the removed image A is determined from the at least one image B.
  • the method for determining the reference image 2 whose shooting content is most similar to the shooting content of the removed image A from at least one image B can refer to the method for determining the reference image 1 in the foregoing embodiment, which will not be repeated here.
  • the ratio of the shooting content of the reference image 1 and the shooting content of the reference image 2 to the shooting content of the image A can be further determined, if the ratio is still less than the second threshold , It is possible to further remove the shooting content of the reference image 1 and the shooting content matching the shooting content of the reference image 2 from the shooting content of the image A to obtain the removed image A, and then further retrieve it from at least one image B The reference image 3 that is most similar to the shooting content of the removed image A.
  • an embodiment of the present application provides an image processing device, including various modules or units for executing the method of the fifth aspect.
  • an embodiment of the present application provides an image processing system, the system includes a terminal and a cloud server, wherein: the first terminal sends the captured image A and the shooting information of the image A to the cloud server, and the cloud server receives According to the shooting information of the image A and the image A, at least one image B is determined according to the shooting information of the image A, wherein the degree of matching between the shooting information of the image B and the shooting information of the image A is greater than a first threshold , The resolution of the image B is greater than the resolution of the image A.
  • the cloud server determines from the at least one image B the reference image 1 whose shooting content is most similar to the shooting content of the image A.
  • the cloud server performs detail restoration processing on the image A according to the reference image 1 to obtain image C.
  • the resolution of the image C is greater than the resolution of the image A, and the image C is sent to all ⁇ terminal.
  • the terminal receives the image C and displays the image C.
  • an embodiment of the present application provides an image processing system, the system includes a terminal and a cloud server, wherein: the first terminal sends the captured image A and the shooting information of the image A to the cloud server, and the cloud server receives According to the shooting information of the image A and the image A, at least one image B is determined according to the shooting information of the image A, wherein the degree of matching between the shooting information of the image B and the shooting information of the image A is greater than a first threshold , The resolution of the image B is greater than the resolution of the image A.
  • the cloud server sends the at least one image B to the terminal device.
  • the terminal device determines from the at least one image B the reference image 1 whose shooting content is most similar to the shooting content of the image A.
  • the terminal device performs detail restoration processing on the image A according to the reference image 1 to obtain the image C, and output the image C.
  • the resolution of the image C is greater than the resolution of the image A.
  • this application provides a terminal, including a camera, a transceiver, one or more processors, and one or more memories.
  • the one or more processors are coupled with the camera, the one or more memories, and the transceiver, and the one or more memories are used to store computer program codes.
  • the computer program codes include computer instructions.
  • this application provides a cloud server, a transceiver, one or more processors, and one or more memories.
  • the one or more memories are coupled with one or more processors, and the one or more memories are used to store computer program codes.
  • the computer program codes include computer instructions.
  • the cloud server executes the image processing method in any possible implementation manner of any one of the foregoing aspects.
  • an embodiment of the present application provides a computer storage medium, including computer instructions, which, when the computer instructions run on a terminal, cause the terminal to execute the image processing method in any one of the possible implementations of any of the above aspects.
  • an embodiment of the present application provides a computer storage medium, including computer instructions, which when the computer instructions run on a cloud server, cause the cloud server to perform image processing in any of the possible implementations of any of the above aspects method.
  • the embodiments of the present application provide a computer program product, which when the computer program product runs on a terminal, causes the terminal to execute the image processing method in any one of the possible implementation manners of any of the foregoing aspects.
  • the embodiments of the present application provide a computer program product that, when the computer program product runs on a cloud server, causes the cloud server to execute the image processing method in any one of the possible implementation manners of any of the foregoing aspects.
  • FIG. 1 is a schematic diagram of the architecture of a communication system provided by an embodiment of this application.
  • FIG. 2A is a schematic structural diagram of a terminal provided by an embodiment of this application.
  • 2B is a schematic diagram of a software architecture of a terminal provided by an embodiment of the application.
  • 3A-3H are schematic diagrams of a set of user interfaces provided by an embodiment of the application.
  • FIGS. 4A-4F are schematic diagrams of another set of user interfaces provided by an embodiment of this application.
  • FIG. 4G is a schematic diagram of comparison of a group of picture processing effects provided by an embodiment of the application.
  • 5A-5G are schematic diagrams of another set of user interfaces provided by the application embodiment.
  • 6A-6P are schematic diagrams of another set of user interfaces provided by the application embodiment.
  • FIGS. 7A-7D are schematic diagrams of another set of user interfaces provided by the application embodiment.
  • FIG. 8 is a schematic flowchart of an image processing method provided by an embodiment of this application.
  • FIG. 9 is a schematic structural diagram of an image processing system provided by an embodiment of the application.
  • FIG. 10 is a schematic structural diagram of yet another image processing system provided by an embodiment of this application.
  • FIG. 11 is a schematic structural diagram of another image processing system provided by an embodiment of this application.
  • FIG. 12 is a schematic structural diagram of another image processing system provided by an embodiment of this application.
  • FIG. 13 is a schematic diagram of the architecture of a data construction system provided by an embodiment of the application.
  • FIG. 14 is a schematic structural diagram of a cloud server provided by an embodiment of this application.
  • 16 is a schematic flowchart of another image processing method provided by an embodiment of the application.
  • FIG. 17 is a schematic flowchart of another image processing method provided by an embodiment of this application.
  • FIG. 18 is a framework diagram of an image processing solution provided by an embodiment of this application.
  • FIG. 19 is a schematic structural diagram of an image processing device provided by an embodiment of this application.
  • 21A is a schematic diagram of a low-definition image provided by an embodiment of the application.
  • 21B is a semantic segmentation mask diagram of a low-definition image provided by an embodiment of the application.
  • FIG. 21C is a schematic diagram of block division for a low-definition image provided by an embodiment of the application.
  • FIG. 21D is a schematic diagram of an image block provided by an embodiment of the application.
  • FIG. 22 is a schematic diagram of an image registration process provided by an embodiment of this application.
  • FIG. 23A is a schematic diagram of an image detail restoration process provided by an embodiment of the application.
  • FIG. 23B is a schematic diagram of a high-definition reference image provided by an embodiment of this application.
  • FIG. 23C is a schematic diagram of a high-definition image block provided by an embodiment of the application.
  • FIG. 24 is a schematic flowchart of another image processing method provided by an embodiment of the application.
  • 25A is a block diagram of a panoramic high-definition image provided by an embodiment of the application.
  • FIG. 25B is an example diagram of a segmented high-definition image in a panoramic high-definition image provided by an embodiment of the application.
  • FIG. 25C is a schematic block diagram of another panoramic high-definition image provided by an embodiment of the application.
  • FIG. 25D is an example diagram of a segmented high-definition image in a panoramic high-definition image provided by an embodiment of the application.
  • first and second are only used for descriptive purposes, and cannot be understood as implying or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, the features defined with “first” and “second” may explicitly or implicitly include one or more of these features. In the description of the embodiments of the present application, unless otherwise specified, “multiple” The meaning is two or more.
  • the following describes a communication system 10 related to an embodiment of the present application.
  • FIG. 1 shows a schematic structural diagram of a communication system 10 related to an application embodiment.
  • the communication system 10 may include a terminal 100 and a cloud server 200.
  • the terminal 100 can be a mobile phone, a tablet computer, a desktop computer, a laptop computer, a handheld computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a cellular phone, a personal digital assistant ( Personal digital assistant (PDA), augmented reality (AR) ⁇ virtual reality (VR) equipment, etc.
  • PDA personal digital assistant
  • AR augmented reality
  • VR virtual reality
  • the embodiment of the present application does not impose special restrictions on the specific type of the terminal 100.
  • the terminal 100 can connect to the cloud server 200 through a 2G network, a 3G network, a 4G network, a 5G network, a wireless local area network (WLAN), etc.
  • the terminal 100 may send image data and environmental information of the image data (for example, geographic location information, shooting posture information of a shooting device, shooting time information, weather information, etc.) to the cloud server 200.
  • the cloud server 200 may send an image processing network model, high-definition picture data, and so on to the terminal 100.
  • the cloud server 200 can establish connections with multiple terminals 100, and can independently process processing tasks requested by multiple terminals 100. Among them, the cloud server 200 can distinguish the terminals through the account (for example, a Huawei account) that the user logs in on the terminal.
  • the account for example, a Huawei account
  • FIG. 2A shows a schematic diagram of the structure of the terminal 100.
  • the terminal 100 shown in FIG. 2A is only an example, and the terminal 100 may have more or fewer components than those shown in FIG. 2A, may combine two or more components, or may have Different component configurations.
  • the various components shown in the figure may be implemented in hardware, software, or a combination of hardware and software including one or more signal processing and/or application specific integrated circuits.
  • the terminal 100 may include: a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, antenna 1, antenna 2 , Mobile communication module 150, wireless communication module 160, audio module 170, speaker 170A, receiver 170B, microphone 170C, earphone jack 170D, sensor module 180, buttons 190, motor 191, indicator 192, camera 193, display screen 194, and Subscriber identification module (subscriber identification module, SIM) card interface 195, etc.
  • SIM Subscriber identification module
  • the sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, and ambient light Sensor 180L, bone conduction sensor 180M, etc.
  • the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the terminal 100.
  • the terminal 100 may include more or fewer components than shown, or combine certain components, or split certain components, or arrange different components.
  • the illustrated components can be implemented in hardware, software, or a combination of software and hardware.
  • the processor 110 may include one or more processing units.
  • the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), and an image signal processor. (image signal processor, ISP), controller, memory, video codec, digital signal processor (digital signal processor, DSP), baseband processor, and/or neural-network processing unit (NPU) Wait.
  • AP application processor
  • modem processor modem processor
  • GPU graphics processing unit
  • image signal processor image signal processor
  • ISP image signal processor
  • controller memory
  • video codec digital signal processor
  • DSP digital signal processor
  • NPU neural-network processing unit
  • the different processing units may be independent devices or integrated in one or more processors.
  • the controller may be the nerve center and command center of the terminal 100.
  • the controller can generate operation control signals according to the instruction operation code and timing signals to complete the control of fetching instructions and executing instructions.
  • a memory may also be provided in the processor 110 to store instructions and data.
  • the memory in the processor 110 is a cache memory.
  • the memory can store instructions or data that have just been used or recycled by the processor 110. If the processor 110 needs to use the instruction or data again, it can be directly called from the memory. Repeated accesses are avoided, the waiting time of the processor 110 is reduced, and the efficiency of the system is improved.
  • the processor 110 may include one or more interfaces.
  • the interface may include an integrated circuit (inter-integrated circuit, I2C) interface, an integrated circuit built-in audio (inter-integrated circuit sound, I2S) interface, a pulse code modulation (pulse code modulation, PCM) interface, a universal asynchronous transceiver receiver/transmitter, UART) interface, mobile industry processor interface (MIPI), general-purpose input/output (GPIO) interface, subscriber identity module (SIM) interface, and / Or Universal Serial Bus (USB) interface, etc.
  • I2C integrated circuit
  • I2S integrated circuit built-in audio
  • PCM pulse code modulation
  • PCM pulse code modulation
  • UART universal asynchronous transceiver receiver/transmitter
  • MIPI mobile industry processor interface
  • GPIO general-purpose input/output
  • SIM subscriber identity module
  • USB Universal Serial Bus
  • the I2C interface is a bidirectional synchronous serial bus, which includes a serial data line (SDA) and a serial clock line (SCL).
  • the processor 110 may include multiple sets of I2C buses.
  • the processor 110 may be coupled to the touch sensor 180K, charger, flash, camera 193, etc., respectively through different I2C bus interfaces.
  • the processor 110 may couple the touch sensor 180K through an I2C interface, so that the processor 110 and the touch sensor 180K communicate through an I2C bus interface to implement the touch function of the terminal 100.
  • the I2S interface can be used for audio communication.
  • the processor 110 may include multiple sets of I2S buses.
  • the processor 110 may be coupled with the audio module 170 through an I2S bus to implement communication between the processor 110 and the audio module 170.
  • the audio module 170 may transmit audio signals to the wireless communication module 160 through an I2S interface, so as to realize the function of answering calls through a Bluetooth headset.
  • the PCM interface can also be used for audio communication to sample, quantize and encode analog signals.
  • the audio module 170 and the wireless communication module 160 may be coupled through a PCM bus interface.
  • the audio module 170 may also transmit audio signals to the wireless communication module 160 through the PCM interface, so as to realize the function of answering calls through the Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.
  • the UART interface is a universal serial data bus used for asynchronous communication.
  • the bus can be a two-way communication bus. It converts the data to be transmitted between serial communication and parallel communication.
  • the UART interface is generally used to connect the processor 110 and the wireless communication module 160.
  • the processor 110 communicates with the Bluetooth module in the wireless communication module 160 through the UART interface to realize the Bluetooth function.
  • the audio module 170 may transmit audio signals to the wireless communication module 160 through a UART interface, so as to realize the function of playing music through a Bluetooth headset.
  • the MIPI interface can be used to connect the processor 110 with the display screen 194, the camera 193 and other peripheral devices.
  • the MIPI interface includes a camera serial interface (camera serial interface, CSI), a display serial interface (display serial interface, DSI), and so on.
  • the processor 110 and the camera 193 communicate through a CSI interface to implement the shooting function of the terminal 100.
  • the processor 110 and the display screen 194 communicate through a DSI interface to realize the display function of the terminal 100.
  • the GPIO interface can be configured through software.
  • the GPIO interface can be configured as a control signal or as a data signal.
  • the GPIO interface can be used to connect the processor 110 with the camera 193, the display screen 194, the wireless communication module 160, the audio module 170, the sensor module 180, and so on.
  • the GPIO interface can also be configured as an I2C interface, I2S interface, UART interface, MIPI interface, etc.
  • the USB interface 130 is an interface that complies with the USB standard specification, and specifically may be a Mini USB interface, a Micro USB interface, a USB Type C interface, and so on.
  • the USB interface 130 can be used to connect a charger to charge the terminal 100, and can also be used to transfer data between the terminal 100 and peripheral devices. It can also be used to connect earphones and play audio through earphones. This interface can also be used to connect to other terminals, such as AR devices.
  • the interface connection relationship between the modules illustrated in the embodiment of the present invention is merely a schematic description, and does not constitute a structural limitation of the terminal 100.
  • the terminal 100 may also adopt different interface connection modes in the foregoing embodiments, or a combination of multiple interface connection modes.
  • the charging management module 140 is used to receive charging input from the charger.
  • the charger can be a wireless charger or a wired charger.
  • the charging management module 140 may receive the charging input of the wired charger through the USB interface 130.
  • the charging management module 140 may receive the wireless charging input through the wireless charging coil of the terminal 100. While the charging management module 140 charges the battery 142, it can also supply power to the terminal through the power management module 141.
  • the power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110.
  • the power management module 141 receives input from the battery 142 and/or the charging management module 140, and supplies power to the processor 110, the internal memory 121, the external memory, the display screen 194, the camera 193, and the wireless communication module 160.
  • the power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle times, and battery health status (leakage, impedance).
  • the power management module 141 may also be provided in the processor 110.
  • the power management module 141 and the charging management module 140 may also be provided in the same device.
  • the wireless communication function of the terminal 100 can be implemented by the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor, and the baseband processor.
  • the antenna 1 and the antenna 2 are used to transmit and receive electromagnetic wave signals.
  • Each antenna in the terminal 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization.
  • Antenna 1 can be multiplexed as a diversity antenna of a wireless local area network.
  • the antenna can be used in combination with a tuning switch.
  • the mobile communication module 150 may provide a wireless communication solution including 2G/3G/4G/5G and the like applied to the terminal 100.
  • the mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), and the like.
  • the mobile communication module 150 can receive electromagnetic waves by the antenna 1, and perform processing such as filtering, amplifying and transmitting the received electromagnetic waves to the modem processor for demodulation.
  • the mobile communication module 150 can also amplify the signal modulated by the modem processor, and convert it into electromagnetic waves for radiation via the antenna 1.
  • at least part of the functional modules of the mobile communication module 150 may be provided in the processor 110.
  • at least part of the functional modules of the mobile communication module 150 and at least part of the modules of the processor 110 may be provided in the same device.
  • the modem processor may include a modulator and a demodulator.
  • the modulator is used to modulate the low frequency baseband signal to be sent into a medium and high frequency signal.
  • the demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal.
  • the demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing.
  • the application processor outputs a sound signal through an audio device (not limited to the speaker 170A, the receiver 170B, etc.), or displays an image or video through the display screen 194.
  • the modem processor may be an independent device.
  • the modem processor may be independent of the processor 110 and be provided in the same device as the mobile communication module 150 or other functional modules.
  • the wireless communication module 160 can provide applications on the terminal 100 including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), and global navigation satellite systems. (global navigation satellite system, GNSS), frequency modulation (frequency modulation, FM), near field communication technology (near field communication, NFC), infrared technology (infrared, IR) and other wireless communication solutions.
  • WLAN wireless local area networks
  • BT Bluetooth
  • GNSS global navigation satellite system
  • frequency modulation frequency modulation, FM
  • NFC near field communication technology
  • infrared technology infrared, IR
  • the wireless communication module 160 may be one or more devices integrating at least one communication processing module.
  • the wireless communication module 160 receives electromagnetic waves via the antenna 2, frequency modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110.
  • the wireless communication module 160 may also receive a signal to be sent from the processor 110, perform frequency modulation, amplify, and convert it into electromagnetic waves to radiate through the antenna 2.
  • the antenna 1 of the terminal 100 is coupled with the mobile communication module 150, and the antenna 2 is coupled with the wireless communication module 160, so that the terminal 100 can communicate with the network and other devices through wireless communication technology.
  • the wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), broadband Code division multiple access (wideband code division multiple access, WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC , FM, and/or IR technology, etc.
  • the GNSS may include global positioning system (GPS), global navigation satellite system (GLONASS), Beidou navigation satellite system (BDS), quasi-zenith satellite system (quasi -zenith satellite system, QZSS) and/or satellite-based augmentation systems (SBAS).
  • GPS global positioning system
  • GLONASS global navigation satellite system
  • BDS Beidou navigation satellite system
  • QZSS quasi-zenith satellite system
  • SBAS satellite-based augmentation systems
  • the terminal 100 implements a display function through a GPU, a display screen 194, and an application processor.
  • the GPU is an image processing microprocessor, which is connected to the display screen 194 and the application processor.
  • the GPU is used to perform mathematical and geometric calculations and is used for graphics rendering.
  • the processor 110 may include one or more GPUs that execute program instructions to generate or change display information.
  • the display screen 194 is used to display images, videos, and the like.
  • the display screen 194 includes a display panel.
  • the display panel can use liquid crystal display (LCD), organic light-emitting diode (OLED), active matrix organic light-emitting diode or active-matrix organic light-emitting diode (active-matrix organic light-emitting diode).
  • LCD liquid crystal display
  • OLED organic light-emitting diode
  • active-matrix organic light-emitting diode active-matrix organic light-emitting diode
  • AMOLED flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, quantum dot light-emitting diode (QLED), etc.
  • the terminal 100 may include one or N display screens 194, and N is a positive integer greater than one.
  • the terminal 100 can implement shooting functions through an ISP, a camera 193, a video codec, a GPU, a display screen 194, and an application processor.
  • the ISP is used to process the data fed back from the camera 193. For example, when taking a picture, the shutter is opened, the light is transmitted to the photosensitive element of the camera through the lens, the light signal is converted into an electrical signal, and the photosensitive element of the camera transmits the electrical signal to the ISP for processing and is converted into an image visible to the naked eye.
  • ISP can also optimize the image noise, brightness, and skin color. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene.
  • the ISP may be provided in the camera 193.
  • the camera 193 is used to capture still images or videos.
  • the object generates an optical image through the lens and is projected to the photosensitive element.
  • the photosensitive element may be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor.
  • CMOS complementary metal-oxide-semiconductor
  • the photosensitive element converts the optical signal into an electrical signal, and then transfers the electrical signal to the ISP to convert it into a digital image signal.
  • ISP outputs digital image signals to DSP for processing.
  • DSP converts digital image signals into standard RGB, YUV and other formats of image signals.
  • the terminal 100 may include one or N cameras 193, and N is a positive integer greater than one.
  • Digital signal processors are used to process digital signals. In addition to digital image signals, they can also process other digital signals. For example, when the terminal 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the energy of the frequency point.
  • Video codecs are used to compress or decompress digital video.
  • the terminal 100 may support one or more video codecs. In this way, the terminal 100 can play or record videos in multiple encoding formats, such as: moving picture experts group (MPEG) 1, MPEG2, MPEG3, MPEG4, and so on.
  • MPEG moving picture experts group
  • MPEG2 MPEG2, MPEG3, MPEG4, and so on.
  • NPU is a neural-network (NN) computing processor.
  • NN neural-network
  • applications such as intelligent cognition of the terminal 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, and so on.
  • the external memory interface 120 may be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the terminal 100.
  • the external memory card communicates with the processor 110 through the external memory interface 120 to realize the data storage function. For example, save music, video and other files in an external memory card.
  • the internal memory 121 may be used to store computer executable program code, where the executable program code includes instructions.
  • the processor 110 executes various functional applications and data processing of the terminal 100 by running instructions stored in the internal memory 121.
  • the internal memory 121 may include a storage program area and a storage data area.
  • the storage program area can store an operating system, an application program (such as a sound playback function, an image playback function, etc.) required by at least one function, and the like.
  • the data storage area can store data (such as audio data, phone book, etc.) created during the use of the terminal 100.
  • the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, a universal flash storage (UFS), and the like.
  • UFS universal flash storage
  • the terminal 100 can implement audio functions through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the earphone interface 170D, and the application processor. For example, music playback, recording, etc.
  • the audio module 170 is used to convert digital audio information into an analog audio signal for output, and is also used to convert an analog audio input into a digital audio signal.
  • the audio module 170 can also be used to encode and decode audio signals.
  • the audio module 170 may be provided in the processor 110, or part of the functional modules of the audio module 170 may be provided in the processor 110.
  • the speaker 170A also called “speaker” is used to convert audio electrical signals into sound signals.
  • the terminal 100 can listen to music through the speaker 170A, or listen to a hands-free call.
  • the receiver 170B also called “earpiece” is used to convert audio electrical signals into sound signals.
  • the terminal 100 answers a call or voice message, it can receive the voice by bringing the receiver 170B close to the human ear.
  • the microphone 170C also called “microphone”, “microphone”, is used to convert sound signals into electrical signals.
  • the user can make a sound by approaching the microphone 170C through the human mouth, and input the sound signal into the microphone 170C.
  • the terminal 100 may be provided with at least one microphone 170C. In other embodiments, the terminal 100 may be provided with two microphones 170C, which can implement noise reduction functions in addition to collecting sound signals. In other embodiments, the terminal 100 may also be provided with three, four or more microphones 170C to collect sound signals, reduce noise, identify sound sources, and realize directional recording functions.
  • the earphone interface 170D is used to connect wired earphones.
  • the earphone interface 170D may be a USB interface 130, or a 3.5mm open mobile terminal platform (open mobile terminal platform, OMTP) standard interface, or a cellular telecommunications industry association (cellular telecommunications industry association of the USA, CTIA) standard interface.
  • OMTP open mobile terminal platform
  • CTIA cellular telecommunications industry association of the USA
  • the pressure sensor 180A is used to sense the pressure signal and can convert the pressure signal into an electrical signal.
  • the pressure sensor 180A may be provided on the display screen 194.
  • the capacitive pressure sensor may include at least two parallel plates with conductive materials. When a force is applied to the pressure sensor 180A, the capacitance between the electrodes changes.
  • the terminal 100 determines the strength of the pressure according to the change in capacitance.
  • the terminal 100 detects the intensity of the touch operation according to the pressure sensor 180A.
  • the terminal 100 may also calculate the touched position according to the detection signal of the pressure sensor 180A.
  • touch operations that act on the same touch position but have different touch operation strengths may correspond to different operation instructions. For example: when a touch operation whose intensity of the touch operation is less than the first pressure threshold is applied to the short message application icon, an instruction to view the short message is executed. When a touch operation with a touch operation intensity greater than or equal to the first pressure threshold acts on the short message application icon, an instruction to create a new short message is executed.
  • the gyro sensor 180B may be used to determine the movement posture of the terminal 100.
  • the angular velocity of the terminal 100 around three axes ie, x, y, and z axes
  • the gyro sensor 180B can be used for image stabilization.
  • the gyroscope sensor 180B detects the shake angle of the terminal 100, calculates the distance that the lens module needs to compensate according to the angle, and allows the lens to counteract the shake of the terminal 100 through a reverse movement to achieve anti-shake.
  • the gyro sensor 180B can also be used for navigation and somatosensory game scenes.
  • the air pressure sensor 180C is used to measure air pressure.
  • the terminal 100 calculates the altitude based on the air pressure value measured by the air pressure sensor 180C to assist positioning and navigation.
  • the magnetic sensor 180D includes a Hall sensor.
  • the terminal 100 may use the magnetic sensor 180D to detect the opening and closing of the flip holster.
  • the terminal 100 can detect the opening and closing of the flip according to the magnetic sensor 180D.
  • features such as automatic unlocking of the flip cover are set.
  • the acceleration sensor 180E can detect the magnitude of the acceleration of the terminal 100 in various directions (generally three axes). When the terminal 100 is stationary, the magnitude and direction of gravity can be detected. It can also be used to recognize the terminal's posture, apply to horizontal and vertical screen switching, pedometer and other applications.
  • the terminal 100 can measure the distance by infrared or laser. In some embodiments, when shooting a scene, the terminal 100 may use the distance sensor 180F to measure the distance to achieve fast focusing.
  • the proximity light sensor 180G may include, for example, a light emitting diode (LED) and a light detector such as a photodiode.
  • the light emitting diode may be an infrared light emitting diode.
  • the terminal 100 emits infrared light to the outside through the light emitting diode.
  • the terminal 100 uses a photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the terminal 100. When insufficient reflected light is detected, the terminal 100 may determine that there is no object near the terminal 100.
  • the terminal 100 can use the proximity light sensor 180G to detect that the user holds the terminal 100 close to the ear to talk, so as to automatically turn off the screen to save power.
  • the proximity light sensor 180G can also be used in leather case mode, and the pocket mode will automatically unlock and lock the screen.
  • the ambient light sensor 180L is used to sense the brightness of the ambient light.
  • the terminal 100 can adaptively adjust the brightness of the display screen 194 according to the perceived brightness of the ambient light.
  • the ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures.
  • the ambient light sensor 180L can also cooperate with the proximity light sensor 180G to detect whether the terminal 100 is in a pocket to prevent accidental touch.
  • the fingerprint sensor 180H is used to collect fingerprints.
  • the terminal 100 can use the collected fingerprint characteristics to implement fingerprint unlocking, access application locks, fingerprint photographs, fingerprint answering calls, and so on.
  • the temperature sensor 180J is used to detect temperature.
  • the terminal 100 uses the temperature detected by the temperature sensor 180J to execute a temperature processing strategy. For example, when the temperature reported by the temperature sensor 180J exceeds a threshold value, the terminal 100 executes to reduce the performance of the processor located near the temperature sensor 180J, so as to reduce power consumption and implement thermal protection.
  • the terminal 100 when the temperature is lower than another threshold, the terminal 100 heats the battery 142 to avoid abnormal shutdown of the terminal 100 due to low temperature.
  • the terminal 100 boosts the output voltage of the battery 142 to avoid abnormal shutdown caused by low temperature.
  • Touch sensor 180K also called “touch panel”.
  • the touch sensor 180K may be disposed on the display screen 194, and the touch screen is composed of the touch sensor 180K and the display screen 194, which is also called a “touch screen”.
  • the touch sensor 180K is used to detect touch operations acting on or near it.
  • the touch sensor can pass the detected touch operation to the application processor to determine the type of touch event.
  • the visual output related to the touch operation can be provided through the display screen 194.
  • the touch sensor 180K may also be disposed on the surface of the terminal 100, which is different from the position of the display screen 194.
  • the bone conduction sensor 180M can acquire vibration signals.
  • the bone conduction sensor 180M can acquire the vibration signal of the human voice vibrating bone mass.
  • the bone conduction sensor 180M can also contact the human pulse and receive the blood pressure pulse signal.
  • the bone conduction sensor 180M may also be provided in the earphone, combined with the bone conduction earphone.
  • the audio module 170 can parse the voice signal based on the vibration signal of the vibrating bone block of the voice obtained by the bone conduction sensor 180M, and realize the voice function.
  • the application processor can analyze the heart rate information based on the blood pressure beating signal obtained by the bone conduction sensor 180M, and realize the heart rate detection function.
  • the button 190 includes a power button, a volume button, and so on.
  • the button 190 may be a mechanical button. It can also be a touch button.
  • the terminal 100 may receive key input, and generate key signal input related to user settings and function control of the terminal 100.
  • the motor 191 can generate vibration prompts.
  • the motor 191 can be used for incoming call vibration notification, and can also be used for touch vibration feedback.
  • touch operations applied to different applications can correspond to different vibration feedback effects.
  • Acting on touch operations in different areas of the display screen 194, the motor 191 can also correspond to different vibration feedback effects.
  • Different application scenarios for example: time reminding, receiving information, alarm clock, games, etc.
  • the touch vibration feedback effect can also support customization.
  • the indicator 192 may be an indicator light, which may be used to indicate the charging status, power change, or to indicate messages, missed calls, notifications, and so on.
  • the SIM card interface 195 is used to connect to the SIM card.
  • the SIM card can be inserted into the SIM card interface 195 or pulled out from the SIM card interface 195 to achieve contact and separation with the terminal 100.
  • the terminal 100 may support 1 or N SIM card interfaces, and N is a positive integer greater than 1.
  • the SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, etc.
  • the same SIM card interface 195 can insert multiple cards at the same time. The types of the multiple cards can be the same or different.
  • the SIM card interface 195 can also be compatible with different types of SIM cards.
  • the SIM card interface 195 may also be compatible with external memory cards.
  • the terminal 100 interacts with the network through the SIM card to implement functions such as call and data communication.
  • the terminal 100 adopts an eSIM, that is, an embedded SIM card.
  • the eSIM card can be embedded in the terminal 100 and cannot be separated from the terminal 100.
  • the software system of the terminal 100 may adopt a layered architecture, an event-driven architecture, a microkernel architecture, a microservice architecture, or a cloud architecture.
  • the embodiment of the present invention takes an Android system with a layered architecture as an example to illustrate the software structure of the terminal 100 by way of example.
  • FIG. 2B is a block diagram of the software structure of the terminal 100 according to an embodiment of the present invention.
  • the layered architecture divides the software into several layers, and each layer has a clear role and division of labor. Communication between layers through software interface.
  • the Android system is divided into four layers, from top to bottom, the application layer, the application framework layer, the Android runtime and system library, and the kernel layer.
  • the application layer can include a series of application packages.
  • the application package may include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message, etc.
  • the application framework layer provides an application programming interface (application programming interface, API) and a programming framework for applications in the application layer.
  • the application framework layer includes some predefined functions.
  • the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, and so on.
  • the window manager is used to manage window programs.
  • the window manager can obtain the size of the display screen, determine whether there is a status bar, lock the screen, take a screenshot, etc.
  • the content provider is used to store and retrieve data and make these data accessible to applications.
  • the data may include videos, images, audios, phone calls made and received, browsing history and bookmarks, phone book, etc.
  • the view system includes visual controls, such as controls that display text, controls that display pictures, and so on.
  • the view system can be used to build applications.
  • the display interface can be composed of one or more views.
  • a display interface that includes a short message notification icon may include a view that displays text and a view that displays pictures.
  • the phone manager is used to provide the communication function of the terminal 100. For example, the management of the call status (including connecting, hanging up, etc.).
  • the resource manager provides various resources for the application, such as localized strings, icons, pictures, layout files, video files, and so on.
  • the notification manager enables the application to display notification information in the status bar, which can be used to convey notification-type messages, and it can automatically disappear after a short stay without user interaction.
  • the notification manager is used to notify download completion, message reminders, etc.
  • the notification manager can also be a notification that appears in the status bar at the top of the system in the form of a chart or a scroll bar text, such as a notification of an application running in the background, or a notification that appears on the screen in the form of a dialog window. For example, text messages are prompted in the status bar, a prompt sound is emitted, the terminal vibrates, and the indicator light flashes.
  • Android Runtime includes core libraries and virtual machines. Android runtime is responsible for the scheduling and management of the Android system.
  • the core library consists of two parts: one part is the function functions that the java language needs to call, and the other part is the core library of Android.
  • the application layer and application framework layer run in a virtual machine.
  • the virtual machine executes the java files of the application layer and the application framework layer as binary files.
  • the virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.
  • the system library can include multiple functional modules. For example: surface manager (surface manager), media library (Media Libraries), three-dimensional graphics processing library (for example: OpenGL ES), 2D graphics engine (for example: SGL), etc.
  • the surface manager is used to manage the display subsystem and provides a combination of 2D and 3D layers for multiple applications.
  • the media library supports playback and recording of a variety of commonly used audio and video formats, as well as still image files.
  • the media library can support multiple audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.
  • the 3D graphics processing library is used to implement 3D graphics drawing, image rendering, synthesis, and layer processing.
  • the 2D graphics engine is a drawing engine for 2D drawing.
  • the kernel layer is the layer between hardware and software.
  • the kernel layer contains at least display driver, camera driver, audio driver, and sensor driver.
  • the corresponding hardware interrupt is sent to the kernel layer.
  • the kernel layer processes touch operations into original input events (including touch coordinates, time stamps of touch operations, etc.).
  • the original input events are stored in the kernel layer.
  • the application framework layer obtains the original input event from the kernel layer and identifies the control corresponding to the input event. Taking the touch operation as a touch click operation, and the control corresponding to the click operation is the control of the camera application icon as an example, the camera application calls the interface of the application framework layer to start the camera application, and then starts the camera driver by calling the kernel layer.
  • the camera 193 captures still images or videos.
  • This application provides an image processing method that can combine the processing capabilities of a terminal and a cloud server, make full use of high-quality image data and image processing algorithms on the cloud server, and empower the terminal camera imaging system.
  • the cloud server can use the image uploaded by the terminal and the environmental information of the image (such as geographic location information, weather information, time information, shooting parameters, shooting posture information, etc.) to match the image processing algorithm suitable for the image and the shooting content of the image Same, high-quality image data with similar shooting environment.
  • the image processing algorithm and the high-quality image data the quality and effect of the image uploaded by the terminal can be enhanced. In this way, pictures taken in different environments can show different styles and effects.
  • This also enables the terminal to break through the terminal's own imaging quality limit, so that the ordinary camera on the terminal can also take pictures of professional SLR equipment, thereby greatly improving the terminal's camera quality and user's camera experience while reducing the cost of mobile phones.
  • the following describes an image processing method related to an embodiment of the present application in combination with application scenarios.
  • a user uses the terminal 100 to take a photo
  • he can select different camera modes in the camera application of the terminal 100 to take a photo.
  • the user uses the terminal 100 to shoot scenery in places such as scenic spots, business districts, etc.
  • the user can select the "cloud enhancement mode" in the camera application to shoot.
  • the terminal 100 can use the processing capabilities of the cloud server to enhance and optimize the images captured by the terminal 100. In this way, the terminal 100 can break through the terminal's own imaging quality limit, so that the ordinary camera on the terminal can also take pictures of professional SLR equipment.
  • the terminal 100 when the terminal 100 is shooting a distant scene, due to the limitation of its own imaging quality limit, when the zoom is to a certain degree, the details of the distant scene photographed by the terminal 100 will become blurred.
  • the terminal 100 can use the processing capabilities of the cloud server to optimize the distant view image captured by the terminal 100, so that the terminal 100 can also take a clear picture when shooting the distant view. .
  • the terminal 100 may display an interface 310 of a home screen.
  • the interface 310 displays a page with application icons.
  • the page includes multiple application icons (for example, weather application icons, stock application icons, computing Browser application icon, settings application icon, mail application icon, Alipay application icon, Facebook application icon, browser application icon, gallery application icon 312, music application icon, video application icon, WeChat application icon, etc.).
  • a page indicator is also displayed below the multiple application icons to indicate the positional relationship between the currently displayed page and other pages.
  • There are multiple tray icons for example, dial-up application icon, information application icon, contact application icon, camera application icon 313) below the page indicator, and the tray icon remains displayed when the page is switched.
  • the above-mentioned page may also include multiple application icons and page indicators.
  • the page indicator may not be a part of the page and exist alone.
  • the above-mentioned picture icon is also optional, which is not limited in the embodiment of the present application.
  • a status bar 311 is displayed in the upper part of the interface 310.
  • the status bar 311 may include: one or more signal strength indicators of mobile communication signals (also called cellular signals), battery status indicators, and time indicators , Wi-Fi signal indicator, etc.
  • the terminal 100 may receive an input operation (for example, a click) of the camera application icon 313 by the user, and in response to the input operation, the terminal 100 may display a photographing interface 320 as shown in FIG. 3B.
  • an input operation for example, a click
  • the shooting interface 320 may display the captured image echo control 321, the shooting control 322, the camera conversion control 323, the image captured by the camera 324, the setting control 325, the zoom magnification control 326, one or more Shooting mode controls (for example, "night scene mode” control 372A, "portrait mode” control 372B, “cloud enhancement mode” control 372C, "normal photo mode” control 372D, "recording mode” control 372E, "professional mode” control 372F, More mode controls 327G).
  • the captured image echo control 321 can be used to display the captured image.
  • the shooting control 322 is used to trigger saving of the image captured by the camera.
  • the camera switching control 323 can be used to switch the camera for taking pictures.
  • the setting control 325 can be used to set a photographing function.
  • the zoom ratio control 326 can be used to set the zoom ratio of the camera.
  • the shooting mode control can be used to trigger the start of the image processing flow corresponding to the shooting mode.
  • the "night scene mode” control 372A can be used to trigger an increase in brightness and color richness in the captured image.
  • the "portrait mode” control 372B can be used to trigger the blurring of the background of the person in the captured image.
  • the "cloud enhancement mode” control 372C can be used to trigger the enhancement of the picture effect of the captured image with the help of the processing capability of the cloud server. As shown in FIG. 3B, the shooting mode currently selected by the user is the "normal shooting mode".
  • the terminal 100 may receive an input operation (for example, a click) of the user acting on the "cloud enhancement mode” control 327C, and in response to the input operation, as shown in FIG. 3C, the terminal 100 may adjust the shooting mode selected by the current user to the "cloud enhancement mode". Mode", and the mode details control 328 is displayed.
  • an input operation for example, a click
  • the terminal 100 may receive an input operation (for example, a click) of the user on the mode details control 328, and in response to the input operation, the terminal 100 may display the "cloud enhanced mode" details page 330 as shown in FIG. .
  • an input operation for example, a click
  • the terminal 100 may display the "cloud enhanced mode" details page 330 as shown in FIG. .
  • the "cloud enhancement mode” details page 330 may display an effect comparison chart including when the "cloud enhancement mode” is turned on and when it is not turned on.
  • the "cloud enhancement mode” details page 330 may display the image 331 taken when the “cloud enhancement mode” is not turned on and the detail image 332 of the image 331, and the image 333 taken when the "cloud enhancement mode” is turned on and the image 333 detailed drawing 334.
  • the “Cloud Enhancement Mode” details page 330 can also display a text description about “Cloud Enhancement Mode” 335 (for example, "Take high-resolution photos with the help of cloud capabilities to highlight details, improve color performance, and enhance image quality. It is recommended to use This mode enables data network and positioning when taking pictures.”).
  • the terminal 100 may receive an input operation (for example, a single click) that the user acts on the mode details control 328 again, and in response to the input operation, as shown in FIG. 3E, the terminal 100 may close the "cloud enhanced mode" details page 330 and return to the display The above-mentioned shooting interface 320.
  • an input operation for example, a single click
  • the shooting mode currently selected by the user is "cloud enhancement mode".
  • the terminal 100 may receive an input operation (for example, a single click) of the photographing control 322 by the user, and in response to the input operation, the terminal 100 may acquire an image photographed by the camera and environmental information of the image.
  • the environmental information may include one or more of the following: geographic location information, weather information, time information, shooting parameters, shooting posture information, and so on.
  • the shooting parameters may include zoom magnification and so on.
  • the shooting posture information may include the orientation information of the shooting lens of the shooting device when the image is taken. For example, the pitch angle and direction angle of the shooting lens, etc., or the three-axis rotation angle in natural coordinates, and so on.
  • the terminal 100 may send the captured image and the environmental information of the captured image to the cloud server 200, and use the processing capability of the cloud server 200 to enhance the captured image.
  • the process of enhancing the captured image by the terminal 100 with the help of the processing capability of the cloud server 200 will be described in detail in subsequent embodiments, and will not be described in detail here.
  • the terminal 100 when the terminal 100 uses the processing capability of the cloud server 200 to enhance the captured image, the terminal 100 may display a processing progress prompt or countdown to remind the user that the terminal 100 currently uses the processing capability of the cloud server 200 to The progress of the enhancement of the captured image.
  • the terminal 100 may display a progress prompt window 341 on the photographing interface 320 for prompting the progress of enhancing the photographed image.
  • the progress prompt window 341 displays a text prompt "The photo is being optimized through the cloud, and the progress is 10%".
  • the progress prompt window 341 may also display a cancel control 342, which may be used to trigger the cancellation of the enhancement of the captured image.
  • the terminal 100 may send a cancel instruction to the cloud server 200 to instruct the cloud service 200 to stop the enhancement process of the captured image. In this way, when the progress is slow or when the user temporarily does not want to enhance the captured image, the enhancement of the captured image can be cancelled immediately.
  • the terminal 100 can display the enhanced effect in real time.
  • the terminal can end the enhancement in advance, and save the halfway enhancement result at the end as the final result. .
  • the terminal 100 may display a completion prompt and save the enhanced image locally.
  • the completion prompt may be used to remind the user that the enhancement of the captured image has been completed currently.
  • the terminal 100 may display a completion prompt 343 after completing enhancement of the captured image with the help of the processing capability of the cloud server 200.
  • the completion prompt 343 may be a text prompt "The photo enhancement has been completed and can be viewed in the gallery".
  • the terminal 100 may receive an input operation (for example, a single click) of the captured image echo control 321 by the user, and in response to the input operation, the terminal 100 may display a photo browsing interface 350 as shown in FIG. 3H.
  • an input operation for example, a single click
  • the photo browsing interface 350 may display an enhanced image 351, a mark 352, image related information 353, a menu 354, and a gallery control 355.
  • the mark 352 is used to remind the user that the image 351 has been enhanced with the processing capability of the cloud server 200.
  • the image related information 353 may include the shooting time of the image 351, shooting weather, geographic location information, and so on. For example, the shooting time can be "December 3, 2019 8:00 AM", the shooting weather can be "cloudy”, the shooting location can be "Shanghai ⁇ Disney” and so on.
  • the menu 354 may include a share button, a favorite button, an edit button, a delete button, and more buttons. The share button can be used to trigger the sharing of the image 351.
  • the favorite button can be used to trigger the favorite image 351 to the picture favorite folder.
  • the editing button can be used to trigger editing functions such as rotation, trimming, adding filters, and blurring of the image 351.
  • the delete button can be used to trigger the deletion of the image 351. More buttons can be used to trigger opening of more functions related to the image 351.
  • the gallery control 355 can be used to trigger the terminal 100 to open the gallery application.
  • the terminal 100 since the terminal 100 uses the processing capability of the cloud server 200 to enhance the image, it needs to first send the image and the environmental information of the image to the cloud server 200. Therefore, when the terminal 100 opens the camera application and starts the "cloud enhanced" mode, the terminal 100 can detect the current network status. If the current network disconnection of the terminal 100 cannot connect to the cloud server, the terminal 100 can output a network disconnection prompt.
  • the network disconnection prompt can be used to remind the user to use the cloud enhanced mode to take pictures after connecting to the network.
  • the terminal 100 displays a shooting interface 320.
  • the terminal 100 detects that the network is disconnected and cannot establish a connection with the cloud server 200
  • the terminal 100 can display a prompt 411, which can be used to prompt the user that the network of the terminal 100 is currently disconnected.
  • the prompt 411 may be a text prompt "The current network is disconnected, please connect to the network first, and then optimize the picture with the help of the cloud" and so on.
  • the terminal 100 When the terminal 100 opens the camera application and starts the cloud enhancement mode, when the network status of the terminal 100 switches from disconnected to reconnected, the terminal 100 can output a network reconnection prompt, which can be used to remind the user that the current network connection has been restored. You can use the cloud enhancement mode to take pictures normally.
  • the terminal 100 displays a shooting interface 320.
  • the terminal 100 detects that the network is switched from disconnected to reconnected, the terminal 100 can display a prompt 412, which can be used to prompt the user that the current network of the terminal 100 has been restored, and the cloud enhanced mode can be used normally to take pictures.
  • the prompt 412 may be a text prompt "The current network is connected, and you can use the cloud enhancement mode to take pictures.
  • the cloud server 200 can process images uploaded by multiple terminals for enhancement, when the cloud server 200 has too many tasks for image enhancement, the cloud server 200 queues these tasks.
  • the terminal 100 requests that a picture enhancement task needs to be queued, the terminal 100 may output a queue prompt, and maintain communication with the cloud server 200 in the background to monitor the queue progress.
  • the picture enhancement task can be used to prompt that the current image enhancement task is queuing and needs to be waited.
  • the terminal 100 displays a shooting interface 320.
  • the terminal 100 may periodically obtain the processing progress of enhancing the captured image from the cloud server 200.
  • the cloud server 200 may queue the task requests from the terminal 100 and periodically send the queue progress to the terminal 100.
  • the terminal 100 may output a prompt 413, which is used to prompt the user that the task of the terminal 100 for enhancing the captured image is currently being queued.
  • the prompt 413 can be a text prompt "The current user's perspective of enhancing pictures through the cloud has been queued for you. You can check the queue progress in the notification bar.”
  • the terminal 100 may maintain communication with the cloud server 200, monitor the queue progress in the background, and may display the queue progress in the notification bar of the terminal 100.
  • the terminal 100 displays a shooting interface 320.
  • the terminal 100 can receive the user's input operation and call up the status bar 311 above the shooting interface 320.
  • the terminal 100 may receive an input operation (for example, sliding down) of the status bar 311 by the user, and in response to the input operation, the terminal 100 may display a menu window 420 and a notification bar 430 as shown in FIG. 4E.
  • the menu window 420 can display switch controls with some functions (for example, wireless local area network (WLAN) switch control, Bluetooth switch control, flashlight switch control, bell switch control, automatic rotary switch control, Huawei share (Huawei Share) switch controls, etc.).
  • the notification bar 430 may display queuing notifications (for example, “There are more users who use the cloud to enhance the picture effect, and they are currently queuing”), enhanced queue time prompts 432 for taking pictures (for example, “Estimated waiting time of 1 minute”) and cancellation Wait for control 431.
  • the cancel waiting control 431 may be used to trigger the terminal 100 to send a cancel request to the cloud server 200, and the cancel request is used to request the cloud server 200 to cancel the enhancement of the uploaded image of the terminal 100.
  • the terminal 100 may display a completion prompt 433 as shown in FIG. 4F.
  • the prompt 433 may be used to prompt the user that the current terminal 100 has completed the enhancement of the captured image.
  • the terminal 100 may receive an input operation (for example, a click) of the prompt 433 by the user, and in response to the input operation, the terminal 100 may display the photo browsing interface 350 shown in FIG. 3H.
  • an input operation for example, a click
  • the terminal 100 may display the photo browsing interface 350 shown in FIG. 3H.
  • FIG. 3H For the text description of the photo browsing interface 350, reference may be made to the foregoing embodiment, which will not be repeated here.
  • the embodiment of the present application provides an image processing method that enables the terminal 100 to enhance and optimize the image captured by the terminal 100 in the "cloud enhancement mode" by using the processing capabilities of the cloud server 200.
  • the terminal 100 can break through the terminal's own imaging quality limit, so that the ordinary camera on the terminal can also take pictures of professional SLR equipment.
  • FIG. 4G shows the image taken before the terminal 100 is enhanced with the processing capability of the cloud server 200
  • FIG. 4G shows the processing capability of the terminal 100 with the aid of the cloud server 200 Enhance the image obtained after taking the image.
  • C) in FIG. 4G shows a partial view of the image taken in (a) of FIG. 4G
  • (d) in FIG. 4G shows a partial view of the enhanced image in (b) of FIG. 4G. Comparing (a) and (b) in FIG. 4G, and (c) and (d) in FIG. 4G, it can be seen that the terminal 100 enhances the captured image by using the processing capabilities of the cloud server 200 to improve the overall image and details The sharpness of the image also improves the contrast, dynamic range, color, etc. of the image.
  • the terminal 100 may store many pictures locally, and these pictures may be taken by the terminal 100, sent from other devices, or downloaded from the Internet. Among them, some pictures cannot present high-quality image effects due to the limitation of the shooting capabilities of the shooting equipment. Therefore, an embodiment of the present application provides an image processing method.
  • the terminal 100 uses the processing capability of the cloud server 200 to enhance the images stored in the gallery. In this way, the image effect of the captured image can be improved, and it is also convenient for the user to shoot After the image is completed, the processing capability of the cloud server 200 is used to enhance the effect of the captured image.
  • the terminal 100 may display an interface 510 on the home screen.
  • the interface 510 displays a page on which application icons are placed.
  • the page includes multiple application icons (for example, weather application icons, stock application icons, and computing icons).
  • a page indicator is also displayed below the multiple application icons to indicate the positional relationship between the currently displayed page and other pages.
  • There are multiple tray icons for example, dial-up application icon, information application icon, contact application icon, camera application icon 513) below the page indicator, and a status bar 511 is displayed in the upper part of the interface 510.
  • the interface 510 is the same as the interface 310 shown in FIG. 3A, the detailed text description of the interface 510 can refer to the embodiment shown in FIG. 3A, which will not be repeated here.
  • the terminal 100 may receive the user's input operation on the gallery application icon 512, and in response to the input operation, the terminal 100 may display the gallery album interface 520 as shown in FIG. 5B.
  • the gallery album interface 520 may display that it includes one or more albums (for example, all photo albums 526, video albums, camera albums, cloud-enhanced albums 527, WeChat albums, Weibo albums, etc.).
  • the terminal 100 may display a gallery menu 521 below the gallery album interface 520.
  • the gallery menu 521 includes a photo control 522, an album control 523, a time control 524, and a discovery control 525.
  • the photo control 522 is used to trigger the terminal 100 to display all local pictures in the form of picture thumbnails.
  • the album control 523 is used to trigger the terminal 100 to display the album to which the local picture belongs. As shown in FIG.
  • the current album control 523 is in a selected state, and the terminal 100 displays the gallery album interface 520.
  • the moment control 524 can be used to trigger the terminal 100 to display locally stored selected pictures.
  • the discovery control 525 can be used to trigger the terminal 100 to display a classified album of pictures. For example, the terminal 100 can automatically classify pictures according to people, locations, scenic spots, and business districts.
  • the terminal 100 may receive an input operation (for example, a single click) of the discovery control 525 from the user, and in response to the input operation, the terminal 100 may display a gallery discovery interface 530 as shown in FIG. 5C.
  • an input operation for example, a single click
  • the gallery discovery interface 530 displays a portrait category 531, a location category 532, and a scenic spot and business district category 533.
  • the portrait category 531 includes one or more pictures containing portrait content.
  • the location category 532 includes one or more location photo albums (for example, "Xiamen” photo album, "Shanghai” photo album, "Shenzhen” photo album, etc.).
  • Scenic spots and business districts 533 include one or more scenic spots and business district albums (for example, "Shanghai ⁇ Disney” album, "Shanghai ⁇ The Bund” album, "Beijing ⁇ Forbidden City” album, etc.).
  • the terminal 100 may receive an input operation (for example, a single click) of the user for the scenic spot and business district classification 533, and in response to the input operation, the terminal 100 may display the scenic spot and business district classification interface 540 as shown in FIG. 5D.
  • an input operation for example, a single click
  • the scenic spot and business district classification interface 540 shows that it includes one or more albums (for example, album 541, album 542, album 543, album 544, etc.) and cloud enhancement control 545.
  • albums for example, album 541, album 542, album 543, album 544, etc.
  • cloud enhancement control 545 under each album, there is the name of the scenic spot or business district corresponding to the album and the number of pictures in the album.
  • the name of the scenic spot or business district corresponding to the album 541 is "Shanghai ⁇ Disney", and there are 3 pictures in the album 541.
  • the name of the scenic spot or business district corresponding to the album 542 is "Shanghai ⁇ The Bund", and there are 2 pictures in the album 542.
  • the name of the scenic spot or business district corresponding to the album 543 is "Beijing ⁇ Forbidden City", and there are 2 pictures in the album 543.
  • the name of the scenic spot or business district corresponding to this album 544 is "Xiamen ⁇ Gulangyu Island”, and there are 2 pictures in the album 544.
  • the terminal 100 may receive an input operation (for example, a click) of the user acting on the cloud enhancement control 545, and in response to the input operation, the terminal 100 may display a cloud enhancement interface 550 as shown in FIG. 5E.
  • an input operation for example, a click
  • the cloud enhancement interface 550 displays a picture adding control 551 and a detailed information control 552.
  • the picture adding control 551 can be used to trigger the terminal 100 to select a picture to be enhanced with the help of the processing capability of the cloud server 200.
  • the learn more control 552 can be used to trigger the terminal 100 to display the effect of the cloud enhancement function and related information.
  • the terminal 100 may receive an input operation (for example, a single click) of the user for the details control 552, and in response to the input operation, the terminal 100 may display a cloud enhancement details page 560 as shown in FIG. 5F.
  • an input operation for example, a single click
  • the cloud enhancement details page 560 displays an effect comparison diagram of the cloud enhancement function.
  • the cloud enhancement details page 560 may display the picture 561 before cloud enhancement processing and the detail map 562 of the picture 561, and the picture 563 after cloud enhancement processing and the detail map 564 of the picture 563, and the comparison result (for example, "Enhanced The front details are blurred, and the details are clearer after enhancement”).
  • the cloud enhancement details page 560 can also display a text description 565 about the cloud enhancement function (for example, "With the help of cloud capabilities to enhance the photo, you can highlight details, improve clarity and color performance, and enhance the quality of the image") .
  • the cloud enhancement details page 560 may also display a page indicator 566, and the page indicator 566 may be used to indicate the positional relationship between the page where the current effect comparison graph is located and the page where other effect comparison graphs are located.
  • the terminal 100 may receive an input operation (for example, swiping to the left) of the user on the cloud enhancement details page 560, and in response to the input operation, as shown in FIG. 5G, the terminal 100 may switch and display another set of effect comparisons in the cloud enhancement details page 560 Figure.
  • an input operation for example, swiping to the left
  • the terminal 100 may switch and display another set of effect comparisons in the cloud enhancement details page 560 Figure.
  • the terminal 100 can display a picture 571 before cloud enhancement processing and a picture 572 after cloud enhancement processing in the cloud enhancement page 560, and a comparison result 573 (for example, "The colors of the enhanced picture are more rich and balanced, rich Texture").
  • the terminal 100 can receive the user's operation and select a picture in the gallery application.
  • the terminal 100 can obtain pictures and environmental information of the pictures (such as geographic location information, weather information, time information, shooting parameters, shooting posture information, etc.), upload them to the cloud server 200, and use the processing capabilities of the cloud server 200 to analyze the pictures selected by the user Make enhancements.
  • the terminal 100 displays a cloud enhanced interface 550.
  • the cloud enhancement interface 550 displays a picture adding control 551 and a detailed information control 552.
  • the terminal 100 may receive an input operation (for example, a single click) of the user on the picture adding control 551, and in response to the operation, the terminal 100 may display a picture selection interface 610 as shown in FIG. 6B.
  • the picture selection interface 610 displays one or more albums that can be cloud-enhanced (for example, "Shanghai ⁇ Disney” album, "Shanghai ⁇ The Bund” album 611, "Beijing ⁇ Forbidden City” album , "Xiamen ⁇ Gulangyu” photo album, etc.).
  • the environmental information of the pictures in these albums is complete, which can be enhanced by using the processing capabilities of the cloud server 200.
  • Each album in the picture selection interface 610 may also display a selection box (for example, the selection box 612 displayed on the "Shanghai ⁇ The Bund” album 611), and the selection box can be used to select all pictures in the album. Below each album shows the number of pictures in the album.
  • the terminal 100 may receive an input operation (for example, a single click) of the user on the "Shanghai Bund" album 611, and in response to the input operation, the terminal 100 may have an album interface 620 as shown in FIG. 6C.
  • an input operation for example, a single click
  • the name of the album interface 620 is "Shanghai ⁇ The Bund".
  • the album interface 620 displays thumbnails including one or more pictures with complete environmental information (for example, thumbnails 621) and a determination control 623.
  • a selection box is displayed on the thumbnail of each picture in the album interface 620, and the selection box can be used to select the picture.
  • the user has selected the picture corresponding to the thumbnail 621.
  • the terminal 100 may receive an input operation (for example, a click) of the user on the determination control 623. In response to the input operation, the terminal 100 may return to display the picture selection interface 610 shown in FIG.
  • the selection interface 610 displays the number of selected pictures and a determination control 613.
  • the terminal 100 displays the number of selected pictures 614 (for example, "a total of 1 picture is selected") and a determination control 613 on the picture selection interface 610.
  • the terminal 100 may receive an input operation (for example, a single click) of the user on the determination control 613, and in response to the input operation, the terminal 100 may display a thumbnail (for example, a thumbnail of the selected picture) on the cloud enhancement interface 550 as shown in FIG. 6E Thumbnail 631) and upload control 633.
  • a delete control 632 is displayed below the thumbnail of the selected picture, and the delete control 632 can be used to trigger the terminal 100 to delete the picture selected by the user.
  • the upload control 633 may display a text description (for example, "upload cloud enhancement effect").
  • the upload control 633 can be used for the terminal 100 to upload the picture selected by the user and the environmental information of the picture to the cloud server 200, and the processing capability of the cloud server 200 is used to enhance the effect of the picture selected by the user.
  • the terminal 100 since the terminal 100 uses the processing capabilities of the cloud server 200 to enhance the picture selected by the user, it is necessary to send the picture selected by the user and the environmental information of the picture to the cloud server 200 first. Therefore, when the user triggers the terminal 100 to start uploading the selected picture and the environmental information of the picture, the terminal 100 can detect the current network status. If the network status is disconnected and cannot connect to the cloud server 200, the terminal 100 can output an upload failure prompt , The upload failure prompt can be used to prompt the user to upload after connecting to the network.
  • the terminal 100 receives a user's input operation (for example, a click) on the upload control 633, and in response to the input operation, the terminal 100 may first detect the current network status. If the current terminal 100 is disconnected from the network and cannot connect to the cloud server 200, the terminal 100 displays an upload failure prompt 634 as shown in FIG. 6F.
  • a user's input operation for example, a click
  • the upload failure prompt 634 displays a text prompt "Upload failed. To enhance the image effect through the cloud, it needs to be connected to the Internet. Please connect to the Internet before uploading the image to the cloud to enhance the effect.”
  • the terminal 100 may receive user input, connect to the network, and communicate with the cloud server 200. After the terminal 100 is connected to the network, it can receive the user's input operation on the upload control 633 again, and send the picture selected by the user and the environmental information of the picture to the cloud server 200.
  • the terminal 100 may receive an input operation (such as sliding down) of the user on the status bar 511, and in response to the input operation, the terminal 100 may display a menu window 640 as shown in FIG. 6G.
  • an input operation such as sliding down
  • the terminal 100 may display a menu window 640 as shown in FIG. 6G.
  • the menu window 420 can display switch controls with some functions (for example, wireless local area network (WLAN) switch control 641, Bluetooth switch control, flashlight switch control, bell switch control, automatic rotary switch control, Huawei Share (Huawei Share) switch controls, etc.).
  • WLAN wireless local area network
  • Bluetooth switch control for example, Bluetooth switch control, flashlight switch control, bell switch control, automatic rotary switch control, Huawei Share (Huawei Share) switch controls, etc.
  • the current wireless local area network control 641 is in a closed state.
  • the terminal 100 may receive an input operation (for example, a single click) of the wireless local area network control 641 from the user, and in response to the input operation, the terminal 100 may turn on the wireless local area network switch and connect to a connectable Wi-Fi network. As shown in FIG. 6H, the terminal 100 connects to the Wi-Fi network after turning on the wireless local area network switch, and displays the name of the connected Wi-Fi network (for example, "HW-ZL") under the wireless local area network control 641, and The Wi-Fi signal indicator 642 is displayed in the status bar 511. The Wi-Fi signal indicator 642 is used to indicate that the current network status of the terminal 100 is normal and can be connected to the cloud server 200.
  • an input operation for example, a single click
  • the terminal 100 can receive an input operation (for example, a click) of the upload control 633 in the cloud enhanced interface 550 by the user.
  • the terminal 100 can select the picture selected by the user and The environmental information of the picture is uploaded to the cloud server 200.
  • the cloud server 200 can process images uploaded by multiple terminals for enhancement, when the cloud server 200 has too many tasks for image enhancement, the cloud server 200 queues these tasks.
  • the terminal 100 requests that a picture enhancement task needs to be queued, the terminal 100 may output a queue prompt, and maintain communication with the cloud server 200 in the background to monitor the queue progress.
  • the picture enhancement task can be used to prompt that the current image enhancement task is queuing and needs to be waited.
  • the terminal 100 may periodically obtain from the cloud server 200 the processing progress of enhancing the selected picture.
  • the cloud server 200 may queue the task requests from the terminal 100 and periodically send the queue progress to the terminal 100.
  • the terminal 100 may output a progress prompt window 650.
  • the progress prompt window 650 shows the reason for the slow progress of the current task (for example, "There are more users who use the cloud to enhance the picture effect, and they are currently queuing"), and the waiting time 651 (for example, "Total 1 picture, estimated waiting time 1 minute”) and cancel the waiting control 652.
  • the cancellation waiting control 652 can be used to trigger the cloud server 200 of the terminal 100 to send a cancellation request, and the cancellation request is used to request the cloud server 200 to cancel the enhancement of the uploaded picture of the terminal 100.
  • the cloud server 200 may send a processing start response to the terminal 100. After receiving the processing start response, the terminal 100 may display a processing prompt, which can be used to prompt the user
  • the cloud server 200 is currently processing an image enhancement task.
  • the terminal 100 may display a processing prompt 653 when the cloud server 200 processes the image enhancement task of the terminal 100.
  • the processing prompt 653 may be a text prompt "The cloud is processing, please wait, there is a total of 1 picture" and so on.
  • the examples are only used to explain the application and should not constitute a limitation.
  • the terminal 100 may output the network disconnection. Open the prompt, the network disconnection prompt can be used to remind the user that the current network is disconnected and needs to be connected to the network to obtain the picture enhancement result.
  • the network of the terminal 100 is disconnected and cannot connect to the cloud server 200.
  • the terminal 100 may display a network disconnection identifier 643 in the status bar 511 to prompt the user that the network is currently disconnected.
  • the terminal 100 may also display a network disconnection prompt 654.
  • the network disconnection prompt 654 may be a text prompt "It has been detected that your current network has been disconnected, please reconnect to the network to obtain a cloud-enhanced picture.”
  • the terminal 100 may display an enhancement success prompt window 660 on the cloud enhancement interface 550.
  • the enhancement success prompt window 660 may display an effect description 661 that includes enhancement of the selected picture.
  • the text content corresponding to the effect description 661 can be "The location of the picture you selected is'Shanghai ⁇ The Bund'. The details of the building in the picture have been made more prominent and clearer through the cloud, and the dynamic range of the picture and the color of the picture have been improved, so as to fully show the city. Style!.
  • the enhancement success prompt window 660 displays a thumbnail 662 corresponding to the enhanced picture, a view control 663, a download control 664, and all download controls 665.
  • the viewing control 663 can be used to view the enhanced picture.
  • the download control 664 can be used to save the enhanced picture to a local designated storage path of the terminal 100.
  • the all download control 665 can be used to simultaneously save the multiple enhanced pictures to the local designated storage path of the terminal 100 when there are multiple enhanced pictures.
  • the terminal 100 may receive an input operation (for example, a click) of the viewing control 663 by the user, and in response to the input operation, the terminal 100 may display a picture browsing interface 670 as shown in FIG. 6N.
  • an input operation for example, a click
  • the picture browsing interface 670 displays an enhanced picture 671, a mark 672, a download control 673, an effect comparison control 674, picture-related information 675, and a menu 676.
  • the mark 672 is used to remind the user that the picture 671 has been enhanced with the processing capability of the cloud server 200.
  • the picture-related information 675 may include the shooting time of the picture 671, shooting weather, geographic location information, and so on. For example, the shooting time can be "December 1, 2019 8:00 AM", the shooting weather can be “cloudy”, the shooting location can be "Shanghai ⁇ The Bund” and so on.
  • the menu 676 may include a share button, a favorite button, an edit button, a delete button, and more buttons.
  • the share button can be used to trigger the sharing of the picture 671.
  • the favorite button can be used to trigger the favorite picture 671 to the picture favorite folder.
  • the editing button can be used to trigger editing functions such as rotation, trimming, adding filters, and blurring of the picture 671.
  • the delete button can be used to trigger the deletion of the picture 671.
  • the More button can be used to trigger opening of more functions related to the picture 671.
  • the effect comparison control 674 can be used to trigger the display of the picture before the enhancement of the picture 671 at the position of the picture 671 (that is, the picture selected by the user).
  • the terminal 100 may receive an input operation (for example, a click) of the user on the effect comparison control 674, and in response to the input operation, as shown in FIG. 60, the terminal 100 may display a picture 677 selected by the user in the display position area of the picture 671.
  • an input operation for example, a click
  • the terminal 100 may again receive an input operation (for example, a single click) of the user on the effect comparison control 674, and in response to the input operation, as shown in FIG. 6P, the terminal 100 may display an enhanced picture 671.
  • an input operation for example, a single click
  • the terminal 100 can receive an input operation (for example, a click) of the user on the download control 673, and in response to the input operation, the terminal 100 can save the enhanced picture 671 to the local designated storage path of the terminal 100. And after the saving is completed, a saving completion prompt 678 is displayed, and the saving completion prompt 678 is used to prompt the user that the enhanced picture 671 has been saved.
  • an input operation for example, a click
  • the terminal 100 may save the captured picture locally after taking a picture.
  • the user can view pictures taken by the terminal 100 and pictures obtained from other devices or the Internet in the gallery application of the terminal 100.
  • the terminal 100 can mark pictures to be enhanced, pictures that have been enhanced, and pictures that have been enhanced in the gallery application.
  • the environmental information of the picture for example, geographic location information, weather information, time information, shooting parameters, shooting posture information, etc.
  • the picture can be classified as the above-mentioned to-be-enhanced picture of. In this way, it is convenient for the user to quickly select a picture that needs to be enhanced with the help of the processing capability of the cloud server 200, and save the user's time.
  • the terminal 100 may display a gallery album interface 520.
  • the gallery album interface 520 may display one or more albums (for example, all photo albums 526, video albums, camera albums, cloud-enhanced albums 527, WeChat albums, Weibo albums, etc.).
  • albums for example, all photo albums 526, video albums, camera albums, cloud-enhanced albums 527, WeChat albums, Weibo albums, etc.
  • FIG. 5B For the detailed text description of the photo gallery album interface 520, please refer to the embodiment shown in FIG. 5B, which will not be repeated here.
  • the terminal 100 can receive an input operation (for example, a single click) of the user on all photo albums 526, and in response to the input operation, the terminal 100 can display an all photo interface 710 as shown in FIG. 7B.
  • an input operation for example, a single click
  • the all photos interface 710 displays thumbnails including one or more pictures.
  • the terminal 100 may mark the picture to be enhanced, the enhanced picture, and the enhanced picture among the one or more pictures.
  • the picture corresponding to the thumbnail 711 has been enhanced, and the terminal 100 uses the processing capability of the cloud server 200 to enhance the picture corresponding to the thumbnail 711 to obtain the picture corresponding to the thumbnail 713.
  • the picture corresponding to the thumbnail 715 is a picture to be enhanced.
  • the terminal 100 may display a mark 712 on the thumbnail 711, and the mark 712 is used to remind the user that the picture corresponding to the thumbnail 711 has been enhanced with the processing capability of the cloud server 200.
  • the terminal 100 may display a mark 714 on the thumbnail 713, and the mark 714 is used to prompt the user that the picture corresponding to the thumbnail 713 is a picture enhanced with the processing capability of the cloud server 200.
  • the terminal 100 may display a mark 716 on the thumbnail 715, and the mark 716 is used to prompt the user that the picture corresponding to the thumbnail 715 is to be enhanced with the help of the processing capability of the cloud server 200.
  • the terminal 100 may receive an input operation (for example, a single click) of the user acting on the thumbnail 715, and in response to the input operation, the terminal 100 may display a picture browsing interface 720 as shown in FIG. 7C. Since the picture corresponding to the thumbnail 715 is waiting to be enhanced with the help of the processing capability of the cloud server 200, the terminal 100 may display the picture corresponding to the thumbnail 715 in the picture browsing interface 720 and a cloud enhancement control, which is used to trigger the terminal 100 The image corresponding to the thumbnail 715 is enhanced with the help of the processing capability of the cloud server 200.
  • an input operation for example, a single click
  • the picture browsing interface 720 may display the picture 721 corresponding to the thumbnail 715, the cloud enhancement control 722, and the picture related information. 723, menu 724.
  • the picture-related information 723 may include the shooting time of the picture 721, shooting weather, geographic location information, and so on. For example, the shooting time can be "December 1, 2019 8:00 AM”, the shooting weather can be “cloudy”, the shooting location can be "Shanghai ⁇ The Bund” and so on.
  • the menu 724 may include a share button, a favorite button, an edit button, a delete button, and more buttons. The share button can be used to trigger the sharing of the picture 721.
  • the collection button can be used to trigger the collection of the picture 721 to the picture collection folder.
  • the edit button can be used to trigger editing functions such as rotation, trimming, adding filters, and blurring of the picture 721.
  • the delete button can be used to trigger the deletion of the picture 721.
  • the More button can be used to trigger the opening of more functions related to the picture 721.
  • the effect comparison control 722 can be used to trigger the terminal 100 to enhance the picture 721 with the help of the processing capability of the cloud server 200.
  • the terminal 100 may receive an input operation (for example, a click) of the cloud enhancement control 722 by the user, and in response to the input operation, the terminal 100 may display a cloud enhancement interface 730 as shown in FIG. 7D.
  • an input operation for example, a click
  • the cloud enhancement interface 730 can display a picture addition control 731, a thumbnail of a selected picture (such as a thumbnail 732), a delete control 733, a detailed information control 734, and an upload control 735.
  • a picture addition control 731 a thumbnail of a selected picture (such as a thumbnail 732)
  • a delete control 733 a thumbnail of a selected picture
  • a detailed information control 734 a detailed information control 734
  • an upload control 735 an upload control 735.
  • the detailed text description of the cloud enhanced interface 730 can refer to the embodiment shown in FIG. 6E, which will not be repeated here.
  • the terminal 100 can receive the user's input operation on the upload control 735. In response to this operation, the terminal 100 can upload the picture selected by the user and the environmental information of the picture to the cloud server 200, and use the processing capabilities of the cloud server 200 to enhance the user's selected picture (That is, the effect of the picture 721) shown in FIG. 7C.
  • the following describes how the terminal 100 in the embodiment of the present application enhances the picture with the help of the processing capability of the cloud server 200.
  • FIG. 8 shows a schematic flowchart of an image processing method provided by an embodiment of the present application. As shown in Figure 8, the method includes:
  • the terminal 100 receives a user's first input.
  • the terminal 100 acquires the first image and the first environment information of the first image.
  • the first environment information may include one or more of geographic location information, weather information, season information, time information, shooting posture information, shooting parameters, and so on.
  • the shooting parameters may include zoom magnification and so on.
  • the shooting posture information may include the orientation information of the shooting lens of the shooting device when the image is taken. For example, the pitch angle and direction angle of the shooting lens, etc., or the three-axis rotation angle in natural coordinates, and so on.
  • the first input may be that in the embodiment shown in FIG. 3E, after the user turns on the cloud enhancement mode in the shooting interface 320, the shooting controls 322 input operation (for example, click).
  • the first image may be a picture 324 (that is, a captured image) captured by the camera shown in FIG. 3E.
  • the terminal 100 may obtain the first environment information of the first image from one or more sensors and the network.
  • the terminal 100 obtains the GPS location information of the captured image through the GPS positioning module.
  • the geographic location information is a geographic name
  • the terminal 100 can also obtain weather information, seasonal information, etc. of the captured image from the Internet, in combination with the shooting time and geographic location information.
  • the terminal 100 may also obtain the shooting posture information of the captured image through sensors such as a gyroscope and an acceleration sensor.
  • the shooting parameters may include zoom magnification, etc., and the terminal 100 may obtain the shooting parameters set by the user through a camera application and the like.
  • the first input may be that in the embodiment shown in FIG. 6I, after the user selects the picture corresponding to the thumbnail 631 in the cloud enhancement interface 550, the upload control 633 input operation.
  • the first input may be an input operation on the upload control 735 after the user selects a picture corresponding to the thumbnail 732 in the cloud enhancement interface 730 in the embodiment shown in FIG. 7D.
  • the first image may be a picture corresponding to the thumbnail 631 in the embodiment shown in FIG. 6I or a picture corresponding to the thumbnail 732 in the embodiment shown in FIG. 7D.
  • the photographing device terminal 100 or other equipment
  • the first environmental information for example, geographic location information, weather information, time information, season information, photographing parameters, and photographing posture
  • Information, etc. are saved to the exchangeable image file format (exchange image file format, Exif) data of the picture. Therefore, when the above-mentioned first image is a picture that is locally saved in the terminal 100 selected by the user, the terminal 100 can obtain the first environment information of the picture selected by the user from the Exif data corresponding to the picture selected by the user.
  • the terminal 100 may mark the picture to be enhanced, the enhanced picture, and the enhanced picture in the gallery application.
  • the picture when the first environment information of the picture is complete and has not been enhanced by the cloud server 200, the picture can be classified as the picture to be enhanced.
  • the picture When a picture has been selected by the user for optimization with the help of the processing capability of the cloud server 200, and a corresponding high-quality picture is generated, the picture is classified as the above-mentioned enhanced picture.
  • the above-mentioned enhanced picture may refer to a high-quality picture generated after the picture selected by the user is optimized with the help of the processing capability of the cloud server 200. In this way, it is convenient for the user to quickly select a picture that needs to be enhanced with the help of the processing capability of the cloud server 200, and save the user's time.
  • S803 The terminal 100 sends the first image and the first environment information of the first image to the cloud server 200.
  • the terminal 100 when the terminal 100 obtains the first image and the first environment information, the terminal 100 can detect the current network status. If the network status is disconnected and cannot connect to the cloud server 200, the terminal 100 can output a prompt to prompt the user to The cloud server 200 can be used to enhance the first image after connecting to the network.
  • the cloud server 200 can be used to enhance the first image after connecting to the network.
  • the terminal 100 determines a first image processing model matching the first environment information from the image processing model library according to the first environment information.
  • the image processing model library stores well-trained image processing models corresponding to various shooting environments.
  • the shooting environment can be distinguished by one or more of geographic location information, weather information, season information, time information, and so on.
  • the image processing model library may be divided into multiple image processing models according to the shooting location and shooting weather.
  • the image processing models corresponding to multiple shooting environments stored in the image processing model library can be shown in Table 1 below:
  • the shooting environment where the shooting location is "Shanghai ⁇ The Bund” and the shooting weather is “sunny” corresponds to the image processing model 1.
  • the shooting environment where the shooting location is “Shanghai ⁇ The Bund” and the shooting weather is “cloudy” corresponds to image processing model 2.
  • the shooting environment where the shooting location is “Shanghai ⁇ The Bund” and the shooting weather is “other weather” corresponds to image processing model 3.
  • the shooting environment where the shooting location is "Beijing ⁇ Forbidden City” and the shooting weather is “sunny” corresponds to image processing model 4.
  • the shooting environment where the shooting location is "Beijing ⁇ Forbidden City” and the shooting weather is "snow” corresponds to the image processing model 5.
  • the shooting environment where the shooting location is "Beijing ⁇ Forbidden City” and the shooting weather is “other weather” corresponds to the image processing model 6, and so on.
  • Table 1 above are only used to explain this application and should not constitute a limitation.
  • the cloud server 200 determines from the image processing model library that the first image processing model associated with the first environmental information may be “image processing model 2”.
  • the cloud server 200 sends the first image processing model to the terminal 100.
  • the cloud server 200 may send the first image processing model to the terminal 100 after determining the first image processing model matching the first environment information.
  • S806 The terminal 100 optimizes the first image according to the first image processing model to obtain the second image.
  • the terminal 100 can input the first image into the first image processing model, and the second image can be obtained after the first image processing model is processed.
  • the second image has improved detail definition, dynamic range/contrast, noise, and color of the second image.
  • the cloud server 200 retrieves the first high-definition image from the high-definition image library according to the first image and the first environment information. Wherein, the similarity between the second environment information of the first high-definition image and the first environment information of the first image is within a first preset range, and the shooting content in the first high-definition image is the same as the shooting in the first image. The similarity between the contents is within the second preset range.
  • the high-definition image library stores multiple high-definition images and second environment information of the high-definition images.
  • the cloud server 200 may first select the similarity between the second environment information and the first environment information from the high-definition image library according to the first environment information. The first preset range of high-definition images. Then, the cloud server 200 may compare the first image with the high-definition image similar to the shooting environment, and filter out the high-definition images with the similar shooting environment that have the similarity with the shooting content in the first image within the second preset range. The first high-definition image.
  • the high-definition images stored in the high-definition image library and the second environment information of the high-definition images may be as shown in Figure 2 below:
  • the shooting location of HD image 1 in the HD image library is “Shanghai ⁇ The Bund", the weather is “sunny”, and the shooting time is "afternoon”.
  • the shooting location of HD image 2 is “Shanghai ⁇ The Bund”, the weather is “cloudy”, and the shooting time is "morning”.
  • the shooting location of HD image 3 is “Shanghai ⁇ The Bund”, the weather is “cloudy”, and the shooting time is “morning”.
  • the shooting location of HD image 4 is “Shanghai ⁇ The Bund", the weather is “cloudy”, and the shooting time is “evening".
  • the shooting location of HD image 5 is "Beijing ⁇ Forbidden City", the weather is “sunny”, and the shooting time is “morning”.
  • the shooting location of HD image 6 is "Beijing ⁇ Forbidden City", the weather is "light rain”, and the shooting time is "afternoon”. and many more.
  • Table 2 above are only used to explain the application and should not constitute a limitation.
  • the shooting location in the first environment information of the first image may be “Shanghai ⁇ The Bund”, the weather may be “cloudy”, and the shooting time may be “morning”.
  • the shooting location of the high-definition image 2 is “Shanghai ⁇ The Bund”
  • the weather is “cloudy”
  • the shooting time is “morning”.
  • the shooting location of HD image 3 is "Shanghai ⁇ The Bund”
  • the weather is “cloudy”
  • the shooting time is “morning”. Therefore, the cloud server 200 can determine that the high-definition images that are the same as the first image shooting environment include the high-definition image 2 and the high-definition image 3.
  • the cloud server 200 may filter out the first high-definition image whose shooting content is within a preset range with the shooting content in the first image from the high-definition image 2 and the high-definition image 3. For example, the similarity between the shooting content in the high-definition image 2 and the shooting content in the first image is 90%, the similarity between the shooting content in the high-definition image 3 and the shooting content in the first image is 40%, and the preset range is 80%. Therefore, the cloud server 200 may determine the high-definition image 3 as the first high-definition image.
  • the above examples are only used to explain the application and should not constitute a limitation.
  • the cloud server 200 performs image registration on the first high-definition image according to the first image to obtain a second high-definition image.
  • the cloud server 200 may extract the matching feature points in the first image and the first high-definition image, and then obtain the image space coordinate transformation parameters by matching the feature point pairs. Then, the cloud server 200 may perform image registration according to the image space coordinate transformation parameter.
  • the cloud server 200 may send the second high-definition picture to the terminal 100.
  • the terminal 100 may perform image fusion between the second image and the second high-definition image to obtain a third image.
  • the terminal 100 may compare the feature information in the second image and the second high-definition image through an image fusion algorithm, and fuse the feature information with certain advantages in the second high-definition image
  • a third image that is, an enhanced image of the first image
  • the high-definition details and rich color feature points in the second high-definition image can be merged into the second image to obtain the third image. In this way, you can improve the details of the image clarity, color, white balance and so on.
  • the cloud server 200 may include a 3D world model and a rendering engine. After the cloud server 200 receives the first image and the first environment information sent by the terminal 100, it can determine the shooting environment of the first image from the 3D world model according to the first environment information, and use the rendering engine to render the 3D world model. A high-definition image that is the same as the shooting environment of the first image, and the rendered high-definition image is used as the first high-definition image for subsequent image registration and image fusion after image registration. In another possible implementation manner, the cloud server 200 may also pre-render high-definition images of each shooting angle in each geographic area in the 3D world model.
  • the cloud server 200 When the cloud server 200 receives the first image and the first image sent by the terminal 100 After a piece of environmental information, the cloud server 200 can find a high-definition image that is consistent with the shooting angle of the first image from among the rendered multiple high-definition images according to the first environmental information, and determine it as the first high-definition image described above. For subsequent image registration and image fusion after image registration.
  • the cloud server 200 may not send the first image processing model to the terminal 100, Instead, the high-definition image is optimized directly according to the first image processing model to obtain the second image.
  • the cloud server 200 retrieves the first high-definition image from the high-definition image library according to the first image and the first environment information, and performs image registration on the first high-definition image, after obtaining the second high-definition image, there is no need to send the second high-definition image to Terminal 100.
  • the cloud server 200 may perform image fusion between the second image and the second high-definition image to obtain a third image (that is, an image enhanced by the first image), and then send the third image to the terminal 100.
  • a third image that is, an image enhanced by the first image
  • the computing resources of the terminal 100 can be greatly saved, so that the terminal 100 can break through the terminal's own imaging quality limit, so that the ordinary camera on the terminal can also take pictures of professional SLR equipment.
  • the terminal 100 may only send the first environment information of the first image to the cloud server 200 after responding to the user's first input.
  • the cloud server 200 may determine the first image processing model matching the first environment information from the image processing model library, and send the first image processing model to the terminal 100.
  • the terminal 100 may optimize the first image through the first image processing model to obtain the second image, and save it locally as an enhanced image.
  • the terminal 100 may send the first image and the first environment information of the first image to the cloud server 200 in response to the first input of the user.
  • the cloud server 200 After the cloud server 200 receives the first image and the first environment information sent by the terminal 100, it can determine the first image processing model matching the first environment information from the image processing model library, and use the first image processing model to compare The first image is optimized to obtain the second image.
  • the cloud server 200 may send the second image to the terminal 100, and save it locally in the terminal 100 as an enhanced image.
  • the terminal 100 may send the first image and the first environment information of the first image to the cloud server 200 after responding to the user's first input.
  • the cloud server 200 may retrieve the first high-definition image from the high-definition image library according to the first environment information, and perform image registration on the first high-definition image according to the first image to obtain the second high-definition image.
  • the cloud service 200 may send the second high-definition image to the terminal 100, and the terminal 100 performs image fusion of the second high-definition image and the first image, and the enhanced image is saved locally in the terminal 100.
  • the cloud server 200 may also directly perform image fusion with the first image after obtaining the second high-definition image, and send the image after the image fusion to the terminal 100.
  • the embodiment of the present application provides an image processing method, which can combine the processing capabilities of the terminal and the cloud server, make full use of high-quality image data and image processing algorithms, and empower the terminal photographing and imaging system.
  • the cloud server can use the image uploaded by the terminal and the environmental information of the image (such as geographic location information, weather information, season information, time information, shooting parameters, shooting posture information, etc.) to match the image processing algorithm suitable for the image and High-quality image data with the same shooting content and similar shooting environment.
  • the image processing algorithm and the high-quality image data the quality and effect of the image uploaded by the terminal can be enhanced. In this way, pictures taken in different environments can show different styles and effects.
  • This also enables the terminal to break through the terminal's own imaging quality limit, so that the ordinary camera on the terminal can also take pictures of professional SLR equipment, thereby greatly improving the terminal's camera quality and user's camera experience while reducing the cost of mobile phones.
  • FIG. 9 shows a schematic structural diagram of an image processing system 900 provided in an embodiment of the present application.
  • the image processing system 900 may include a terminal 100 and a cloud server 200.
  • the terminal 100 may include a camera 911, an image signal processor 912, a homology processing module 913, and a heterogeneous fusion module 914.
  • the camera 911 can be used to capture light signals when the user takes pictures, convert the light signals into electrical signals, and send them to the image signal processor 912.
  • the image signal processor 912 can convert the electrical signal sent by the camera 911 into a digital image signal, also called a first image, where the format of the first image can be a Raw image, a YUV image, an RGB image, and so on.
  • the image signal processor 912 may also obtain the first environmental information (including geographic location information, weather information, etc.) of the first image from one or more sensors (not shown in the figure) of the terminal 100 and the network connected to the terminal 100. , Season information, time information, shooting parameters, shooting posture information, etc.).
  • the homology processing module 913 can be used to optimize the first image generated by the image processor 912 according to the first image processing model sent by the cloud server 200 to obtain the second image.
  • the heterogeneous fusion module 914 may be used to perform image fusion on the second image processed by the homologous processing module 913 and the second high-definition image sent by the cloud server 200 to obtain a third image.
  • the cloud server 200 may include an image processing model library 921, a model selection module 922, a high-definition image library 923, an image retrieval module 924, and an image registration module 925.
  • the image processing model library 921 may be used to store multiple image processing models classified according to the shooting environment.
  • the model selection module 922 can be used to determine the first image processing model matching the data 1 from the image processing model library 921 according to the data 1 in the first environment information sent by the terminal 100.
  • the data 1 may be all or part of the information in the first environmental information.
  • the first environment information includes geographic location information, weather information, season information, time information, shooting parameters, and shooting posture information
  • data 1 may include geographic location information and weather information.
  • the high-definition image library 923 can be used to store multiple high-definition images and second environment information of the high-definition images.
  • the image retrieval module 924 can be used to retrieve the first high-definition image from the high-definition image library 923 according to the data 2 in the first image and the first environment information.
  • the similarity between the shooting environment of the first high-definition image and the shooting environment of the first image is within a first preset range
  • the similarity between the shooting content in the first high-definition image and the shooting content in the first image is within Within the second preset range.
  • the data 2 may be all or part of the information in the first environmental information.
  • the first environment information includes geographic location information, weather information, season information, time information, shooting parameters, and shooting posture information
  • data 2 may include geographic location information, season information, time information, and shooting posture information.
  • the image registration module 925 can be used to perform image registration on the first high-definition image according to the first image to obtain a second high-definition image.
  • the cloud server 200 may send the second high-definition image to the terminal 100.
  • an image processing system 900 which can combine the processing capabilities of the terminal and the cloud server, make full use of high-quality image data and image processing algorithms, and empower the terminal photographing and imaging system.
  • the cloud server can use the image uploaded by the terminal and the environmental information of the image (such as geographic location information, weather information, time information, shooting parameters, shooting posture information, etc.) to match the image processing algorithm suitable for the image and the shooting content of the image Same, high-quality image data with similar shooting environment.
  • the image processing algorithm and the high-quality image data the quality and effect of the image uploaded by the terminal can be enhanced. In this way, pictures taken in different environments can show different styles and effects.
  • This also enables the terminal to break through the terminal's own imaging quality limit, so that the ordinary camera on the terminal can also take pictures of professional SLR equipment, thereby greatly improving the terminal's camera quality and user's camera experience while reducing the cost of mobile phones.
  • FIG. 10 shows a schematic structural diagram of an image processing system 1000 provided in an embodiment of the present application.
  • the image processing system 1000 may include a terminal 100 and a cloud server 200.
  • the terminal 100 may include a camera 1011, an image signal processor 1012, a homology processing module 1013, and a heterogeneous fusion module 1014.
  • the camera 1011 can be used to capture light signals when the user takes pictures, convert the light signals into electrical signals, and send them to the image signal processor 1012.
  • the image signal processor 1012 can convert the electrical signal sent by the camera 1011 into a digital image signal, also called a first image, where the format of the first image can be a Raw image, a YUV image, an RGB image, or the like.
  • the image signal processor 1012 can also obtain the first environmental information (including geographic location information, weather information, etc.) of the first image from one or more sensors (not shown in the figure) of the terminal 100 and the network connected to the terminal 100. , Season information, time information, shooting parameters, shooting posture information, etc.).
  • the homology processing module 1013 can be used to optimize the first image generated by the image processor 1012 according to the first image processing model sent by the cloud server 200 to obtain the second image.
  • the heterogeneous fusion module 1014 can be used to perform image fusion on the second image processed by the homologous processing module 1013 and the second high-definition image sent by the cloud server 200 to obtain a third image.
  • the cloud server 200 may include an image processing model library 1021, a model selection module 1022, a 3D world model 1023, a rendering engine 1024, and an image registration module 1025.
  • the image processing model library 1021 may be used to store multiple image processing models classified according to the shooting environment.
  • the model selection module 1022 can be used to determine the first image processing model matching the data 1 from the image processing model library 1021 according to the data 1 of the first environment information sent by the terminal 100.
  • the data 1 may be all or part of the information in the first environmental information.
  • the first environment information includes geographic location information, weather information, season information, time information, shooting parameters, and shooting posture information
  • data 1 may include geographic location information and weather information.
  • the 3D world model 1023 includes 3D map information of the real world.
  • the rendering engine 1024 can be used to render the first high-definition image from the 3D world model 1023 according to the data 2 in the first environment information of the first image.
  • the similarity between the shooting content in the first high-definition image and the shooting content in the first image is within a preset range, and the shooting environment of the first high-definition image is the same as the shooting environment of the first image.
  • the data 2 may be all or part of the information in the first environmental information.
  • the first environment information includes geographic location information, weather information, season information, time information, shooting parameters, and shooting posture information
  • the data 2 may include geographic location information, season information, time information, and shooting posture information.
  • the image registration module 1025 can be used to perform image registration on the first high-definition image according to the first image to obtain a second high-definition image.
  • the cloud server 200 may send the second high-definition image to the terminal 100.
  • an image processing system 1000 which can combine the processing capabilities of a terminal and a cloud server, make full use of high-quality image data and image processing algorithms, and empower the terminal photographing and imaging system.
  • the cloud server can use the image uploaded by the terminal and the environmental information of the image (such as geographic location information, weather information, time information, shooting parameters, shooting posture information, etc.) to match the image processing algorithm suitable for the image and the shooting content of the image Same, high-quality image data with similar shooting environment.
  • the image processing algorithm and the high-quality image data the quality and effect of the image uploaded by the terminal can be enhanced. In this way, pictures taken in different environments can show different styles and effects. This also enables the terminal to break through the terminal's own imaging quality limit, so that the ordinary camera on the terminal can also take pictures of professional SLR equipment, thereby greatly improving the terminal's camera quality and user's camera experience while reducing the cost of mobile phones.
  • FIG. 11 shows a schematic diagram of the architecture of an image processing system 1100 provided in an embodiment of the present application.
  • the image processing system 1100 may include a terminal 100 and a cloud server 200.
  • the terminal 100 may include a camera 1111 and an image signal processor 1112.
  • the camera 1111 can be used to capture light signals when the user takes pictures, convert the light signals into electrical signals, and send them to the image signal processor 1112.
  • the image signal processor 1112 can convert the electrical signal sent by the camera 1111 into a digital image signal, also called a first image, where the format of the first image can be a Raw image, a YUV image, an RGB image, and so on.
  • the image signal processor 1112 may also obtain the first environmental information (including geographic location information, weather information) of the first image from one or more sensors (not shown in the figure) of the terminal 100 and the network connected to the terminal 100. , Season information, time information, shooting parameters, shooting posture information, etc.).
  • the cloud server 200 may include an image processing model library 1121, a model selection module 1122, a high-definition image library 1123, an image retrieval module 1124, an image registration module 1125, a homology processing module 1126, and a heterogeneous fusion module 1127.
  • the image processing model library 1121 can be used to store multiple image processing models classified according to the shooting environment.
  • the model selection module 1122 can be used to determine the first image processing model matching the data 1 from the image processing model library 1121 according to the data 1 in the first environment information sent by the terminal 100.
  • the data 1 may be all or part of the information in the first environmental information.
  • the first environment information includes geographic location information, weather information, season information, time information, shooting parameters, and shooting posture information
  • data 1 may include geographic location information and weather information.
  • the high-definition image library 1123 can be used to store multiple high-definition images and second environment information of the high-definition images.
  • the image retrieval module 1124 can be used to retrieve the first high-definition image from the high-definition image library 1123 according to the first image and data 2 in the first environment information of the first image.
  • the similarity between the shooting environment of the first high-definition image and the shooting environment of the first image is within a first preset range
  • the shooting content in the first high-definition image is similar to the shooting content in the first image
  • the degree is within the preset range.
  • data 2 can be all or part of the first environmental information.
  • the first environment information includes geographic location information, weather information, season information, time information, shooting parameters, and shooting posture information
  • data 2 may include geographic location information, season information, time information, and shooting posture information.
  • the image registration module 1125 can be used to perform image registration on the first high-definition image according to the first image to obtain a second high-definition image.
  • the homology processing module 1126 can be used to optimize the first image sent by the terminal 100 according to the first image processing model selected by the model selection module 1122 to obtain the second image.
  • the heterogeneous fusion module 1127 can be used to perform image fusion on the second image and the second high-definition image processed by the homology processing module 1126 to obtain a third image.
  • the cloud server 200 may send the third image to the terminal 100.
  • an image processing system 1100 which can combine the processing capabilities of the terminal and the cloud server, make full use of high-quality image data and image processing algorithms, and empower the terminal photographing and imaging system.
  • the cloud server can use the image uploaded by the terminal and the environmental information of the image (such as geographic location information, weather information, time information, shooting parameters, shooting posture information, etc.) to match the image processing algorithm suitable for the image and the shooting content of the image Same, high-quality image data with similar shooting environment.
  • the image processing algorithm and the high-quality image data the quality and effect of the image uploaded by the terminal can be enhanced. In this way, pictures taken in different environments can show different styles and effects.
  • This also enables the terminal to break through the terminal's own imaging quality limit, so that the ordinary camera on the terminal can also take pictures of professional SLR equipment, thereby greatly improving the terminal's camera quality and user's camera experience while reducing the cost of mobile phones.
  • FIG. 12 shows a schematic structural diagram of an image processing system 1200 provided in an embodiment of the present application.
  • the image processing system 1200 may include a terminal 100 and a cloud server 200.
  • the terminal 100 may include a camera 1211, an image signal processor 1212.
  • the camera 1211 can be used to capture light signals when the user takes pictures, convert the light signals into electrical signals, and send them to the image signal processor 1212.
  • the image signal processor 1212 can convert the electrical signal sent by the camera 1211 into a digital image signal, also called a first image, where the format of the first image can be a Raw image, a YUV image, an RGB image, or the like.
  • the image signal processor 1212 may also obtain the first environmental information (including geographic location information, weather information, etc.) of the first image from one or more sensors (not shown in the figure) of the terminal 100 and the network connected to the terminal 100. , Season information, time information, shooting parameters, shooting posture information, etc.).
  • the cloud server 200 may include an image processing model library 1221, a model selection module 1222, a 3D world model 1223, a rendering engine 1224, an image registration module 1225, a homologous processing module 1226, and a heterogeneous fusion module 1227.
  • the image processing model library 1221 can be used to store multiple image processing models classified according to the shooting environment.
  • the model selection module 1222 can be used to determine the first image processing model matching the data 1 in the first environment information from the image processing model library 1221 according to the data 1 in the first environment information sent by the terminal 100.
  • the data 1 may be all or part of the information in the first environmental information.
  • the first environment information includes geographic location information, weather information, season information, time information, shooting parameters, and shooting posture information
  • data 1 may include geographic location information and weather information.
  • the 3D world model 1223 includes 3D map information of the real world.
  • the rendering engine 1224 can be used to render the first high-definition image from the 3D world model 1223 according to the data 2 of the first environment information of the first image.
  • the similarity between the shooting content in the first high-definition image and the shooting content in the first image is within a preset range, and the shooting environment of the first high-definition image is the same as the shooting environment of the first image.
  • the data 2 may be all or part of the information in the first environmental information.
  • the first environment information includes geographic location information, weather information, season information, time information, shooting parameters, and shooting posture information
  • data 2 may include geographic location information, season information, time information, and shooting posture information.
  • the image registration module 1225 can be used to perform image registration on the first high-definition image according to the first image to obtain a second high-definition image.
  • the homology processing module 1226 can be used to optimize the first image sent by the terminal 100 according to the first image processing model selected by the model selection module 1222 to obtain the second image.
  • the heterogeneous fusion module 1227 can be used to perform image fusion on the second image and the second high-definition image processed by the homology processing module 1226 to obtain a third image.
  • the cloud server 200 may send the third image to the terminal 100.
  • an image processing system 1200 which can combine the processing capabilities of a terminal and a cloud server, make full use of high-quality image data and image processing algorithms, and empower the terminal photographing and imaging system.
  • the cloud server can use the image uploaded by the terminal and the environmental information of the image (such as geographic location information, weather information, time information, shooting parameters, shooting posture information, etc.) to match the image processing algorithm suitable for the image and the shooting content of the image Same, high-quality image data with similar shooting environment.
  • the image processing algorithm and the high-quality image data the quality and effect of the image uploaded by the terminal can be enhanced. In this way, pictures taken in different environments can show different styles and effects. This also enables the terminal to break through the terminal's own imaging quality limit, so that the ordinary camera on the terminal can also take pictures of professional SLR equipment, thereby greatly improving the terminal's camera quality and user's camera experience while reducing the cost of mobile phones.
  • FIG. 13 shows a schematic diagram of the architecture of a data construction system 1300 provided in an embodiment of the present application.
  • the data construction system 1300 may include a data quality assessment module 1301, a high-definition image library 1302, an image database to be enhanced 1303, an image retrieval module 1304, an image registration module 1305, an image pairing module 1306, and a verification module 1307 , Image processing model library 1308.
  • the data quality evaluation module 1301 can be used to evaluate the image quality of images uploaded by professional photographers or photography enthusiasts, remove low-quality images, for example, images with low resolution or incomplete environmental information, and filter out high-definition images and their environment The information is uploaded to the high-definition image library 1302.
  • the image database 1303 to be enhanced may store multiple images to be enhanced and their environment information.
  • the image retrieval module 1304 can be used to retrieve the image to be enhanced and its environment information from the image database 1303 to be enhanced, and retrieve from the high-definition image data 1302 a high-definition image with the same environment information as the image to be enhanced and similar shooting content.
  • the image retrieval module 1305 can be used to perform image registration on the high-definition image according to the image to be enhanced to obtain a high-definition training image.
  • the image pairing module 1306 can be used to bind the image to be enhanced and the high-definition training image into a training image pair.
  • the verification module 1307 can be used to filter out image pairs with large alignment deviations, and to classify image pairs with small alignment deviations according to the environment information to classify the corresponding training set category for image processing in the image processing model library 1308 The model is trained.
  • a data construction system 1300 which can optimize and update the image processing model library in the cloud server 200, and improve the training efficiency of the image processing model.
  • FIG. 14 is a schematic structural diagram of a cloud server 200 according to an embodiment of the present application.
  • the cloud server 200 is applied to the image processing system 900 described in FIG. 9 or the image processing system 1000 described in FIG. 10 or the image processing system 1100 described in FIG. 11 or the image processing system 1200 described in FIG. 12.
  • the cloud server 200 includes one or more processors 201A, a communication interface 202A, and a memory 203A.
  • the processor 201A, the communication interface 202A, and the memory 203A can be connected by a bus or other methods. Take the bus 204 A connection as an example. among them:
  • the processor 201A may be composed of one or more general-purpose processors, such as a CPU.
  • the processor 201A may be used to run related program codes of the device control method.
  • the communication interface 202A may be a wired interface (for example, an Ethernet interface) or a wireless interface (for example, a cellular network interface or using a wireless local area network interface) for communicating with other nodes. In the embodiment of the present application, the communication interface 202A may be specifically used to communicate with the electronic device 100.
  • the memory 203A may include volatile memory, such as RAM; the memory may also include non-volatile memory, such as ROM, flash memory, HDD, or solid-state drive SSD; 203A may also include a combination of the above-mentioned types of memories.
  • the memory 203A may be used to store a group of program codes, so that the processor 201A can call the program codes stored in the memory 203A to implement the implementation method on the server in the embodiment of the present application.
  • server shown in FIG. 14 is only an implementation manner of the embodiment of the present application. In actual applications, the server may also include more or fewer components, which is not limited here.
  • FIG. 15 is a schematic flowchart of the image processing method provided in this embodiment of the application.
  • the terminal 100 sends the image A and the shooting information of the image A to the cloud server 200.
  • the cloud server 200 determines the reference image 1 from the high-definition image library.
  • the reference image 1 is not only similar to the shooting information of the image A, but also the shooting content is consistent with the shooting content of the image A.
  • the cloud server 200 performs detail restoration processing on the image A according to the reference image 1 to obtain the image C, and returns the image C to the terminal 100.
  • the image processing method includes but is not limited to the following steps:
  • S1501 The terminal 100 detects a photographing instruction.
  • the GPS of the terminal 100 can be turned on, and the terminal 100 can be connected to the network.
  • the terminal 100 may install an application program (application, APP) for uploading the image A and the shooting information of the image A.
  • the application program may include, but is not limited to, a photographing application, a photo editing application, and so on.
  • the application on the terminal 100 can also log in to an account.
  • the user can initiate a photographing instruction through a touch screen operation, or the user can initiate a photographing instruction through a button.
  • the terminal 100 detects the photographing instruction, and photographs the image A.
  • the terminal 100 can also obtain the geographic location information of the image A through GPS.
  • the terminal 100 may also obtain the weather condition information of the photographed geographic location information as the photographed weather condition information of the image A through the network.
  • the terminal 100 may also obtain the shooting magnification of the image A.
  • S1502 The terminal 100 captures the image A.
  • S1503 The terminal 100 sends the image A and the shooting information of the image A to the cloud server.
  • the terminal 100 may send the photographing information of the image A and the image A to the cloud server through the photographing application or the image retouching application, and the cloud server corresponds to the photographing application or the image retouching application.
  • the cloud server can be configured with a high-definition image library, the high-definition image library stores multiple high-definition images, the shooting information of each of the multiple high-definition images, and the first high-definition image corresponding to each of the multiple high-definition images
  • One feature and second feature can be included.
  • the first feature can be extracted through a search network, and the second feature can be extracted through a reordering network.
  • the reordering network can be a feature extraction network that is more complex than the retrieval network. Therefore, the second feature can be extracted through the reordering network.
  • the feature is more refined than the first feature extracted through the retrieval network, that is, the second feature can express more detailed features of the image.
  • the first feature extracted by the search network can be used for a rough quick search
  • the second feature extracted by the reordering network can be further used for a more accurate search to determine the reference image with the most similar shooting content. 1.
  • a high-definition camera device is used to collect multiple high-definition images, so that the collected multiple high-definition images form a high-definition image library.
  • the geographical area may be divided into large geographical areas. For example, Shenzhen, Guangzhou, Chongqing, etc. may be regarded as large geographical areas.
  • the retrieval network is used to extract the first feature corresponding to each high-definition image in the multiple high-definition images. For example, each high-definition image in the multiple high-definition images can be input into the retrieval network separately to obtain the corresponding high-definition image. The first feature.
  • the reordering network is used to extract the second feature corresponding to each of the multiple high-definition images. For example, each high-definition image of the multiple high-definition images can be input into the reordering network separately to obtain each high-definition image. The second feature corresponding to the image.
  • the shooting information of each high-definition image includes but not limited to shooting geographic location information, shooting weather condition information, shooting magnification, etc.
  • the plurality of high-definition images and the plurality of high-definition images respectively correspond to the first feature and The second feature is classified and stored.
  • the shooting weather condition information may include whether it was sunny, cloudy, foggy, rainy, etc. when the high-definition image was shot.
  • the multiple high-definition images can be divided into multiple large categories according to the geographic area where each high-definition image was taken. For example, all the high-definition images belonging to Shenzhen are divided into one large category, and the geographic area All HD images belonging to Guangzhou are divided into another big category, and so on. It is understandable that the division of geographical areas may also be other, for example, it may be divided according to the name of the scenic spot, which is not limited in the embodiment of the present application.
  • all high-definition images belonging to the same geographic area are further classified.
  • multiple high-definition pictures with the same or similar shooting magnification, the same or similar GPS information, and the same weather condition information at the time of shooting may be classified into the same category.
  • the weather condition information is not limited in the embodiment of this application.
  • the above-mentioned shooting magnifications being the same or similar may mean that the absolute value of the difference of the shooting magnifications is less than or equal to the set magnification threshold.
  • the foregoing GPS information being identical or similar may mean that the distance between geographic locations indicated by the GPS information may be less than or equal to a set distance threshold.
  • the GPS information in the embodiments of the present application refers to the GPS information of the geographic location of a photographing device (such as a terminal 100 or a high-definition photographing device deployed in multiple geographic regions) when an image is taken.
  • a photographing device such as a terminal 100 or a high-definition photographing device deployed in multiple geographic regions
  • the first feature and the second feature corresponding to the high-definition image can be stored together, which is convenient for subsequent retrieval and use. It should be noted that, in this embodiment of the application, the resolution of the high-definition image stored in the high-definition image library is greater than the resolution of the image A captured by the terminal 100.
  • the cloud server 200 determines at least one image B according to the shooting information of the image A.
  • the cloud server 200 receives the image A sent by the terminal 100 and the shooting information of the image A, and determines at least one image B from the high-definition image library according to the shooting information of the image A.
  • the degree of matching between the shooting information of each image B in the at least one image B and the shooting information of the image A is greater than the first threshold.
  • the shooting information may include one or more of the following information: shooting magnification, shooting geographic location information, or shooting weather condition information, etc.
  • the shooting information includes shooting geographic location information
  • the degree of matching between the shooting geographic location information of each image B in the at least one image B and the shooting geographic location information of image A is greater than a first threshold, for example, shooting The distance between the geographic location of the image B and the geographic location of the image A is less than or equal to the distance threshold.
  • the matching degree between the shooting magnification of each image B in the at least one image B and the shooting magnification of the image A is greater than a first threshold, for example, the shooting magnification of the image B
  • the absolute value of the difference with the shooting magnification of the image A is less than or equal to the magnification threshold.
  • the degree of matching between the shooting weather condition information of each image B in the at least one image B and the shooting weather condition information of image A is greater than a first threshold, for example, both It was shot on a sunny day, or all on a rainy day, or all on a foggy day.
  • a first threshold for example, both It was shot on a sunny day, or all on a rainy day, or all on a foggy day.
  • the high-definition images in the high-definition image library may be classified and stored according to shooting information.
  • the cloud server 200 determines at least one image B according to the shooting information of the image A, it may determine the corresponding category according to the shooting information of the image A, so as to obtain the at least one image B from the determined category.
  • the shooting information of image A includes shooting geographical location information
  • the geographical area to which the shooting geographical location information belongs is determined according to the shooting geographical location information, and then at least one image with a high degree of matching shooting information is determined from the high-definition images belonging to the geographical area.
  • the high-definition images belonging to the geographic area can also be classified and stored according to the shooting information, for example, multiple high-definition pictures with the same or similar shooting magnification, the same or similar GPS information, and the same weather condition information at the time of shooting are grouped into multiple high-definition images. Is the same category, and the shooting magnification, GPS information, and weather condition information are used as classification tags.
  • the cloud server 200 may determine the corresponding image according to the matching between the shooting information of image A and each classification tag.
  • the category of the at least one image B is determined from the category, and the first feature and the second feature corresponding to each image B in the stored at least one image B are obtained.
  • the cloud server 200 determines, from the at least one image B, the reference image 1 whose shooting content is most similar to the shooting content of the image A.
  • the cloud server 200 may extract the first feature corresponding to the image A through the retrieval network, and calculate the first feature corresponding to the image A and the first feature corresponding to each image B in the at least one image B. The similarity between the M images B with the largest similarity is obtained. It is understandable that the first feature corresponding to each image B may be extracted in advance through the retrieval network and stored in the high-definition image library. For details, please refer to the description of the foregoing embodiment, which will not be repeated here.
  • the cloud server 200 may sort the similarity between the first feature corresponding to the image A and the first feature corresponding to each image B in the at least one image B in descending order, and The images B corresponding to the M similarities ranked at the top are regarded as the M images B with the largest similarity.
  • the cloud server 200 may further extract the second feature corresponding to the image A through the reordering network , And respectively calculate the similarity between the second feature corresponding to the image A and the second feature corresponding to each image B in the M images B to obtain the image B with the largest similarity as the reference image 1.
  • the second feature corresponding to each image B may be extracted in advance through the reordering network and stored in the high-definition image library. For details, please refer to the description of the foregoing embodiment, which will not be repeated here.
  • the cloud server 200 may reorder the similarity between the second feature corresponding to the image A and the second feature corresponding to each image B in the M images B in descending order, And the image B corresponding to the similarity ranked at the top is used as the reference image 1.
  • the second feature since the second feature is more refined than the first feature, the second feature can express more detailed features of the image. Therefore, the similarity ranking result based on the first feature may be different from the similarity ranking result based on the second feature. It can be understood that the similarity of the first feature can be matched to obtain approximately similar M images B, and the similarity of the second feature can be matched to obtain the most similar reference image 1.
  • the reference image 1 not only has a high degree of matching between the shooting information and the shooting information of the image A, but also the shooting content is very close. Therefore, through the method of the embodiment of the present application, the shooting angle, the shooting content, and the transparency of the picture can be obtained.
  • the reference image 1 with a higher resolution can better restore the details of the image A with a lower resolution.
  • S1506 The cloud server 200 performs detail restoration processing on the image A according to the reference image 1, and obtains the image C.
  • the resolution of image A is relatively low, and correspondingly, the resolution is relatively low.
  • the cloud server 200 performs detail restoration processing on the image A according to the reference image 1 to obtain the image C.
  • the resolution of the image C is greater than the resolution of the image A.
  • the definition of the image C is greater than that of the image A.
  • the reference image 1 and image A can be input to the deep learning neural network, the high frequency information of the reference image 1 can be extracted through the deep learning neural network, and the details of the image A can be restored through the high frequency information, so as to obtain the sharpness Higher image C.
  • the ratio of the shooting content of reference image 1 to the shooting content of image A can also be determined, where the ratio can reflect the retrieved reference image 1 Whether the included shooting content covers the shooting content included in the full image A.
  • the shooting content matching the shooting content of the reference image 1 is removed from the shooting content of the image A, and the removed image A is obtained.
  • the second threshold can be set according to actual needs. For example, the second threshold can be set to 80%. If the ratio of the shooting content of the reference image 1 to the shooting content of the image A is less than 80%, the reference image 1 is described. The included shooting content does not cover the shooting content included in the full image A, and the shooting content included in the image A can be further searched for shooting content other than the shooting content included in the reference image 1, that is, the removed image A.
  • the reference image 2 whose shooting content is most similar to the shooting content of the removed image A is determined from at least one image B whose shooting information is highly matched.
  • the method for determining the reference image 2 whose shooting content is most similar to the shooting content of the removed image A from at least one image B can refer to the method for determining the reference image 1 in the foregoing embodiment, which will not be repeated here.
  • the ratio of the shooting content of the reference image 1 and the shooting content of the reference image 2 to the shooting content of the image A can be further determined, if the ratio is still less than the second threshold , It is possible to further remove the shooting content of the reference image 1 and the shooting content matching the shooting content of the reference image 2 from the shooting content of the image A to obtain the removed image A, and then further retrieve it from at least one image B The reference image 3 that is most similar to the shooting content of the removed image A.
  • the ratio of the shooting content of the at least one reference image to the shooting content of the image A can be understood as that the intersection of the shooting content of the at least one reference image and the shooting content of the image A accounts for the percentage of the shooting content of the image A. proportion.
  • the cloud server 200 sends the image C to the terminal 100.
  • the terminal 100 receives the image C and displays the image C.
  • the terminal 100 may display the image A and the image C, where the shooting content of the image C and the image A are the same, but the definition of the image C is higher.
  • the terminal only needs to send the captured image A with a relatively low resolution to the cloud server, and the cloud server retrieves the reference image with highly matching shooting information and highly similar shooting content, and the cloud server Perform detail restoration processing on image A according to the reference image to obtain image C and return it to the terminal.
  • the configuration requirements for the terminal can be reduced, and the compatibility is strong.
  • FIG. 16 is another schematic flowchart of the image processing method provided by this embodiment of the application.
  • the terminal 100 sends the shooting information of image A to the cloud server 200, and the cloud server 200
  • the shooting information of the image A determines at least one image B from the high-definition image library, and the degree of matching between the shooting information of the image B and the shooting information of the image A is greater than the first threshold.
  • the cloud server 200 sends the at least one image B to the terminal 100, and the terminal 100 determines from the at least one image B the reference image 1 whose shooting content is most similar to the shooting content of the image A.
  • the image processing method includes but is not limited to the following steps:
  • S1601 The terminal 100 detects a photographing instruction.
  • S1602 The terminal 100 captures the image A.
  • steps S1601-step S1602 please refer to steps S1501-S1502 in the embodiment of FIG. 15, which will not be repeated here.
  • S1603 The terminal 100 sends the shooting information of the image A to the cloud server 200.
  • the terminal 100 may send the photographing information of the image A to a cloud server through a photographing application or a picture retouching application, and the cloud server corresponds to the photographing application or a picture retouching application.
  • the cloud server may be configured with a high-definition image library, and the high-definition image library stores multiple high-definition images and shooting information of each of the multiple high-definition images.
  • a high-definition camera device is used to collect multiple high-definition images, so that the collected multiple high-definition images form a high-definition image library.
  • the geographical area may be divided into large geographical areas. For example, Shenzhen, Guangzhou, Chongqing, etc. may be regarded as large geographical areas.
  • the shooting information of each high-definition image includes but not limited to shooting geographic location information, shooting weather condition information, shooting magnification, etc.
  • the shooting weather condition information may include whether it was sunny, cloudy, foggy, rainy, etc. when the high-definition image was shot.
  • the multiple high-definition images can be divided into multiple large categories according to the geographic area where each high-definition image was taken. For example, all the high-definition images belonging to Shenzhen are divided into one large category, and the geographic area All HD images belonging to Guangzhou are divided into another big category, and so on. It is understandable that the division of geographical areas may also be other, for example, it may be divided according to the name of the scenic spot, which is not limited in the embodiment of the present application.
  • all high-definition images belonging to the same geographic area are further classified.
  • multiple high-definition pictures with the same or similar shooting magnification, the same or similar GPS information, and the same weather condition information at the time of shooting may be classified into the same category.
  • the weather condition information is not limited in the embodiment of this application.
  • the above-mentioned shooting magnifications being the same or similar may mean that the absolute value of the difference of the shooting magnifications is less than or equal to the set magnification threshold.
  • the foregoing GPS information being identical or similar may mean that the distance between geographic locations indicated by the GPS information may be less than or equal to a set distance threshold.
  • the GPS information in the embodiments of the present application refers to the GPS information of the geographic location of a photographing device (such as a terminal 100 or a high-definition photographing device deployed in multiple geographic regions) when an image is taken.
  • a photographing device such as a terminal 100 or a high-definition photographing device deployed in multiple geographic regions
  • the resolution of the high-definition image stored in the high-definition image library is greater than the resolution of the image A captured by the terminal 100.
  • the cloud server 200 determines at least one image B according to the shooting information of the image A.
  • the cloud server 200 may determine at least one image B whose photographing information is highly matched with the photographing information of the image A. For a specific determination method, refer to step S1504 of the embodiment of FIG. 15, and details are not described herein again.
  • the cloud server 200 sends the at least one image B to the terminal 100.
  • the terminal 100 determines, from the at least one image B, the reference image 1 whose shooting content is most similar to the shooting content of the image A.
  • the terminal 100 receives at least one image B sent by the cloud server 200.
  • the terminal 100 may use a retrieval network to separately extract the first feature corresponding to each image B in the at least one image B and the first feature corresponding to the image A.
  • each image in the at least one image B may be B is input into the retrieval network respectively, so that the first feature corresponding to each image B is obtained respectively.
  • the image A is input into the retrieval network to obtain the first feature corresponding to the image A.
  • the terminal 100 respectively calculates the similarity between the first feature corresponding to the image A and the first feature corresponding to each image B in the at least one image B to obtain M images B with the greatest similarity.
  • the cloud server 200 may sort the similarity between the first feature corresponding to the image A and the first feature corresponding to each image B in the at least one image B in descending order, and The images B corresponding to the M similarities ranked at the top are regarded as the M images B with the largest similarity.
  • the terminal 100 may use a reordering network to extract each of the M images B separately
  • the second feature corresponding to B and the second feature corresponding to image A for example, each image B of the M images B can be input into the reordering network respectively, so as to obtain the second feature corresponding to each image B respectively.
  • the image A is input into the reordering network, so that the second feature corresponding to the image A is obtained.
  • the terminal 100 respectively calculates the similarity between the second feature corresponding to the image A and the second feature corresponding to each image B in the M images B, and obtains the image B with the largest similarity as the reference image 1.
  • the cloud server 200 may reorder the similarity between the second feature corresponding to the image A and the second feature corresponding to each image B in the M images B in descending order, And the image B corresponding to the similarity ranked at the top is used as the reference image 1.
  • S1607 The terminal 100 performs detail restoration processing on the image A according to the reference image 1, and obtains the image C.
  • step S1506 For the specific steps for the terminal 100 to perform detail restoration processing on the image A, refer to step S1506 in the embodiment of FIG. 15, which will not be repeated here.
  • step S1608 please refer to step S1608 in the embodiment of FIG. 15, which will not be repeated here.
  • the cloud server returns at least one image B with highly matching shooting information to the terminal, and the terminal extracts the features corresponding to the at least one image B and the features corresponding to the image A according to its own needs, without the need for the entire HD Feature extraction is performed on all images in the image library to avoid waste of processing resources.
  • FIG. 17 is another schematic flowchart of the image processing method provided by this embodiment of the present application.
  • the terminal 100 sends the image A and the shooting information of the image A to the cloud server 200, and the cloud The server 200 determines at least one image B from the high-definition image library according to the shooting information of the image A, and the degree of matching between the shooting information of the image B and the shooting information of the image A is greater than the first threshold.
  • the cloud server 200 further extracts the first feature and the second feature corresponding to the image A, and sends the first feature and the second feature corresponding to the image A and the first feature and the second feature corresponding to each image B to the terminal 100, and
  • the terminal 100 determines from the at least one image B the reference image 1 whose shooting content is most similar to the shooting content of the image A.
  • the image processing method includes but is not limited to the following steps:
  • S1701 The terminal 100 detects a photographing instruction.
  • S1702 The terminal 100 captures image A.
  • the terminal 100 sends the image A and the shooting information of the image A to the cloud server 200.
  • the cloud server 200 determines at least one image B according to the shooting information of the image A.
  • steps S1701-step S1704 please refer to step S1701-step S1704 in the embodiment of FIG. 17, which will not be repeated here.
  • the cloud server sends the first feature and the second feature corresponding to the image A and the first feature and the second feature corresponding to each image B to the terminal 100.
  • the cloud server 200 may extract the first feature corresponding to the image A through a retrieval network, and extract the second feature corresponding to the image A through a reordering network.
  • the cloud server 200 sends the first feature and the second feature corresponding to the image A and the first feature and the second feature corresponding to each image B in the at least one image B stored in the high-definition image library to the terminal 100.
  • the terminal 100 determines, from the at least one image B, the reference image 1 whose shooting content is most similar to the shooting content of the image A.
  • the terminal 100 determines the difference between the captured content and the image A according to the first feature and the second feature corresponding to the received image A and the first feature and the second feature corresponding to each image B in the at least one image B.
  • S1707 The terminal 100 performs detail restoration processing on the image A according to the reference image 1, and obtains the image C.
  • the terminal 100 may refer to step S1506 in the embodiment of FIG. 15 for the process of performing the detail restoration processing on the image A according to the reference image 1, which will not be repeated here.
  • step S1708 please refer to step S1508 of the embodiment in FIG. 15, which will not be repeated here.
  • the features corresponding to the image and the features corresponding to the stored image are extracted through the cloud server, and the terminal matches the features corresponding to the image A and the feature corresponding to at least one image B returned by the cloud server to obtain a reference Image 1, which can avoid occupying a large amount of memory of the terminal to store image features.
  • FIG. 18 is a framework diagram of an image processing solution provided by an embodiment of the present application; as shown in the figure, the first feature is extracted offline for the high-definition image in the high-definition image library of the cloud server by searching the network.
  • the cloud server will not only extract the first feature (ie feature vector) corresponding to the image taken by the mobile phone online through the search network, but also based on the shooting information of the image taken by the mobile phone (including real-time GPS information, weather information, shooting magnification, etc.), from The high-definition image library is screened to obtain a feature vector set, and the degree of matching between the shooting information of the high-definition image in the feature vector set and the shooting information of the image captured by the mobile phone is greater than the first threshold.
  • extracting the first feature online may refer to a network connection between a mobile phone and a cloud server, and obtaining an image taken by the mobile phone through the network connection, and extracting the first feature of the image.
  • the M images with the greatest similarity are obtained, for example, if M is 15, the 15 high-definition images with the greatest similarity are obtained. image.
  • the 15 high-definition images with the greatest similarity are extracted through a reordering network to extract the second feature corresponding to each high-definition image, and the second feature corresponding to the image taken by the mobile phone is extracted through the reordering network. Then, according to the similarity measure between the second feature corresponding to each of the 15 high-definition images and the second feature corresponding to the image taken by the mobile phone, the reference image with the greatest similarity is obtained.
  • the reference image may be a high-definition image that is highly consistent with the image taken by the mobile phone at a shooting angle, has a similar shooting magnification, and is consistent with the shooting weather condition information.
  • the details of the image taken by the mobile phone are restored, thereby obtaining an image with higher definition than the image taken by the mobile phone.
  • FIG. 19 is a schematic structural diagram of an image processing apparatus provided by an embodiment of this application.
  • the image processing apparatus may include an acquisition module 1901, a first determination module 1902, and a second determination module 1903.
  • the acquiring module 1901 is used to acquire the photographed image A and the photographing information of the image A;
  • the first determining module 1902 is configured to determine at least one image B according to the shooting information of the image A, wherein the degree of matching between the shooting information of the image B and the shooting information of the image A is greater than a first threshold, The resolution of the image B is greater than the resolution of the image A;
  • the second determining module 1903 is configured to determine, from the at least one image B, the reference image 1 whose shooting content is most similar to the shooting content of the image A.
  • the shooting information includes one or more of the following information: shooting magnification, shooting geographic location information, or shooting weather condition information.
  • the image processing apparatus may further include:
  • a restoration processing module configured to perform detail restoration processing on the image A according to the reference image 1 to obtain an image C, the resolution of the image C is greater than the resolution of the image A;
  • the output module is used to output the image C.
  • the second determining module 1903 may include:
  • An acquiring unit configured to respectively acquire a first feature corresponding to each image B in the at least one image B and a first feature corresponding to the image A, wherein one image B corresponds to one first feature;
  • the first determining unit is configured to determine the shooting from the at least one image B according to the similarity between the first feature corresponding to each image B in the at least one image B and the first feature corresponding to the image A M images B whose content is most similar to the shooting content of the image A, where M is an integer greater than or equal to 1;
  • the second determining unit is configured to determine the reference image 1 whose shooting content is most similar to the shooting content of the image A from the M images B.
  • the second determining unit is specifically configured to:
  • the second feature is more refined than the first feature.
  • the image processing apparatus may further include:
  • the third determining module is configured to determine the ratio of the shooting content of the reference image 1 to the shooting content of the image A;
  • a removal processing module configured to remove, from the shooting content of the image A, the shooting content that matches the shooting content of the reference image 1 from the shooting content of the image A to obtain the removed image A if the ratio is less than a second threshold;
  • a fourth determining module configured to determine, from the at least one image B, the reference image 2 whose shooting content is most similar to the shooting content of the removed image A;
  • the restoration processing module is specifically configured to perform detail restoration processing on the image A according to the reference image 1 and the reference image 2 to obtain the image C.
  • each of the above-mentioned modules and/or units may be software, hardware, or a combination of software and hardware.
  • the terminal 100 in order to obtain a larger field of view (filedofview, FOV) when the terminal 100 is taking pictures or recording videos, it usually uses a lower zoom magnification for shooting.
  • the terminal 100 may use a wide-angle lens or an ultra-wide-angle lens for shooting, thereby reducing the zoom magnification and increasing the FOV of the shooting image.
  • the angle of view of the shooting screen is negatively related to the quality of the shooting screen. Relying on the camera device on the terminal 100 alone, it is impossible to have both the large field of view and the high-definition picture quality of the shooting picture.
  • a wide-angle camera to take a photo or video
  • the field of view of the shot picture is larger, its image quality is blurry, and the detail texture of the photographed object is not clear, especially the detail texture of the distant view is more blurry.
  • the embodiment of the present application provides an image processing method that uses the high-definition image database on the cloud server 200 to send low-definition images with low zoom magnification (that is, large field of view) shot on the terminal 100 to the cloud.
  • Server 200 The cloud server 200 cuts the low-definition image with a low zoom magnification based on the zoom magnification of the high-definition image in the high-definition image library to obtain multiple image blocks.
  • the cloud server can search and match in the high-definition image library to obtain one or more high-definition reference images, and then through the image enhancement network based on the high-definition reference image to improve the image quality, obtain multiple high-definition image blocks .
  • the cloud server splices these multiple high-definition image blocks into an ultra-clear image with a larger resolution and higher definition, and returns it to the terminal 100.
  • the processing capability of the cloud server 200 can be used to enable the terminal 100 to capture a picture or video when the terminal 100 not only has a wide field of view, but also has the rich detailed texture and sharpness that can be captured at a high-power zoom.
  • FIG. 20 shows a schematic flowchart of another image processing method provided in an embodiment of the present application.
  • the method may include but is not limited to the following steps:
  • the terminal 100 detects a photographing instruction.
  • the terminal 100 may install an application (application, APP) for uploading low-definition images.
  • the application program may include, but is not limited to, a photographing application, a photo editing application, and so on.
  • the application on the terminal 100 can also log in to the account.
  • the user can initiate a photographing instruction through a touch screen operation, or the user can initiate a photographing instruction through a button.
  • the terminal 100 detects a photographing instruction and photographs a low-definition image.
  • the terminal 100 After detecting the photographing instruction, the terminal 100 photographs a low-definition image and obtains shooting information of the low-definition image.
  • the shooting information of the low-definition image includes zoom magnification, and one or more of the following: geographic location information, weather information, season information, time information, shooting parameters (for example, aperture, exposure time, etc.), and shooting posture information.
  • the terminal 100 may capture the low-definition image at a zoom magnification of 1x.
  • the smaller the zoom magnification the larger the field of view of the low-definition image, the less detail in the low-definition image, and the more blurred the picture in the low-definition image.
  • the terminal 100 sends the low-definition image and the shooting information of the low-definition image to the cloud server 200.
  • the terminal 100 may automatically transmit the low-definition image and the shooting information of the low-definition image to the cloud server 200 after shooting the low-definition image.
  • the terminal 100 may also upload the low-definition image to the cloud server 200 by receiving the user's operation after shooting the low-definition image.
  • the terminal 100 may also upload the low-definition image saved in the gallery to the cloud server 200.
  • the cloud server 200 recognizes the semantic information of the low-resolution image.
  • the cloud server 200 After the cloud server 200 receives the low-definition image sent by the terminal 100, it uses the semantic segmentation model to identify regions where different types of image content in the low-definition image are located, and obtains a semantic segmentation mask map of the low-definition image.
  • the semantic segmentation mask map can mark the regions where different types of image content in the low-definition image are located. Among them, the cloud server 200 can mark the category of the image content in the low-definition image.
  • the category of the image content in the low-definition image may include sky, building, ground, and so on.
  • the cloud server 200 can identify the regions of the sky, the building, and the ground in the low-definition image through the semantic segmentation model, and obtain the semantic segmentation mask map.
  • the semantic segmentation mask image includes a region 2101, a region 2102, and a region 2103.
  • the area 2101 is used to indicate the area where the sky is in the low-resolution image
  • the area 2102 is used to indicate the area where the building is in the low-resolution image
  • the area 2103 is used to indicate the area where the ground is in the low-resolution image.
  • the cloud server 200 divides the low-resolution image into N image blocks with a specified magnification based on the semantic information and the zoom magnification information of the high-definition image library.
  • a high-definition image library may be configured in the cloud server 200, and the high-definition image library stores multiple high-definition images and shooting information of each of the multiple high-definition images.
  • the shooting information of the high-definition image may include one or more of the following: zoom magnification, geographic location information, weather information, season information, time information, shooting parameters (for example, aperture, exposure time, etc.), shooting posture information, and the like.
  • all the high-definition image libraries in the high-definition image library may be at a fixed zoom magnification (for example, a zoom magnification of 10 times).
  • the cloud server 200 may determine the number of image blocks based on the zoom magnification of the high-definition image in the high-definition image library and the zoom magnification of the low-definition image. Among them, the zoom magnification of the high-definition image is greater than or equal to the zoom magnification of the low-definition image. The greater the difference between the zoom magnification of the high-definition image and the zoom magnification of the low-definition image, the greater the number of image blocks.
  • the zoom magnification of each image block is corresponding or similar to the zoom magnification of the high-definition image or meets the preset ratio relationship.
  • the zoom magnification of the high-definition image may be 10 times the zoom magnification
  • the zoom magnification of the low-definition image may be 1 times the zoom magnification
  • the cloud server 200 may divide the low-definition image into a window of 10 times the zoom magnification.
  • the image is divided into N image blocks, where the size of each image block is the same as the window size of the 10x zoom magnification.
  • the low-definition image can be divided into N image blocks.
  • FIG. 21C only exemplarily shows some of the N image blocks.
  • the image block 2131 may be specifically as shown in FIG. 21D.
  • the cloud server 200 may also divide the voice segmentation mask image of the low-definition image with the same window size to obtain a mask block corresponding to each image block.
  • the cloud server 200 can mark the category of each image block.
  • the cloud server 200 retrieves T high-definition retrieval images of each image block from the high-definition image library based on semantic information, and T is a positive integer.
  • the semantic information may include the category of each image block in the low-definition image.
  • the cloud server 200 may adopt different retrieval strategies for different types of image blocks.
  • the categories of image blocks may include sky, building, and ground. Since the texture of the sky region and the texture of the ground region are relatively simple, the cloud server 200 may not perform image retrieval or detail restoration for the image blocks classified as “sky” and “ground”. Since the texture details of the building area are rich, the cloud server 200 can retrieve the high-definition retrieval image corresponding to the image block of the “building” category for the image block of the “building” category.
  • the cloud server 200 can retrieve T high-definition search images corresponding to the image block from the high-definition image library according to the shooting information of the low-definition image (also referred to as first environment information in other embodiments) and the image block.
  • the cloud server 200 determines at least one first matching image from the high-definition image library based on the shooting information of the low-definition image, where the shooting information of the first matching image is the same as the shooting information of the low-definition image.
  • the matching degree between the two is greater than the first threshold, the resolution of the first matching image is greater than the resolution of the low-definition image; the cloud server separately obtains the first feature and the first feature corresponding to each first matching image in the at least one first matching image.
  • the first feature corresponding to the type of image block, where one first matching image corresponds to one first feature.
  • the cloud server determines from the at least one first matching image that the shooting content and the first feature are the same. T high-definition search images with the most similar shooting content of a type of image block.
  • the cloud server determines based on the similarity between the first feature corresponding to each first matching image in the at least one first matching image and the first feature corresponding to the first type of image block At least one second matching image that is most similar to the image block of the first type in the first feature.
  • the cloud server separately obtains the second feature corresponding to each second matching image and the second feature corresponding to the image block of the first type in the at least one second matching image, where one second matching image corresponds to one second feature.
  • the cloud server determines that the second feature is similar to the first type The most similar T high-definition retrieved images of the image block. Among them, the second feature is more refined than the first feature.
  • the cloud server 200 performs matching and registration on the T high-definition retrieval images of each image block based on the semantic information and the low-definition image to obtain T high-definition registered images.
  • the cloud server 200 may use semantic information as a priori guidance information, and adopt different matching and registration strategies for different types of image blocks.
  • the categories of image blocks may include sky, building, and ground. Since the texture of the sky area and the texture of the ground area are relatively simple, the cloud server 200 may not perform image matching registration or detail restoration for image blocks classified as “sky” and “ground”. Due to the rich texture details of the building area, for the image block of the category "building", the cloud server 200 can perform image registration on the high-definition image after retrieving T high-definition retrieval images corresponding to the image block of the "building" category. , Get T high-definition registration images.
  • the cloud server 200 repairs the details of each image block based on the semantic information and the T high-definition registered images of each image block to obtain N high-definition image blocks.
  • the process of the cloud server 200 performing image registration for the T high-definition retrieval images corresponding to the image block may be as follows:
  • the cloud server 200 may determine the type of the image block based on the semantic segmentation mask of the low-definition image, and then determine whether the image block needs to be registered. If image registration is required, the cloud server 200 may use a traditional matching algorithm to obtain the feature points matching the image block and each high-definition retrieval image. Then, the cloud server 200 can obtain the image space coordinate transformation parameters between the image block and each high-definition retrieval image through the matching feature point pairs between the image block and each high-definition retrieval image. The cloud server 200 may perform image registration on each high-definition retrieval image according to the image space coordinate transformation parameters to obtain T high-definition registered images.
  • the cloud server will match the features in each high-definition retrieval image with the first type of image block Point, adjusted to the same position as in the image block, to obtain T high-definition registration images.
  • the traditional matching algorithm may include, but is not limited to, any one of the following two types of algorithms: an algorithm based on gray-level template matching and an algorithm based on feature matching.
  • algorithms based on gray-level template matching include mean absolute differences (MAD), sum of absolute differences (SAD), sum of squared differences (SSD), average Mean Square Differences (MSD), Normalized Cross Correlation (NCC), Sequential Similarity Detection Algorithm (SSDA), Hadamard Transformation Algorithm (sum of absolute transformed difference) , SATD), local gray value coding algorithm, etc.
  • Algorithms based on feature matching may include, but are not limited to, one or more of the following: FAST algorithm, ORB algorithm, SIFT algorithm, SURF algorithm, LBP algorithm, traditional optical flow method, etc.
  • the cloud server 200 may filter out the optimal L high-definition reference images from the T high-definition registered images based on the optimization strategy, where: L is a positive integer and L is less than or equal to T.
  • the optimal selection strategy may be to select L high-definition reference images with the highest matching degree with the image block from the T high-definition registered images.
  • the optimal selection strategy may be to select L high-definition reference images that are most similar to the image space positions of the matching feature point pairs in the image block and the T high-definition registered images.
  • the cloud server 200 may use semantic information as a priori guidance information, and adopt different detail restoration strategies for different types of image blocks.
  • the categories of image blocks may include sky, building, and ground. Since the texture of the sky area and the texture of the ground area are relatively simple, the cloud server 200 may not perform image matching registration or detail restoration for image blocks classified as “sky” and “ground”. Since the texture details of the building area are rich, for the image block of the category "building", the cloud server 200 may perform detail restoration on the image block after obtaining L high-definition reference images of the image block.
  • the process of the cloud server 200 using L high-definition registration images to repair the details of the image block may be as follows:
  • the cloud server 200 may divide the semantic segmentation mask image based on the low-definition image to determine the category of the image block, and then determine whether the image block needs to be detailed. If image detail restoration is required, the cloud server 200 may input L high-definition reference images and image blocks into the superdivision network algorithm model after L high-definition reference images, and use the detailed texture information in the L high-definition reference images , Repair the details in the image block, improve the resolution in the image block, and obtain a high-definition image block.
  • the super-division network algorithm model may include a matching degree detection network, a deep registration network, and a deep fusion network.
  • the matching degree detection network can be used to detect areas in the high-definition reference image that are inconsistent with the content in the image block.
  • the deep registration network can be used for pixel-level registration of areas in the high-definition reference image that are consistent with the content in the image block.
  • the deep fusion network can be used to fuse high-frequency details in the high-definition reference image to the image block, improve the resolution of the image block, and obtain a high-definition image block.
  • the cloud server 200 may go through the image registration process shown in FIG. 22 to obtain the high-definition reference image 2331 corresponding to the image block 2131.
  • the cloud server 200 can input the above-mentioned image block 2131 and the high-definition reference image 2331 into the hyperdivision network algorithm model, and merge the high-frequency detail information in the high-definition reference image 2331 into the image block 2131, to obtain the image shown in FIG. 23C HD image block 2332.
  • the cloud server 200 stitches N high-definition image blocks to obtain a high-definition output image.
  • the cloud server 200 may stitch the high-definition image block corresponding to each image block according to the position of the image block in the low-definition image to obtain a high-definition output image.
  • the cloud server 200 may directly use the image blocks that have not undergone detail restoration and other high-definition image blocks obtained after detail restoration for splicing to obtain a high-definition output image.
  • the cloud server 200 does not perform detail restoration on the image blocks of the “sky” category and the “ground” category in the low-definition image, but performs detail restoration on the image blocks of the “building” category in the low-definition image. Therefore, the cloud server 200 may use the image blocks of the “sky” category and the “ground” category in the low-definition image to stitch the high-definition image blocks corresponding to the image blocks of the “building” category to obtain a high-definition output image.
  • the cloud server 200 sends the high-definition output image to the terminal 100.
  • the terminal 100 displays a high-definition output image.
  • the high-definition output image may also be referred to as the third image or image C in the foregoing embodiment.
  • the cloud server 200 may first use the shooting information of the low-definition image (also referred to as the first image or image A in the foregoing embodiment) before step S2004 (in the foregoing embodiment). (Referred to as the first environment information) to match the first image processing model, perform optimization processing on the low-definition image, and sequentially perform subsequent steps S2004 to S2009 on the optimized low-definition image.
  • the shooting information of the low-definition image also referred to as the first image or image A in the foregoing embodiment
  • the first environment information to match the first image processing model, perform optimization processing on the low-definition image, and sequentially perform subsequent steps S2004 to S2009 on the optimized low-definition image.
  • the image processing method provided by the embodiments of this application, it is possible to divide the image with a smaller zoom magnification according to the zoom magnification of the cloud high-definition image library, zoom in to a certain scale, and search and match in the cloud high-definition image library. , And then through the image enhancement network based on the reference image to improve the image quality, the final result is then stitched into a larger resolution, higher definition ultra-definition image. In this way, with the help of semantic information, first local enhancement and then overall splicing and fusion are obtained to obtain a higher-resolution ultra-clear image, which effectively solves the misdetection and mismatch of areas lacking feature information.
  • FIG. 24 shows a schematic flowchart of an image processing method provided in an embodiment of the present application.
  • a high-definition image library may be arranged on the cloud server 200, where multiple high-definition images and shooting information of the high-definition images may be stored in the high-definition image library (which may also be referred to as second environment information in the above embodiment),
  • the shooting information of the high-definition image may include zoom magnification, and one or more of the following: geographic location information, weather information, season information, time information, shooting parameters (for example, aperture, exposure time, etc.), and shooting posture information Wait.
  • the construction device can obtain multiple high-definition images with high zoom magnification for the same subject in the same geographic location.
  • the construction device can associate multiple high-resolution images with high zoom magnification according to the image content. Sexually stitched into a panoramic high-definition picture. Then, the construction device can cut the panoramic high-definition image according to the cutting size corresponding to the multiple zoom magnifications, so that the panoramic high-definition image is divided into multiple high-definition images at each type of zoom magnification. Among them, the larger the zoom magnification, the smaller the cutting size, and the higher the zoom magnification of the high-definition image obtained after cutting.
  • the constructing device may splice high-definition images with high zoom magnification into a panoramic high-definition image 2500.
  • the construction device can divide the panoramic image 2500 at 5 times zoom ratio, 10 times zoom ratio, 20 times zoom ratio and other zoom ratios, respectively, and divide into multiple high-definition images at each zoom ratio.
  • the size of each divided high-definition image may be as shown in the high-definition image 2511 in FIG. 25B.
  • FIG. 25A when the construction device divides the panoramic high-definition image at a zoom ratio of 10 times, the size of each divided high-definition image may be as shown in the high-definition image 2511 in FIG. 25B.
  • FIG. 25A when the construction device divides the panoramic high-definition image at a zoom ratio of 10 times, the size of each divided high-definition image may be as shown in the high-definition image 2511 in FIG. 25B.
  • the size of each divided high-definition image may be as shown in the high-definition image 2521 in FIG. 25D.
  • the larger the zoom magnification used when dividing the panoramic high-definition image 2500 the greater the number of high-definition images obtained by the division, and the less image information included in each high-definition image.
  • the above-mentioned Figs. 25A and 25C only show the cutting of the panoramic image.
  • the image processing method may include but is not limited to the following steps:
  • the terminal 100 detects a photographing instruction.
  • step S2001 in the embodiment shown in FIG. 20, which will not be repeated here.
  • the terminal 100 photographs a low-definition image.
  • step S2002 in the embodiment shown in FIG. 20, which will not be repeated here.
  • the terminal 100 sends the low-definition image and the shooting information of the low-definition image to the cloud server 200.
  • step S2003 in the embodiment shown in FIG. 20, which will not be repeated here.
  • S2404 The cloud server 200 recognizes the semantic information of the low-definition image.
  • step S2004 in the embodiment shown in FIG. 20, which will not be repeated here.
  • the cloud server 200 retrieves T high-definition retrieval images of the low-definition image from the high-definition image database.
  • the cloud server 200 may retrieve T high-definition search images similar to the low-definition image content from the high-definition image library based on the semantic information of the low-definition image.
  • the cloud server 200 may retrieve the low-definition image database from the high-definition image library based on the semantic information of the low-definition image and the shooting information of the low-definition image (in other embodiments, it may also be referred to as the first environment information). T high-definition search images with similar image content.
  • the cloud server 200 performing image retrieval for the image block, reference may be made to the embodiment shown in FIG. 18, which will not be repeated here.
  • the cloud server 200 Based on the semantic information and the low-definition image, the cloud server 200 registers T high-definition retrieval images of the low-definition image to obtain T high-definition registered images of the low-definition image.
  • the cloud server 200 repairs the low-definition image based on the semantic information and T high-definition registered images of the low-definition image to obtain a high-definition output image.
  • S2408 The cloud server 200 sends the high-definition output image to the terminal 100.
  • the terminal 100 displays a high-definition output image.
  • the high-definition output image may also be referred to as the third image or image C in the foregoing embodiment.
  • the cloud server 200 registers T high-definition retrieval images of low-definition images based on semantic information and low-definition images to obtain T high-definition registered images of low-definition images, and then can The T high-definition registration images are sent to the terminal 100.
  • the terminal 100 repairs the low-definition image based on the T high-definition registered images of the low-definition image to obtain a high-definition output image.
  • a large number of high-zoom high-definition images can be used to construct and stitch the panoramic images of the high-definition image library on the cloud server 200, and then the panoramic images are cut according to the image sizes of different zoom magnifications to obtain different zoom magnifications.
  • High-definition image The cloud server 200 can retrieve high-definition reference images with the same or similar zoom magnification from the low-definition image captured by the terminal 100 in the high-definition image library, which improves the effect of using the high-definition reference image to restore the details of the low-definition image.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Multimedia (AREA)
  • General Health & Medical Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Computing Systems (AREA)
  • Databases & Information Systems (AREA)
  • Evolutionary Computation (AREA)
  • Medical Informatics (AREA)
  • Software Systems (AREA)
  • Computational Linguistics (AREA)
  • Processing Or Creating Images (AREA)

Abstract

一种图像处理方法，涉及计算机视觉领域，方法包括：云服务器基于高清图像库中高清图像的变焦倍率对终端发送的低变焦倍率的低清图像进行切块，得到多个图像块；针对每一个图像块，云服务器都可以在高清图像库中进行检索匹配，得到一张或多张高清参考图，再经过基于高清参考图的图像增强网络进行图像质量提升，得到多张高清图像块；最后，云端的服务器再将这多张高清图像块拼接为一张分辨率更大，清晰度更高的超清图像，返回给终端。这样，可以借助云服务器的处理能力，使得终端拍摄照片或视频时的拍摄画面不仅具有较广视场角，还可以兼具高倍变焦时所能拍到的丰富细节纹理以及清晰度。

Description

一种图像处理方法及相关装置
本申请要求于2019年12月13日提交中国专利局、申请号为201911285717.4、申请名称为“一种图像处理方法及相关装置”，以及2020年04月29日提交中国专利局、申请号为202010357077.X、申请名称为“图像处理方法及装置”，以及2020年12月11日提交中国专利局、申请号为202011446454.3、申请名称为“一种图像处理方法及相关装置”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及计算机视觉领域，尤其涉及一种图像处理的方法及相关装置。
背景技术
智能手机发展至今，拍照已然成为其最为重要的特性之一。无论是基础成像器件还是成像算法，均产生了极大的发展和变化，一次又一次地推动了手机拍照的变革，提升用户拍照体验。然而，在这一次又一次的变革背后，器件的成本也越来越高，成像质量也越来越趋近瓶颈。
现有方案中，为了提升终端拍摄到照片的质量，通常终端在摄像头获取到拍摄画面后，会通过图像处理器(image signal processing，ISP)生成图像数据，并通过预设的算法对该图像数据进行处理优化，以提升图像效果。但是，终端利用固定的算法对图片进行处理，处理后的图像都是同一风格，并且由于终端能力的限制，处理后的图像质量并不高。而且不同的景物，呈现风格是不一样的。例如，北京故宫等名胜古迹呈现的是古风古韵的风格，上海外滩等都市景点，呈现的是大都市风格。同一景物不同的环境下，呈现的风格也是不一样的。例如，夏天的故宫和冬天的故宫，所呈现出的风格也是不一样的。对于摄影师来说，每一种风格都有不同的清晰度、色彩丰富度、对比度、白平衡、细节、纹理等等图像参数。现有方案中终端提升拍摄图片的质量的手段十分有限，成像质量不高。
发明内容
本申请提供了一种图像处理方法及相关装置，实现了通过云端服务器和终端结合，充分利用高质量图片数据和算法，赋能终端拍照成像系统，是终端能够突破终端本身成像质量极限，使得普通摄像头也能拍出媲美专业单反相机的效果，从而，在降低手机成本的同时，极大的提升终端拍摄照片的质量，以及用户拍照体验。
第一方面，本申请提供了一种图像处理方法，所述方法包括：云服务器获取终端发送的低清图像以及拍摄所述低清图像时的拍摄信息，所述拍摄信息包括所述低清图像的变焦倍率；所述云服务器识别所述低清图像的语义信息；所述云服务器基于高清图像库中高清图像的变焦倍率以及所述低清图像的变焦倍率，确定出图像块的数量N，所述N为正整数；所述云服务器基于图像块的数量N将所述低清图像分割成N个图像块，并基于所述低清图像的语义信息标记出所述N个图像块中每个图像块中图像内容的类型；所述云服务器从所述高清图像库中检索出与第一类型的图像块的拍摄内容匹配的T张高清检索图像，所述T为正整数；所述云服务器基于所述T张高清检索图像，对所述第一类型的图像块中的细节 进行修复，得到第一类型的高清图像块，所述第一类型的高清图像块的分辨率大于所述第一类型的图像块；所述云服务器基于第一类型的高清图像块替换所述低清图像中的第一类型的图像块，得到高清输出图，并将所述高清输出图发送给所述终端。
在一种可能的实现方式中，所述低清图像的拍摄信息还包括以下一种或多种：地理位置信息、天气信息、季节信息、时间信息、拍摄参数和拍摄姿态信息。
在一种可能的实现方式中，所述云服务器从所述高清图像库中检索出与第一类型的图像块的拍摄内容匹配的T张高清检索图像，具体包括：所述云服务器基于所述低清图像的拍摄信息，从所述高清图像库中确定出至少一个第一匹配图像，其中，所述第一匹配图像的拍摄信息与所述低清图像的拍摄信息之间的匹配度大于第一阈值，所述第一匹配图像的分辨率大于所述低清图像的分辨率；所述云服务器分别获取所述至少一个第一匹配图像中每个第一匹配图像对应的第一特征和所述第一类型的图像块对应的第一特征，其中，一个第一匹配图像对应一个第一特征；所述云服务器基于所述至少一个第一匹配图像中每个第一匹配图像对应的第一特征与所述第一图像块对应的第一特征之间的相似度，从所述至少一个第一匹配图像中确定拍摄内容与所述第一类型的图像块的拍摄内容最相似的T张高清检索图像。
在一种可能的实现方式中，所述云服务器基于所述至少一个第一匹配图像中每个第一匹配图像对应的第一特征与所述第一类型的图像块对应的第一特征之间的相似度，从所述至少一个第一匹配图像中确定拍摄内容与所述第一类型的图像块的拍摄内容最相似的T张高清检索图像，具体包括：所述云服务器基于所述至少一个第一匹配图像中每个第一匹配图像对应的第一特征与所述第一类型的图像块对应的第一特征之间的相似度，确定出在第一特征上与所述第一类型的图像块最相似的至少一张第二匹配图像；所述云服务器分别获取所述至少一个第二匹配图像中每个第二匹配图像对应的第二特征和所述第一类型的图像块对应的第二特征，其中，一个第二匹配图像对应一个第二特征；所述云服务器基于所述至少一个第二匹配图像中每个第二匹配图像对应的第二特征与所述第一类型的图像块对应的第二特征之间的相似度，确定出在第二特征上与所述第一类型的图像块最相似的T张高清检索图像；其中，所述第二特征比所述第一特征更加精细。
在一种可能的实现方式中，所述云服务器利用所述T张高清检索图像，对所述第一类型的图像块中的细节进行修复，得到第一类型的高清图像块，具体包括：所述云服务器获取所述第一类型的图像块与所述T张高清检索图像中每一张高清检索图像的图像空间坐标变换参数；所述云服务器基于所述第一类型的图像块与所述T张高清检索图像中每一张高清检索图像中匹配特征点对的图像空间坐标变换参数，将每一张高清检索图像中与所述第一类型的图像块中相匹配的特征点，调整到与所述第一类型的图像块中相同的位置，得到T张高清配准图像；所述云服务器基于所述T张高清配准图像对所述第一类型的图像块的细节进行修复，得到第一类型的高清图像块。
在一种可能的实现方式中，所述云服务器基于所述T张高清配准图像对所述第一类型的图像块的细节进行修复，得到第一类型的高清图像块，具体包括：所述云服务器基于所述第一类型的图像块与所述T张高清检索图像中每一张高清检索图像中匹配特征点对的图像空间坐标变换参数，从所述T张高清配准图像中确定出匹配特征点对的图像空间位置最 相似的L张高清参考图像，L为正整数；所述云服务器基于所述L张高清参考图像对所述第一类型的图像块的细节进行修复，得到所述第一类型的高清图像块。
在一种可能的实现方式中，所述云服务器基于第一类型的高清图像块替换所述低清图像中的第一类型的图像块，得到高清输出图，具体包括：当所述低清图像中只包括第一类型的图像块时，所述云服务器将所述第一类型的高清图像块拼接成所述高清输出图；当所述低清图像中包括第一类型的图像块和第二类型的图像块时，所述云服务器将所述第一类型的高清图像块与所述第二类型的高清图像块拼接成所述高清输出图。
在一种可能的实现方式中，在所述云服务器基于图像块的数量N将所述低清图像分割成N个图像块之前，所述方法还包括：所述云服务器根据所述低清图像的拍摄信息，确定出与所述环境信息对应的第一图像处理模型；所述云服务器通过所述第一处理模型对所述低清图像进行处理。所述云服务器基于图像块的数量N将所述低清图像分割成N个图像块，具体包括：所述云服务器基于图像块的数量N将经过所述第一图像处理模型处理后的所述低清图像分割成所述N个图像块。
在一种可能的实现方式中，所述云服务器通过所述第一处理模型对所述低清图像进行处理，具体包括：所述云服务器通过所述第一图像处理模型对所述低清图像的细节清晰度、和/或动态范围、和/或对比度、和/或噪声、和/或色彩进行处理。
本申请通过提供一种图像处理方法，借助云服务器上的高清图像数据库，将终端上拍摄的低变焦倍率(也即，大视场角)的低清图像，发送给云服务器。云服务器基于高清图像库中高清图像的变焦倍率对低变焦倍率的低清图像进行切块，得到多个图像块。针对每一个图像块，云服务器都可以在高清图像库中进行检索匹配，得到一张或多张高清参考图，再经过基于高清参考图的图像增强网络进行图像质量提升，得到多张高清图像块。最后，云端的服务器再将这多张高清图像块拼接为一张分辨率更大，清晰度更高的超清图像，返回给终端。这样，可以借助云服务器的处理能力，使得终端拍摄照片或视频时的拍摄画面不仅具有较广视场角，还可以兼具高倍变焦时所能拍到的丰富细节纹理以及清晰度。
第二方面，本申请提供了一种图像处理方法，应用于图像处理系统，该图像处理系统包括终端和云服务器，该方法包括：该终端接收用户的第一输入。响应于该第一输入，该终端获取第一图像以及拍摄该第一图像时的第一环境信息。该终端发送该第一环境信息给该云服务器。该云服务器根据该第一环境信息，确定与该环境信息对应的第一图像处理模型；该第一图像处理模型用于提高图像的质量。该云服务器发送该第一图像处理模型至该终端。该终端通过该第一图像处理模型对该第一图像进行处理，得到第二图像。
通过本申请提供一种图像处理方法，可以结合终端和云服务器的处理能力，充分利用云服务器上的高质量图像数据和图像处理算法，赋能终端拍照成像系统。云服务器可以通过终端上传的图像以及图像的环境信息(例如地理位置信息、天气信息、时间信息、拍摄参数、拍摄姿态信息等等)，匹配出适合该图像的图像处理算法以及与该图像拍摄内容相同，拍摄环境相近的高质量图像数据。通过该图像处理算法以及该高质量图像数据，可以增强终端上传图像的质量和效果。这样，可以让不同环境下拍摄的图片，呈现出不同的风格和效果。这样也使终端能够突破终端自身成像质量极限，使得终端上的普通摄像头也能拍出 专业单反设备的效果，从而在降低手机成本的同时，极大的提升终端的拍照质量和用户拍照体验。
在一种可能的实现方式中，在该终端接收用户的第一输入之前，该方法还包括：该终端显示拍摄界面，该拍摄界面用于显示预览图像；该拍摄界面包括拍摄控件。其中，该第一输入为用户针对该拍摄控件的操作，该第一图像为拍摄界面中的该预览图像。这样，可以在用户拍照时对终端拍摄的图像，借助云服务器的处理能力进行优化增强，提升终端的拍照时的成像质量。
在一种可能的实现方式中，在该终端接收用户的第一输入之前，该方法还包括：该终端显示图库应用中的云增强界面；该云增强界面显示包括有该第一图像对应的缩略图以及上传控件；其中，该第一输入为用户针对该上传控件的操作。述终端获取第一图像以及该第一图像的第一环境信息，具体包括：该终端从本地存储中获取该第一图像。该终端从该第一图像的可交换图像文件格式Exif数据中获取到该第一环境信息。这样，可以在用户在图库中选中想优化的图片后，借助云服务器的处理能力进行优化增强，提升终端上已保存图片的质量效果。
在一种可能的实现方式中，该方法还包括：该云服务器发送该第一图像给该云服务器。该云服务器根据该第一环境信息，确定出第一高清图像。其中，该第一高清图片的第二环境信息与该第一环境信息的相似度在第一预设范围内，且该第一高清图像中的拍摄内容与该第一图像中的拍摄内容之间的相似度在第二预设范围内。该云服务器将该第一图像和该第一高清图像进行图像配准，得到第二高清图像。该云服务器发送该第二高清图像给该终端。该终端对该第二高清图像和该第二图像进行图像融合，得到第三图像，该第三图像的清晰度高于该第二图像的清晰度。这样，可以针对特定的拍摄环境，使用特定的图像处理模型对第一图像进行优化，提升图像的质量。
在一种可能的实现方式中，该云服务器根据该第一环境信息，确定出第一高清图像，具体包括：该云服务器根据该第一环境信息，从高清图像库中检索出该第一高清图像。其中，该高清图像库中存储有多张高清图像和该多张高清图像各自对应的第二环境信息。
在一种可能的实现方式中，该云服务器根据该第一环境信息，确定出第一高清图像，具体包括：该云服务器根据该第一环境信息，通过渲染引擎从三维世界模型中渲染出该第一高清图像。
在一种可能的实现方式中，该终端对该第二高清图像和该第二图像进行图像融合，得到第三图像，具体包括：该终端提取该第二高清图像的特征信息，融合到该第二图像中，得到该第三图像；其中，该特征信息包括高清晰度的细节、高频信息。
在一种可能的实现方式中，该终端通过该第一图像处理模型对该第一图像的质量进行处理，得到第二图像，具体包括：该终端通过第一图像处理模型对该第一图像的细节清晰度、和/或动态范围、和/或对比度、和/或噪声、和/或色彩进行处理，得到该第二图像。
在一种可能的实现方式中，在该终端得到该第二图像后，该方法还包括：该终端显示该第二图像。
在一种可能的实现方式中，在该终端得到该第三图像后，该方法还包括：该终端显示该第三图像。
在一种可能的实现方式中，该第一环境信息包括以下一种或多种：地理位置信息、天气信息、季节信息、时间信息、拍摄姿态信息、拍摄参数等等。
第三方面，本申请提供了一种图像处理方法，应用于图像处理系统，该图像处理系统包括终端和云服务器。该包括：该终端接收用户的第一输入。响应于该第一输入，该终端获取第一图像以及该第一图像的第一环境信息。该终端发送该第一图像和该第一环境信息给该云服务器。该云服务器根据该第一环境信息和该第一图像，确定出第一高清图像。其中，该第一高清图像的环境信息与该第一环境信息之间的相似度在第一预设范围内，该第一高清图像中的拍摄内容与该第一图像中的拍摄内容之间的相似度在第二预设范围内。该云服务器根据该第一图像对该第一高清图像进行图像配准，得到第二高清图像。该云服务器发送该第二高清图像给该终端。该终端对该第二高清图像和该第一图像进行图像融合，得到第三图像，该第三图像的清晰度高于该第一图像的清晰度。
通过本申请提供一种图像处理方法，可以结合终端和云服务器的处理能力，充分利用云服务器上的高质量图像数据和图像处理算法，赋能终端拍照成像系统。云服务器可以通过终端上传的图像以及图像的环境信息(例如地理位置信息、天气信息、时间信息、拍摄参数、拍摄姿态信息等等)，匹配出适合该图像的图像处理算法以及与该图像拍摄内容相同，拍摄环境相近的高质量图像数据。通过该图像处理算法以及该高质量图像数据，可以增强终端上传图像的质量和效果。这样，可以让不同环境下拍摄的图片，呈现出不同的风格和效果。这样也使终端能够突破终端自身成像质量极限，使得终端上的普通摄像头也能拍出专业单反设备的效果，从而在降低手机成本的同时，极大的提升终端的拍照质量和用户拍照体验。
在一种可能的实现方式中，在该终端接收用户的第一输入之前，该方法还包括：该终端显示拍摄界面，该拍摄界面用于显示预览图像；该拍摄界面包括拍摄控件。其中，该第一输入为用户针对该拍摄控件的操作，该第一图像为拍摄界面中的该预览图像。这样，可以在用户拍照时对终端拍摄的图像，借助云服务器的处理能力进行优化增强，提升终端的拍照时的成像质量。
在一种可能的实现方式中，在该终端接收用户的第一输入之前，该方法还包括：该终端显示图库应用中的云增强界面；该云增强界面显示包括有该第一图像对应的缩略图以及上传控件；其中，该第一输入为用户针对该上传控件的操作。该终端获取第一图像以及该第一图像的第一环境信息，具体包括：该终端从本地存储中获取该第一图像。该终端从该第一图像的可交换图像文件格式Exif数据中获取到该第一环境信息。这样，可以在用户在图库中选中想优化的图片后，借助云服务器的处理能力进行优化增强，提升终端上已保存图片的质量效果。
在一种可能的实现方式中，该方法还包括：在该终端对该第二高清图像和该第一图像进行图像融合，得到第三图像之前，该方法还包括：该云服务器根据该第一环境信息，确定与该第一环境信息对应的第一图像处理模型。该第一图像处理模型用于提高图像的质量。该云服务器发送该第一图像处理模型给该终端。该终端通过该第一图像处理模型对该第一图像进行处理。该终端对该第二高清图像和该第一图像进行图像融合，得到第三图像，具 体包括：该终端对该第二高清图像和经过该第一图像处理模型处理后的该第一图像进行图像融合，得到该第三图像。这样，可以通过云服务器上相同拍摄环境下的高清图片，对第一图像进行优化，提升图像的细节清晰度等。
在一种可能的实现方式中，该云服务器根据该第一环境信息，确定出第一高清图像，具体包括：该云服务器根据该第一环境信息，从高清图像库中检索出该第一高清图像。其中，该高清图像库中存储有多张高清图像和该多张高清图像各自对应的第二环境信息。
在一种可能的实现方式中，该云服务器根据该第一环境信息，确定出第一高清图像，具体包括：该云服务器根据该第一环境信息，通过渲染引擎从三维世界模型中渲染出该第一高清图像。
在一种可能的实现方式中，该终端对该第二高清图像和该第二图像进行图像融合，得到第三图像，具体包括：该终端提取该第二高清图像的特征信息，融合到该第二图像中，得到该第三图像。其中，该特征信息包括高清晰度的细节。
在一种可能的实现方式中，该终端通过该第一图像处理模型对该第一图像进行处理，具体包括：该终端通过第一图像处理模型对该第一图像的细节清晰度、和/或动态范围、和/或对比度、和/或噪声、和/或色彩进行处理。
在一种可能的实现方式中，在该终端得到该第三图像后，该方法还包括：该终端显示该第三图像。
在一种可能的实现方式中，该第一环境信息包括以下一种或多种：地理位置信息、天气信息、季节信息、时间信息、拍摄姿态信息、拍摄参数等等。
第四方面，本申请提供了一种图像处理方法，应用于图像处理系统，该图像处理系统包括终端和云服务器。该方法包括：该终端接收用户的第一输入。响应于该第一输入，该终端获取第一图像以及该第一图像的第一环境信息。该终端发送该第一图像和该第一环境信息给该云服务器。该云服务器根据该第一环境信息，确定与该第一环境信息对应的第一图像处理模型。该第一图像处理模型用于提高图像的质量。该云服务器通过该第一图像处理模型对该第一图像的质量进行处理，得到第二图像。该云服务器发送该第二图像给该终端。
在一种可能的实现方式中，该方法还包括：该云服务器根据该第一环境信息，确定出第一高清图像；其中，该第一高清图片的第二环境信息与该第一环境信息在第二预设范围内，且该第一高清图像中的拍摄内容与该第一图像中的拍摄内容之间的相似度在第二预设范围内。该云服务器根据该第一图像对该第一高清图像进行图像配准，得到第二高清图像。该云服务器对该第二高清图像和该第二图像进行图像融合，得到第三图像。该云服务器将该第三图像发送给该终端。
第五方面，本申请提供了一种图像处理方法，应用于图像处理系统，该图像处理系统包括终端和云服务器。该方法包括：该终端接收用户的第一输入。响应于该第一输入，该终端获取第一图像以及该第一图像的第一环境信息。该终端发送该第一图像和该第一环境信息给该云服务器。该云服务器根据该第一环境信息，确定出第一高清图像。其中，该第 一高清图像的第二环境信息与该第一环境信息的相似度在第一预设范围内，且该第一高清图像中的拍摄内容与该第一图像中的拍摄内容之间的相似度在第二预设范围内。该云服务器根据该第一图像对该第一高清图像进行图像配准，得到第二高清图像。该云服务器对该第二高清图像和该第一图像进行图像融合，得到第三图像，该第三图像的清晰度高于该第一图像的清晰度。该云服务器发送该第三图像给该终端。
在一种可能的实现方式中，在该服务器对该第二高清图像和该第一图像进行图像融合，得到第三图像之前，该方法还包括：该云服务器根据该第一环境信息，确定与该第一环境信息对应的第一图像处理模型。该第一图像处理模型用于提高图像的质量。该云服务器通过该第一图像处理模型对该第一图像进行处理。
第六方面，本申请实施例提供一种图像处理方法，其中，该方法可以由终端或者云服务器执行，该图像处理方法可以包括：获取终端拍摄的图像A和该图像A的拍摄信息。可选的，若该方法是由云服务器执行，则终端可以将拍摄的图像A和该图像A的拍摄信息发送至云服务器，该云服务器接收图像A和该图像A的拍摄信息。
进一步，根据图像A的拍摄信息，确定至少一个图像B，其中，该至少一个图像B中的每个图像B的拍摄信息与图像A的拍摄信息之间的匹配度大于第一阈值。该至少一个图像B中的每个图像B的分辨率大于图像A的分辨率。例如，该至少一个图像B是高清图像，而图像A是低清图像。
可选的，可以是终端将图像A的拍摄信息发送给云服务器，云服务器从高清图像库中选择该至少一个图像B，并将该至少一个图像B发送给终端，终端从该至少一个图像B中确定拍摄内容与图像A的拍摄内容最相似的参考图像1。
或者，可以是终端将图像A和图像A的拍摄信息发送给云服务器，云服务器从高清图像库中选择该至少一个图像B，并从该至少一个图像B中确定拍摄内容与图像A的拍摄内容最相似的参考图像1。
实施本申请实施例，通过图像A的拍摄信息，确定至少一个图像B，再从至少一个图像B中确定拍摄内容与图像A的拍摄内容最相似的参考图像1，即可以先通过图像A的拍摄信息缩小检索范围，从而提高图像的检索效率。
结合第六方面，在一些实施例中，拍摄信息可以包括以下信息中的一种或者多种：拍摄倍率、拍摄地理位置信息或拍摄天气状况信息等。其中，拍摄倍率可以是放大倍率。拍摄地理位置信息可以是指拍摄该图像A或者图像B时，拍照设备所处的地理位置信息，地理位置信息可以通过全球定位系统(global positioning system，GPS)信息进行定义。拍摄天气状况信息可以是指拍摄该图像A或者图像B时的天气状况，例如，晴天、阴天、雨天等等。
可选的，在拍摄信息包括拍摄地理位置信息时，该至少一个图像B中每个图像B的拍摄地理位置信息与图像A的拍摄地理位置信息之间的匹配度大于第一阈值，通过地理位置信息的匹配，可以保证图像A和该至少一个图像B的拍摄地理位置的一致性。最终从该至少一个图像B中确定拍摄内容与图像A的拍摄内容最相似的参考图像1，该参考图像1不仅拍摄内容与图像A高度相似，而且拍摄角度(即拍摄地理位置信息)与图像A也高度一 致。
可选的，在拍摄信息包括拍摄倍率时，该至少一个图像B中每个图像B的拍摄倍率与图像A的拍摄倍率之间的匹配度大于第一阈值，可以保证图像A和该至少一个图像B的放大倍率相差不大。
可选的，在拍摄信息包括拍摄天气状况信息时，该至少一个图像B中每个图像B的拍摄天气状况信息与图像A的拍摄天气状况信息之间的匹配度大于第一阈值，例如，都是晴天拍摄的，或者都是雨天拍摄的，可以保证图像的透明度的一致性。
结合第六方面，在一些实施例中，根据参考图像1，对图像A进行细节恢复处理，获得图像C。其中，所获得的图像C的分辨率大于图像A的分辨率，即图像C的清晰度大于图像A的清晰度。
可选的，可以将参考图像1和图像A输入深度学习神经网络，通过深度学习神经网络提取该参考图像1的高频信息，并通过该高频信息对图像A进行细节恢复，从而得到清晰度更高的图像C。
结合第六方面，在一些实施例中，从至少一个图像B中确定拍摄内容与图像A的拍摄内容最相似的参考图像1，可以包括：可以分别获取该至少一个图像B中每个图像B对应的第一特征和该图像A对应的第一特征，其中，每个图像B对应一个第一特征。
其中，该至少一个图像B中的每个图像B对应的第一特征可以是通过检索网络提取的，该图像A对应的第一特征也可以是通过该检索网络提取的。第一特征包括但不限于CNN特征，相应的，检索网络可以是CNN网络。
计算该至少一个图像B中每个图像B对应的第一特征与图像A对应的第一特征之间的相似度，并从该至少一个图像B中确定拍摄内容与图像A的拍摄内容最相似的M个图像B，M为大于或者等于1的整数。例如，M可以为15，即从该至少一个图像B中选择与图像A的拍摄内容最相似的15个图像B。
其中，可以是将每个图像B对应的第一特征与图像A对应的第一特征之间的相似度进行排序，从而得到与图像A之间相似度最大的M个图像B。
进一步从M个图像B中确定拍摄内容与图像A的拍摄内容最相似的参考图像1，例如可以通过更加细节的特征进行匹配，从而得到参考图像1。
通过实施该实施例，通过逐渐缩减检索范围，得到与图像A的拍摄内容最相似的参考图像1，不仅提高检索效率，还能保证检索得到参考图像1与图像A的拍摄内容的一致性。
结合第六方面，在一些实施例中，从M个图像B中确定拍摄内容与图像A的拍摄内容最相似的参考图像1，包括：分别获取M个图像B中每个图像B对应的第二特征和图像A对应的第二特征，其中，每个图像B对应一个第二特征。
其中，该M个图像B中的每个图像B对应的第二特征可以是通过重排序网络提取的，该图像A对应的第二特征也可以是通过该重排序网络提取的。重排序网络可以是比检索网络更复杂的特征提取网络，通过重排序网络提取的第二特征可以是比检索网络提取的第一特征更加精细的图像特征，例如，第二特征比第一特征更加体现图像的细节特征。
计算该M个图像B中每个图像B对应的第二特征与图像A对应的第二特征之间的相似度，并从该M个图像B中确定拍摄内容与图像A的拍摄内容最相似的图像B，作为参考 图像1。
其中，可以是将每个图像B对应的第二特征与图像A对应的第二特征之间的相似度进行排序，从而将相似度最大的图像B作为参考图像1。
可以理解的是，将每个图像B对应的第一特征与图像A对应的第一特征之间的相似度进行排序的排序结果，可能与，将每个图像B对应的第二特征与图像A对应的第二特征之间的相似度进行排序的排序结果不同。
通过实施该实施例，由于检索网络比重排序网络简单，因此，检索效率也更高，本申请实施例中，先通过检索网络提取的第一特征进行匹配，从至少一个图像B中，得到与图像A大致相似的M个图像B，再通过重排序网络提取的第二特征进行匹配，从M个图像B中，得到最相似的参考图像1，可以在保证检索效率的前提下，提高检索的准确度。
结合第六方面，在一些实施例中，根据参考图像1，对图像A进行细节恢复处理之前，还包括：确定该参考图像1的拍摄内容占图像A的拍摄内容的比例，其中，该比例可以反映所检索的参考图像1包含的拍摄内容是否覆盖全图像A所包含的拍摄内容。
若参考图像1的拍摄内容占图像A的拍摄内容的比例小于第二阈值，从该图像A的拍摄内容中去除与参考图像1的拍摄内容匹配的拍摄内容，获得去除后的图像A。其中，该第二阈值可以根据实际需要进行设定，例如，该第二阈值可以设置为80％，若参考图像1的拍摄内容占图像A的拍摄内容的比例小于80％，则说明参考图像1包含的拍摄内容未覆盖全图像A所包含的拍摄内容，可以进一步检索图像A包含的拍摄内容中除参考图像1包含的拍摄内容以外的拍摄内容，即去除后的图像A。
进一步，从至少一个图像B中确定拍摄内容与去除后的图像A的拍摄内容最相似的参考图像2。
其中，从至少一个图像B中确定拍摄内容与去除后的图像A的拍摄内容最相似的参考图像2的确定方法可以参照前述实施例中参考图像1的确定方法，在此不再赘述。
在获得参考图像1和参考图像2后，可以根据参考图像1和参考图像2，对图像A进行细节恢复处理，获得图像C。
可以理解的是，在获得参考图像1和参考图像2后，还可以进一步确定参考图像1的拍摄内容和参考图像2的拍摄内容占图像A的拍摄内容的比例，若该比例仍然小于第二阈值，还可以进一步从该图像A的拍摄内容中去除与参考图像1的拍摄内容以及与参考图像2的拍摄内容匹配的拍摄内容，获得去除后的图像A，再进一步从至少一个图像B中检索得到与该去除后的图像A的拍摄内容最相似的参考图像3。以此循环，直至所获得的至少一个参考图像的拍摄内容占图像A的拍摄内容的比例大于或者等于第二阈值，再根据所获得的至少一个参考图像，对图像A进行细节恢复处理，获得分辨率更高的图像C。
第七方面，本申请实施例提供了一种图像处理装置，包括用于执行第五方面的方法的各个模块或单元。
第八方面，本申请实施例提供一种图像处理系统，所述系统包括终端和云服务器，其中：所述第一终端将拍摄的图像A和图像A的拍摄信息发送给云服务器，云服务器接收该图像A和图像A的拍摄信息，并根据图像A的拍摄信息，确定至少一个图像B，其中，所述图像B的拍摄信息与所述图像A的拍摄信息之间的匹配度大于第一阈值，所述图像B的 分辨率大于所述图像A的分辨率。云服务器从所述至少一个图像B中确定拍摄内容与所述图像A的拍摄内容最相似的参考图像1。
可选的，云服务器根据所述参考图像1，对所述图像A进行细节恢复处理，获得图像C，所述图像C的分辨率大于所述图像A的分辨率，并将图像C发送给所述终端。
终端接收该图像C，并显示该图像C。
第九方面，本申请实施例提供一种图像处理系统，所述系统包括终端和云服务器，其中：所述第一终端将拍摄的图像A和图像A的拍摄信息发送给云服务器，云服务器接收该图像A和图像A的拍摄信息，并根据图像A的拍摄信息，确定至少一个图像B，其中，所述图像B的拍摄信息与所述图像A的拍摄信息之间的匹配度大于第一阈值，所述图像B的分辨率大于所述图像A的分辨率。云服务器将所述至少一个图像B发送给终端设备。
终端设备从该至少一个图像B中确定拍摄内容与所述图像A的拍摄内容最相似的参考图像1。
可选的，终端设备根据所述参考图像1，对所述图像A进行细节恢复处理，获得图像C，并输出该图像C，所述图像C的分辨率大于所述图像A的分辨率。
第十方面，本申请提供了一种终端，包括：摄像头、收发器、一个或多个处理器和一个或多个存储器。该一个或多个处理器与该摄像头、该一个或多个存储器以及该收发器耦合，该一个或多个存储器用于存储计算机程序代码，计算机程序代码包括计算机指令，当该一个或多个处理器执行该计算机指令时，使得该终端执行上述任一方面任一项可能的实现方式中的图像处理方法。
第十一方面，本申请提供了一种云服务器，收发器、一个或多个处理器和一个或多个存储器。该一个或多个存储器与一个或多个处理器耦合，该一个或多个存储器用于存储计算机程序代码，计算机程序代码包括计算机指令，当该一个或多个处理器执行该计算机指令时，使得该云服务器执行上述任一方面任一项可能的实现方式中的图像处理方法。
第十二方面，本申请实施例提供了一种计算机存储介质，包括计算机指令，当计算机指令在终端上运行时，使得终端执行上述任一方面任一项可能的实现方式中的图像处理方法。
第十三方面，本申请实施例提供了一种计算机存储介质，包括计算机指令，当计算机指令在云服务器上运行时，使得云服务器执行上述任一方面任一项可能的实现方式中的图像处理方法。
第十四方面，本申请实施例提供了一种计算机程序产品，当计算机程序产品在终端上运行时，使得终端执行上述任一方面任一项可能的实现方式中的图像处理方法。
第十五方面，本申请实施例提供了一种计算机程序产品，当计算机程序产品在云服务器上运行时，使得云服务器执行上述任一方面任一项可能的实现方式中的图像处理方法。
附图说明
图1为本申请实施例提供的一种通信系统的架构示意图；
图2A为本申请实施例提供的一种终端的结构示意图；
图2B为本申请实施例提供的一种终端的软件架构示意图；
图3A-图3H为本申请实施例提供的一组用户界面示意图；
图4A-图4F为本申请实施例提供的另一组用户界面示意图；
图4G为本申请实施例提供的一组图片处理效果对比示意图；
图5A-图5G为申请实施例提供的又一组用户界面示意图；
图6A-图6P为申请实施例提供的又一组用户界面示意图；
图7A-图7D为申请实施例提供的又一组用户界面示意图；
图8为本申请实施例提供的一种图像处理方法的流程示意图；
图9为本申请实施例提供的一种图像处理系统的架构示意图；
图10为本申请实施例提供的又一种图像处理系统的架构示意图；
图11为本申请实施例提供的又一种图像处理系统的架构示意图；
图12为本申请实施例提供的又一种图像处理系统的架构示意图；
图13为本申请实施例提供的一种数据构建系统的架构示意图；
图14为本申请实施例提供的一种云服务器的结构示意图；
图15为本申请实施例提供的一种图像处理方法的流程示意图；
图16为本申请实施例提供的另一种图像处理方法的流程示意图；
图17为本申请实施例提供的又一种图像处理方法的流程示意图；
图18为本申请实施例提供的一种图像处理方案的框架图；
图19为本申请实施例提供的一种图像处理装置的结构示意图；
图20为本申请实施例提供的另一种图像处理方法的流程示意图；
图21A为本申请实施例提供的低清图像的示意图；
图21B为本申请实施例提供的低清图像的语义分割mask图；
图21C为本申请实施例提供的针对低清图像的分块示意图；
图21D为本申请实施例提供的图像块的示意图；
图22为本申请实施例提供的图像配准的流程示意图；
图23A为本申请实施例提供的图像细节修复的流程示意图；
图23B为本申请实施例提供的高清参考图像的示意图；
图23C为本申请实施例提供的高清图像块的示意图；
图24为本申请实施例提供的另一种图像处理方法的流程示意图；
图25A为本申请实施例提供的全景高清图像的分块示意图；
图25B为本申请实施例提供的全景高清图像中分割的高清图像示例图；
图25C为本申请实施例提供的另一种全景高清图像的分块示意图；
图25D为本申请实施例提供的全景高清图像中分割的高清图像示例图。
具体实施方式
下面将结合附图对本申请实施例中的技术方案进行清除、详尽地描述。其中，在本申请实施例的描述中，除非另有说明，“/”表示或的意思，例如，A/B可以表示A或B；文本中的“和/或”仅仅是一种描述关联对象的关联关系，表示可以存在三种关系，例如，A和/ 或B，可以表示：单独存在A，同时存在A和B，单独存在B这三种情况，另外，在本申请实施例的描述中，“多个”是指两个或多于两个。
以下，术语“第一”、“第二”仅用于描述目的，而不能理解为暗示或暗示相对重要性或者隐含指明所指示的技术特征的数量。由此，限定有“第一”、“第二”的特征可以明示或者隐含地包括一个或者更多个该特征，在本申请实施例的描述中，除非另有说明，“多个”的含义是两个或两个以上。
下面介绍本申请实施例涉及的一种通信系统10。
请参照图1，图1示出了申请实施例涉及的一种通信系统10的架构示意图。该通信系统10可以包括终端100和云服务器200。其中，终端100可以是手机、平板电脑、桌面型计算机、膝上型计算机、手持计算机、笔记本电脑、超级移动个人计算机(ultra-mobile personal computer，UMPC)、上网本，以及蜂窝电话、个人数字助理(personal digital assistant，PDA)、增强现实(augmented reality，AR)\虚拟现实(virtual reality，VR)设备等。本申请实施例对该终端100的具体类型不作特殊限制。
终端100可以通过2G网络、3G网络、4G网络、5G网络、无线局域网(wireless local area network，WLAN)等连接上云服务器200。其中，终端100可以向云服务器200发送图像数据、图像数据的环境信息(例如地理位置信息、拍摄设备拍摄姿态信息、拍摄时间信息、天气信息等等)。云服务器200可以向终端100发送图像处理网络模型、高清图片数据等等。
云服务器200可以与多个终端100建立连接，可以独立处理多个终端100请求的处理任务。其中，云服务器200可以通过终端上用户登录的账户(例如，华为账户)对终端进行区分。
图2A示出了终端100的结构示意图。
下面以终端100为例对实施例进行具体说明。应该理解的是，图2A所示终端100仅是一个范例，并且终端100可以具有比图2A中所示的更多的或者更少的部件，可以组合两个或多个的部件，或者可以具有不同的部件配置。图中所示出的各种部件可以在包括一个或多个信号处理和/或专用集成电路在内的硬件、软件、或硬件和软件的组合中实现。
终端100可以包括：处理器110，外部存储器接口120，内部存储器121，通用串行总线(universal serial bus，USB)接口130，充电管理模块140，电源管理模块141，电池142，天线1，天线2，移动通信模块150，无线通信模块160，音频模块170，扬声器170A，受话器170B，麦克风170C，耳机接口170D，传感器模块180，按键190，马达191，指示器192，摄像头193，显示屏194，以及用户标识模块(subscriber identification module，SIM)卡接口195等。其中传感器模块180可以包括压力传感器180A，陀螺仪传感器180B，气压传感器180C，磁传感器180D，加速度传感器180E，距离传感器180F，接近光传感器180G，指纹传感器180H，温度传感器180J，触摸传感器180K，环境光传感器180L，骨传导传感器180M等。
可以理解的是，本发明实施例示意的结构并不构成对终端100的具体限定。在本申请另一些实施例中，终端100可以包括比图示更多或更少的部件，或者组合某些部件，或者 拆分某些部件，或者不同的部件布置。图示的部件可以以硬件，软件或软件和硬件的组合实现。
处理器110可以包括一个或多个处理单元，例如：处理器110可以包括应用处理器(application processor，AP)，调制解调处理器，图形处理器(graphics processing unit，GPU)，图像信号处理器(image signal processor，ISP)，控制器，存储器，视频编解码器，数字信号处理器(digital signal processor，DSP)，基带处理器，和/或神经网络处理器(neural-network processing unit，NPU)等。其中，不同的处理单元可以是独立的器件，也可以集成在一个或多个处理器中。
其中，控制器可以是终端100的神经中枢和指挥中心。控制器可以根据指令操作码和时序信号，产生操作控制信号，完成取指令和执行指令的控制。
处理器110中还可以设置存储器，用于存储指令和数据。在一些实施例中，处理器110中的存储器为高速缓冲存储器。该存储器可以保存处理器110刚用过或循环使用的指令或数据。如果处理器110需要再次使用该指令或数据，可从所述存储器中直接调用。避免了重复存取，减少了处理器110的等待时间，因而提高了系统的效率。
在一些实施例中，处理器110可以包括一个或多个接口。接口可以包括集成电路(inter-integrated circuit，I2C)接口，集成电路内置音频(inter-integrated circuit sound，I2S)接口，脉冲编码调制(pulse code modulation，PCM)接口，通用异步收发传输器(universal asynchronous receiver/transmitter，UART)接口，移动产业处理器接口(mobile industry processor interface，MIPI)，通用输入输出(general-purpose input/output，GPIO)接口，用户标识模块(subscriber identity module，SIM)接口，和/或通用串行总线(universal serial bus，USB)接口等。
I2C接口是一种双向同步串行总线，包括一根串行数据线(serial data line，SDA)和一根串行时钟线(derail clock line，SCL)。在一些实施例中，处理器110可以包含多组I2C总线。处理器110可以通过不同的I2C总线接口分别耦合触摸传感器180K，充电器，闪光灯，摄像头193等。例如：处理器110可以通过I2C接口耦合触摸传感器180K，使处理器110与触摸传感器180K通过I2C总线接口通信，实现终端100的触摸功能。
I2S接口可以用于音频通信。在一些实施例中，处理器110可以包含多组I2S总线。处理器110可以通过I2S总线与音频模块170耦合，实现处理器110与音频模块170之间的通信。在一些实施例中，音频模块170可以通过I2S接口向无线通信模块160传递音频信号，实现通过蓝牙耳机接听电话的功能。
PCM接口也可以用于音频通信，将模拟信号抽样，量化和编码。在一些实施例中，音频模块170与无线通信模块160可以通过PCM总线接口耦合。在一些实施例中，音频模块170也可以通过PCM接口向无线通信模块160传递音频信号，实现通过蓝牙耳机接听电话的功能。所述I2S接口和所述PCM接口都可以用于音频通信。
UART接口是一种通用串行数据总线，用于异步通信。该总线可以为双向通信总线。它将要传输的数据在串行通信与并行通信之间转换。在一些实施例中，UART接口通常被用于连接处理器110与无线通信模块160。例如：处理器110通过UART接口与无线通信模块160中的蓝牙模块通信，实现蓝牙功能。在一些实施例中，音频模块170可以通过UART 接口向无线通信模块160传递音频信号，实现通过蓝牙耳机播放音乐的功能。
MIPI接口可以被用于连接处理器110与显示屏194，摄像头193等外围器件。MIPI接口包括摄像头串行接口(camera serial interface，CSI)，显示屏串行接口(display serial interface，DSI)等。在一些实施例中，处理器110和摄像头193通过CSI接口通信，实现终端100的拍摄功能。处理器110和显示屏194通过DSI接口通信，实现终端100的显示功能。
GPIO接口可以通过软件配置。GPIO接口可以被配置为控制信号，也可被配置为数据信号。在一些实施例中，GPIO接口可以用于连接处理器110与摄像头193，显示屏194，无线通信模块160，音频模块170，传感器模块180等。GPIO接口还可以被配置为I2C接口，I2S接口，UART接口，MIPI接口等。
USB接口130是符合USB标准规范的接口，具体可以是Mini USB接口，Micro USB接口，USB Type C接口等。USB接口130可以用于连接充电器为终端100充电，也可以用于终端100与外围设备之间传输数据。也可以用于连接耳机，通过耳机播放音频。该接口还可以用于连接其他终端，例如AR设备等。
可以理解的是，本发明实施例示意的各模块间的接口连接关系，只是示意性说明，并不构成对终端100的结构限定。在本申请另一些实施例中，终端100也可以采用上述实施例中不同的接口连接方式，或多种接口连接方式的组合。
充电管理模块140用于从充电器接收充电输入。其中，充电器可以是无线充电器，也可以是有线充电器。在一些有线充电的实施例中，充电管理模块140可以通过USB接口130接收有线充电器的充电输入。在一些无线充电的实施例中，充电管理模块140可以通过终端100的无线充电线圈接收无线充电输入。充电管理模块140为电池142充电的同时，还可以通过电源管理模块141为终端供电。
电源管理模块141用于连接电池142，充电管理模块140与处理器110。电源管理模块141接收电池142和/或充电管理模块140的输入，为处理器110，内部存储器121，外部存储器，显示屏194，摄像头193，和无线通信模块160等供电。电源管理模块141还可以用于监测电池容量，电池循环次数，电池健康状态(漏电，阻抗)等参数。在其他一些实施例中，电源管理模块141也可以设置于处理器110中。在另一些实施例中，电源管理模块141和充电管理模块140也可以设置于同一个器件中。
终端100的无线通信功能可以通过天线1，天线2，移动通信模块150，无线通信模块160，调制解调处理器以及基带处理器等实现。
天线1和天线2用于发射和接收电磁波信号。终端100中的每个天线可用于覆盖单个或多个通信频带。不同的天线还可以复用，以提高天线的利用率。例如：可以将天线1复用为无线局域网的分集天线。在另外一些实施例中，天线可以和调谐开关结合使用。
移动通信模块150可以提供应用在终端100上的包括2G/3G/4G/5G等无线通信的解决方案。移动通信模块150可以包括至少一个滤波器，开关，功率放大器，低噪声放大器(low noise amplifier，LNA)等。移动通信模块150可以由天线1接收电磁波，并对接收的电磁波进行滤波，放大等处理，传送至调制解调处理器进行解调。移动通信模块150还可以对经调制解调处理器调制后的信号放大，经天线1转为电磁波辐射出去。在一些实施例中，移动通信模块150的至少部分功能模块可以被设置于处理器110中。在一些实施例中，移动 通信模块150的至少部分功能模块可以与处理器110的至少部分模块被设置在同一个器件中。
调制解调处理器可以包括调制器和解调器。其中，调制器用于将待发送的低频基带信号调制成中高频信号。解调器用于将接收的电磁波信号解调为低频基带信号。随后解调器将解调得到的低频基带信号传送至基带处理器处理。低频基带信号经基带处理器处理后，被传递给应用处理器。应用处理器通过音频设备(不限于扬声器170A，受话器170B等)输出声音信号，或通过显示屏194显示图像或视频。在一些实施例中，调制解调处理器可以是独立的器件。在另一些实施例中，调制解调处理器可以独立于处理器110，与移动通信模块150或其他功能模块设置在同一个器件中。
无线通信模块160可以提供应用在终端100上的包括无线局域网(wireless local area networks，WLAN)(如无线保真(wireless fidelity，Wi-Fi)网络)，蓝牙(bluetooth，BT)，全球导航卫星系统(global navigation satellite system，GNSS)，调频(frequency modulation，FM)，近距离无线通信技术(near field communication，NFC)，红外技术(infrared，IR)等无线通信的解决方案。无线通信模块160可以是集成至少一个通信处理模块的一个或多个器件。无线通信模块160经由天线2接收电磁波，将电磁波信号调频以及滤波处理，将处理后的信号发送到处理器110。无线通信模块160还可以从处理器110接收待发送的信号，对其进行调频，放大，经天线2转为电磁波辐射出去。
在一些实施例中，终端100的天线1和移动通信模块150耦合，天线2和无线通信模块160耦合，使得终端100可以通过无线通信技术与网络以及其他设备通信。所述无线通信技术可以包括全球移动通讯系统(global system for mobile communications，GSM)，通用分组无线服务(general packet radio service，GPRS)，码分多址接入(code division multiple access，CDMA)，宽带码分多址(wideband code division multiple access，WCDMA)，时分码分多址(time-division code division multiple access，TD-SCDMA)，长期演进(long term evolution，LTE)，BT，GNSS，WLAN，NFC，FM，和/或IR技术等。所述GNSS可以包括全球卫星定位系统(global positioning system，GPS)，全球导航卫星系统(global navigation satellite system，GLONASS)，北斗卫星导航系统(beidou navigation satellite system，BDS)，准天顶卫星系统(quasi-zenith satellite system，QZSS)和/或星基增强系统(satellite based augmentation systems，SBAS)。
终端100通过GPU，显示屏194，以及应用处理器等实现显示功能。GPU为图像处理的微处理器，连接显示屏194和应用处理器。GPU用于执行数学和几何计算，用于图形渲染。处理器110可包括一个或多个GPU，其执行程序指令以生成或改变显示信息。
显示屏194用于显示图像，视频等。显示屏194包括显示面板。显示面板可以采用液晶显示屏(liquid crystal display，LCD)，有机发光二极管(organic light-emitting diode，OLED)，有源矩阵有机发光二极体或主动矩阵有机发光二极体(active-matrix organic light emitting diode的，AMOLED)，柔性发光二极管(flex light-emitting diode，FLED)，Miniled，MicroLed，Micro-oLed，量子点发光二极管(quantum dot light emitting diodes，QLED)等。在一些实施例中，终端100可以包括1个或N个显示屏194，N为大于1的正整数。
终端100可以通过ISP，摄像头193，视频编解码器，GPU，显示屏194以及应用处理 器等实现拍摄功能。
ISP用于处理摄像头193反馈的数据。例如，拍照时，打开快门，光线通过镜头被传递到摄像头感光元件上，光信号转换为电信号，摄像头感光元件将所述电信号传递给ISP处理，转化为肉眼可见的图像。ISP还可以对图像的噪点，亮度，肤色进行算法优化。ISP还可以对拍摄场景的曝光，色温等参数优化。在一些实施例中，ISP可以设置在摄像头193中。
摄像头193用于捕获静态图像或视频。物体通过镜头生成光学图像投射到感光元件。感光元件可以是电荷耦合器件(charge coupled device，CCD)或互补金属氧化物半导体(complementary metal-oxide-semiconductor，CMOS)光电晶体管。感光元件把光信号转换成电信号，之后将电信号传递给ISP转换成数字图像信号。ISP将数字图像信号输出到DSP加工处理。DSP将数字图像信号转换成标准的RGB，YUV等格式的图像信号。在一些实施例中，终端100可以包括1个或N个摄像头193，N为大于1的正整数。
数字信号处理器用于处理数字信号，除了可以处理数字图像信号，还可以处理其他数字信号。例如，当终端100在频点选择时，数字信号处理器用于对频点能量进行傅里叶变换等。
视频编解码器用于对数字视频压缩或解压缩。终端100可以支持一种或多种视频编解码器。这样，终端100可以播放或录制多种编码格式的视频，例如：动态图像专家组(moving picture experts group，MPEG)1，MPEG2，MPEG3，MPEG4等。
NPU为神经网络(neural-network，NN)计算处理器，通过借鉴生物神经网络结构，例如借鉴人脑神经元之间传递模式，对输入信息快速处理，还可以不断的自学习。通过NPU可以实现终端100的智能认知等应用，例如：图像识别，人脸识别，语音识别，文本理解等。
外部存储器接口120可以用于连接外部存储卡，例如Micro SD卡，实现扩展终端100的存储能力。外部存储卡通过外部存储器接口120与处理器110通信，实现数据存储功能。例如将音乐，视频等文件保存在外部存储卡中。
内部存储器121可以用于存储计算机可执行程序代码，所述可执行程序代码包括指令。处理器110通过运行存储在内部存储器121的指令，从而执行终端100的各种功能应用以及数据处理。内部存储器121可以包括存储程序区和存储数据区。其中，存储程序区可存储操作系统，至少一个功能所需的应用程序(比如声音播放功能，图像播放功能等)等。存储数据区可存储终端100使用过程中所创建的数据(比如音频数据，电话本等)等。此外，内部存储器121可以包括高速随机存取存储器，还可以包括非易失性存储器，例如至少一个磁盘存储器件，闪存器件，通用闪存存储器(universal flash storage，UFS)等。
终端100可以通过音频模块170，扬声器170A，受话器170B，麦克风170C，耳机接口170D，以及应用处理器等实现音频功能。例如音乐播放，录音等。
音频模块170用于将数字音频信息转换成模拟音频信号输出，也用于将模拟音频输入转换为数字音频信号。音频模块170还可以用于对音频信号编码和解码。在一些实施例中，音频模块170可以设置于处理器110中，或将音频模块170的部分功能模块设置于处理器110中。
扬声器170A，也称“喇叭”，用于将音频电信号转换为声音信号。终端100可以通过扬声器170A收听音乐，或收听免提通话。
受话器170B，也称“听筒”，用于将音频电信号转换成声音信号。当终端100接听电话或语音信息时，可以通过将受话器170B靠近人耳接听语音。
麦克风170C，也称“话筒”，“传声器”，用于将声音信号转换为电信号。当拨打电话或发送语音信息时，用户可以通过人嘴靠近麦克风170C发声，将声音信号输入到麦克风170C。终端100可以设置至少一个麦克风170C。在另一些实施例中，终端100可以设置两个麦克风170C，除了采集声音信号，还可以实现降噪功能。在另一些实施例中，终端100还可以设置三个，四个或更多麦克风170C，实现采集声音信号，降噪，还可以识别声音来源，实现定向录音功能等。
耳机接口170D用于连接有线耳机。耳机接口170D可以是USB接口130，也可以是3.5mm的开放移动终端平台(open mobile terminal platform，OMTP)标准接口，美国蜂窝电信工业协会(cellular telecommunications industry association of the USA，CTIA)标准接口。
压力传感器180A用于感受压力信号，可以将压力信号转换成电信号。在一些实施例中，压力传感器180A可以设置于显示屏194。压力传感器180A的种类很多，如电阻式压力传感器，电感式压力传感器，电容式压力传感器等。电容式压力传感器可以是包括至少两个具有导电材料的平行板。当有力作用于压力传感器180A，电极之间的电容改变。终端100根据电容的变化确定压力的强度。当有触摸操作作用于显示屏194，终端100根据压力传感器180A检测所述触摸操作强度。终端100也可以根据压力传感器180A的检测信号计算触摸的位置。在一些实施例中，作用于相同触摸位置，但不同触摸操作强度的触摸操作，可以对应不同的操作指令。例如：当有触摸操作强度小于第一压力阈值的触摸操作作用于短消息应用图标时，执行查看短消息的指令。当有触摸操作强度大于或等于第一压力阈值的触摸操作作用于短消息应用图标时，执行新建短消息的指令。
陀螺仪传感器180B可以用于确定终端100的运动姿态。在一些实施例中，可以通过陀螺仪传感器180B确定终端100围绕三个轴(即，x，y和z轴)的角速度。陀螺仪传感器180B可以用于拍摄防抖。示例性的，当按下快门，陀螺仪传感器180B检测终端100抖动的角度，根据角度计算出镜头模组需要补偿的距离，让镜头通过反向运动抵消终端100的抖动，实现防抖。陀螺仪传感器180B还可以用于导航，体感游戏场景。
气压传感器180C用于测量气压。在一些实施例中，终端100通过气压传感器180C测得的气压值计算海拔高度，辅助定位和导航。
磁传感器180D包括霍尔传感器。终端100可以利用磁传感器180D检测翻盖皮套的开合。在一些实施例中，当终端100是翻盖机时，终端100可以根据磁传感器180D检测翻盖的开合。进而根据检测到的皮套的开合状态或翻盖的开合状态，设置翻盖自动解锁等特性。
加速度传感器180E可检测终端100在各个方向上(一般为三轴)加速度的大小。当终端100静止时可检测出重力的大小及方向。还可以用于识别终端姿态，应用于横竖屏切换，计步器等应用。
距离传感器180F，用于测量距离。终端100可以通过红外或激光测量距离。在一些实 施例中，拍摄场景，终端100可以利用距离传感器180F测距以实现快速对焦。
接近光传感器180G可以包括例如发光二极管(LED)和光检测器，例如光电二极管。发光二极管可以是红外发光二极管。终端100通过发光二极管向外发射红外光。终端100使用光电二极管检测来自附近物体的红外反射光。当检测到充分的反射光时，可以确定终端100附近有物体。当检测到不充分的反射光时，终端100可以确定终端100附近没有物体。终端100可以利用接近光传感器180G检测用户手持终端100贴近耳朵通话，以便自动熄灭屏幕达到省电的目的。接近光传感器180G也可用于皮套模式，口袋模式自动解锁与锁屏。
环境光传感器180L用于感知环境光亮度。终端100可以根据感知的环境光亮度自适应调节显示屏194亮度。环境光传感器180L也可用于拍照时自动调节白平衡。环境光传感器180L还可以与接近光传感器180G配合，检测终端100是否在口袋里，以防误触。
指纹传感器180H用于采集指纹。终端100可以利用采集的指纹特性实现指纹解锁，访问应用锁，指纹拍照，指纹接听来电等。
温度传感器180J用于检测温度。在一些实施例中，终端100利用温度传感器180J检测的温度，执行温度处理策略。例如，当温度传感器180J上报的温度超过阈值，终端100执行降低位于温度传感器180J附近的处理器的性能，以便降低功耗实施热保护。在另一些实施例中，当温度低于另一阈值时，终端100对电池142加热，以避免低温导致终端100异常关机。在其他一些实施例中，当温度低于又一阈值时，终端100对电池142的输出电压执行升压，以避免低温导致的异常关机。
触摸传感器180K，也称“触控面板”。触摸传感器180K可以设置于显示屏194，由触摸传感器180K与显示屏194组成触摸屏，也称“触控屏”。触摸传感器180K用于检测作用于其上或附近的触摸操作。触摸传感器可以将检测到的触摸操作传递给应用处理器，以确定触摸事件类型。可以通过显示屏194提供与触摸操作相关的视觉输出。在另一些实施例中，触摸传感器180K也可以设置于终端100的表面，与显示屏194所处的位置不同。
骨传导传感器180M可以获取振动信号。在一些实施例中，骨传导传感器180M可以获取人体声部振动骨块的振动信号。骨传导传感器180M也可以接触人体脉搏，接收血压跳动信号。在一些实施例中，骨传导传感器180M也可以设置于耳机中，结合成骨传导耳机。音频模块170可以基于所述骨传导传感器180M获取的声部振动骨块的振动信号，解析出语音信号，实现语音功能。应用处理器可以基于所述骨传导传感器180M获取的血压跳动信号解析心率信息，实现心率检测功能。
按键190包括开机键，音量键等。按键190可以是机械按键。也可以是触摸式按键。终端100可以接收按键输入，产生与终端100的用户设置以及功能控制有关的键信号输入。
马达191可以产生振动提示。马达191可以用于来电振动提示，也可以用于触摸振动反馈。例如，作用于不同应用(例如拍照，音频播放等)的触摸操作，可以对应不同的振动反馈效果。作用于显示屏194不同区域的触摸操作，马达191也可对应不同的振动反馈效果。不同的应用场景(例如：时间提醒，接收信息，闹钟，游戏等)也可以对应不同的振动反馈效果。触摸振动反馈效果还可以支持自定义。
指示器192可以是指示灯，可以用于指示充电状态，电量变化，也可以用于指示消息， 未接来电，通知等。
SIM卡接口195用于连接SIM卡。SIM卡可以通过插入SIM卡接口195，或从SIM卡接口195拔出，实现和终端100的接触和分离。终端100可以支持1个或N个SIM卡接口，N为大于1的正整数。SIM卡接口195可以支持Nano SIM卡，Micro SIM卡，SIM卡等。同一个SIM卡接口195可以同时插入多张卡。所述多张卡的类型可以相同，也可以不同。SIM卡接口195也可以兼容不同类型的SIM卡。SIM卡接口195也可以兼容外部存储卡。终端100通过SIM卡和网络交互，实现通话以及数据通信等功能。在一些实施例中，终端100采用eSIM，即：嵌入式SIM卡。eSIM卡可以嵌在终端100中，不能和终端100分离。
终端100的软件系统可以采用分层架构，事件驱动架构，微核架构，微服务架构，或云架构。本发明实施例以分层架构的Android系统为例，示例性说明终端100的软件结构。
图2B是本发明实施例的终端100的软件结构框图。
分层架构将软件分成若干个层，每一层都有清晰的角色和分工。层与层之间通过软件接口通信。在一些实施例中，将Android系统分为四层，从上至下分别为应用程序层，应用程序框架层，安卓运行时(Android runtime)和系统库，以及内核层。
应用程序层可以包括一系列应用程序包。
如图2B所示，应用程序包可以包括相机，图库，日历，通话，地图，导航，WLAN，蓝牙，音乐，视频，短信息等应用程序。
应用程序框架层为应用程序层的应用程序提供应用编程接口(application programming interface，API)和编程框架。应用程序框架层包括一些预先定义的函数。
如图2B所示，应用程序框架层可以包括窗口管理器，内容提供器，视图系统，电话管理器，资源管理器，通知管理器等。
窗口管理器用于管理窗口程序。窗口管理器可以获取显示屏大小，判断是否有状态栏，锁定屏幕，截取屏幕等。
内容提供器用来存放和获取数据，并使这些数据可以被应用程序访问。所述数据可以包括视频，图像，音频，拨打和接听的电话，浏览历史和书签，电话簿等。
视图系统包括可视控件，例如显示文字的控件，显示图片的控件等。视图系统可用于构建应用程序。显示界面可以由一个或多个视图组成的。例如，包括短信通知图标的显示界面，可以包括显示文字的视图以及显示图片的视图。
电话管理器用于提供终端100的通信功能。例如通话状态的管理(包括接通，挂断等)。
资源管理器为应用程序提供各种资源，比如本地化字符串，图标，图片，布局文件，视频文件等等。
通知管理器使应用程序可以在状态栏中显示通知信息，可以用于传达告知类型的消息，可以短暂停留后自动消失，无需用户交互。比如通知管理器被用于告知下载完成，消息提醒等。通知管理器还可以是以图表或者滚动条文本形式出现在系统顶部状态栏的通知，例如后台运行的应用程序的通知，还可以是以对话窗口形式出现在屏幕上的通知。例如在状态栏提示文本信息，发出提示音，终端振动，指示灯闪烁等。
Android Runtime包括核心库和虚拟机。Android runtime负责安卓系统的调度和管理。
核心库包含两部分：一部分是java语言需要调用的功能函数，另一部分是安卓的核心库。
应用程序层和应用程序框架层运行在虚拟机中。虚拟机将应用程序层和应用程序框架层的java文件执行为二进制文件。虚拟机用于执行对象生命周期的管理，堆栈管理，线程管理，安全和异常的管理，以及垃圾回收等功能。
系统库可以包括多个功能模块。例如：表面管理器(surface manager)，媒体库(Media Libraries)，三维图形处理库(例如：OpenGL ES)，2D图形引擎(例如：SGL)等。
表面管理器用于对显示子系统进行管理，并且为多个应用程序提供了2D和3D图层的融合。
媒体库支持多种常用的音频，视频格式回放和录制，以及静态图像文件等。媒体库可以支持多种音视频编码格式，例如:MPEG4，H.264，MP3，AAC，AMR，JPG，PNG等。
三维图形处理库用于实现三维图形绘图，图像渲染，合成，和图层处理等。
2D图形引擎是2D绘图的绘图引擎。
内核层是硬件和软件之间的层。内核层至少包含显示驱动，摄像头驱动，音频驱动，传感器驱动。
下面结合捕获拍照场景，示例性说明终端100软件以及硬件的工作流程。
当触摸传感器180K接收到触摸操作，相应的硬件中断被发给内核层。内核层将触摸操作加工成原始输入事件(包括触摸坐标，触摸操作的时间戳等信息)。原始输入事件被存储在内核层。应用程序框架层从内核层获取原始输入事件，识别该输入事件所对应的控件。以该触摸操作是触摸单击操作，该单击操作所对应的控件为相机应用图标的控件为例，相机应用调用应用框架层的接口，启动相机应用，进而通过调用内核层启动摄像头驱动，通过摄像头193捕获静态图像或视频。
本申请提供一种图像处理方法，可以结合终端和云服务器的处理能力，充分利用云服务器上的高质量图像数据和图像处理算法，赋能终端拍照成像系统。云服务器可以通过终端上传的图像以及图像的环境信息(例如地理位置信息、天气信息、时间信息、拍摄参数、拍摄姿态信息等等)，匹配出适合该图像的图像处理算法以及与该图像拍摄内容相同，拍摄环境相近的高质量图像数据。通过该图像处理算法以及该高质量图像数据，可以增强终端上传图像的质量和效果。这样，可以让不同环境下拍摄的图片，呈现出不同的风格和效果。这样也使终端能够突破终端自身成像质量极限，使得终端上的普通摄像头也能拍出专业单反设备的效果，从而在降低手机成本的同时，极大的提升终端的拍照质量和用户拍照体验。
下面结合应用场景，介绍本申请实施例涉及的一种图像处理方法。
在一些应用场景中，用户在使用终端100拍照时，可以选择终端100的相机应用中的不同拍照模式进行拍照。当用户通过终端100在景点、商圈等地方拍摄风景时，用户可以选择相机应用中的“云增强模式”进行拍摄。在“云增强模式”下，终端100可以借助云服务器的处理能力对终端100拍摄到的画面进行增强优化。这样，可以使终端100能够突破终端自身成像质量极限，使得终端上的普通摄像头也能拍出专业单反设备的效果。例如， 终端100在拍摄远景时，由于自身成像质量极限的限制，当变焦到一定程度时，终端100拍摄的远景的细节会变得模糊。而当终端100开启“云增强模式”拍摄远景时，终端100可以借助云服务器的处理能力，对终端100拍摄到的远景图像进行优化，使得终端100在拍摄远景时，也能拍出清晰的图片。
如图3A所示，终端100可以显示有主屏幕的界面310，该界面310中显示了一个放置有应用图标的页面，该页面包括多个应用图标(例如，天气应用图标、股票应用图标、计算器应用图标、设置应用图标、邮件应用图标、支付宝应用图标、脸书应用图标、浏览器应用图标、图库应用图标312、音乐应用图标、视频应用图标、微信应用图标等等)。多个应用图标下方还显示包括有页面指示符，以表明当前显示的页面与其他页面的位置关系。页面指示符的下方有多个托盘图标(例如拨号应用图标、信息应用图标、联系人应用图标、相机应用图标313)，托盘图标在页面切换时保持显示。在一些实施例中，上述页面也可以包括多个应用图标和页面指示符，页面指示符可以不是页面的一部分，单独存在，上述图片图标也是可选的，本申请实施例对此不作限制。在该界面310的上方部分区域显示有状态栏311，该状态栏311可以包括：移动通信信号(又可称为蜂窝信号)的一个或多个信号强度指示符，电池状态指示符，时间指示符，Wi-Fi信号指示符等等。
终端100可以接收用户作用于相机应用图标313的输入操作(例如单击)，响应于该输入操作，终端100可以显示如图3B所示的拍摄界面320。
如图3B所示，该拍摄界面320可以显示包括有已拍摄图像回显控件321、拍摄控件322、摄像头转换控件323、摄像头捕捉的画面324、设置控件325、变焦倍率控件326、一个或多个拍摄模式控件(例如，“夜景模式”控件372A、“人像模式”控件372B、“云增强模式”控件372C、“普通拍照模式”控件372D、“录像模式”控件372E、“专业模式”控件372F、更多模式控件327G)。其中，该已拍摄图像回显控件321可用于显示已拍摄的图像。该拍摄控件322用于触发保存摄像头拍摄到的图像。该摄像头转换控件323可用于切换拍照的摄像头。该设置控件325可用于设置拍照功能。该变焦倍率控件326可用于设置摄像头的变焦倍数。该拍摄模式控件可用于触发开启该拍摄模式对应的图像处理流程。例如，“夜景模式”控件372A可用于触发增加拍摄图像中的亮度和色彩丰富度等。“人像模式”控件372B可用于触发对拍摄图像中人物背景的虚化。“云增强模式”控件372C可用于触发借助云服务器的处理能力增强拍摄图像的画面效果。如图3B所示，当前用户选择的拍摄模式为“普通拍照模式”。
终端100可以接收用户作用于“云增强模式”控件327C的输入操作(例如单击)，响应于该输入操作，如图3C所示，终端100可以将当前用户选中的拍摄模式调整为“云增强模式”，并显示出模式详情控件328。
如图3C所示，终端100可以接收用户作用于模式详情控件328的输入操作(例如单击)，响应于该输入操作，终端100可以显示如图3D所示的“云增强模式”详情页330。
如图3D所示，该“云增强模式”详情页330可以显示包括有“云增强模式”开启时与未开启时的效果对比图。例如，该“云增强模式”详情页330可以显示有未开启“云增强模式”时拍摄的图像331以及该图像331的细节图332，和开启“云增强模式”时拍摄的图像333以及该图像333的细节图334。该“云增强模式”详情页330还可以显示有关 于“云增强模式”的文字说明335(例如，“借助云端能力拍摄高清晰度的照片，突显细节、提升颜色表现，增强图片质量。建议使用该模式拍照时开启数据网络以及定位。”)。
终端100可以接收用户再次作用于模式详情控件328的输入操作(例如单击)，响应于该输入操作，如图3E所示，终端100可以关闭该“云增强模式”详情页330，并返回显示上述拍摄界面320。
如图3E所示，用户当前选择的拍摄模式为“云增强模式”。终端100可以接收用户针对拍摄控件322的输入操作(例如单击)，响应于该输入操作，终端100可以获取摄像头拍摄到的图像以及该图像的环境信息。其中，该环境信息可以包括以下一种或多种：地理位置信息、天气信息、时间信息、拍摄参数、拍摄姿态信息等等。其中，拍摄参数可以包括变焦倍数等等。拍摄姿态信息可以包括拍摄设备的拍摄镜头在拍摄图像时的朝向信息。例如，拍摄镜头的俯仰角以及方向角度等等，或者自然坐标下的三轴旋转角等等。终端100可以将拍摄图像以及该拍摄图像的环境信息发送给云服务器200，借助云服务器200的处理能力对拍摄图像进行增强。其中，终端100借助云服务器200的处理能力对拍摄图像进行增强的过程将在后续实施例中详细介绍，在此不予详述。
在一些实施例中，在终端100借助云服务器200的处理能力对拍摄图像进行增强的过程中，终端100可以显示处理进度提示或倒计时，用于提示用户当前终端100借助云服务器200的处理能力对拍摄图像增强的进度。
示例性的。如图3F所示，终端100可以在拍摄界面320上显示进度提示窗口341，用于提示对拍摄图像增强的进度。例如，该进度提示窗口341中显示有文字提示“正在通过云端优化拍摄的照片，进度10％”。该进度提示窗口341中还可以显示有取消控件342，可用于触发取消对拍摄图像的增强。例如，当终端100接收用户的针对该取消控件342的输入操作(例如单击)，终端100可以向云服务器200发送取消指令，指示云服务200停止对拍摄图像的增强进程。这样，可以在进度较缓慢时或者用户临时不想对拍摄图像增强时，即时取消对拍摄图像的增强。
在一种可能的实现方式中，终端100可以实时显示增强的效果,当用户对中间的某个增强效果已经满意的时候，终端可以提前结束增强,将结束时的中途增强结果作为最终结果保存下来。
在终端100借助云服务器200的处理能力对拍摄图像完成增强后，终端100可以显示完成提示，并保存增强后的图像至本地。该完成提示可用于提示用户当前已对拍摄图像完成增强。
示例性的，如图3G所示，终端100可以在借助云服务器200的处理能力对拍摄图像完成增强后，显示完成提示343。例如，完成提示343可以是文字提示“已完成照片增强，可到图库中查看”。
终端100可以接收用户针对已拍摄图像回显控件321的输入操作(例如单击)，响应于该输入操作，终端100可以显示如图3H所示照片浏览界面350。
如图3H所示，该照片浏览界面350可以显示包括有增强后的图像351、标记352、图像相关信息353、菜单354、图库控件355。其中，该标记352用于提示用户该图像351已借助云服务器200的处理能力进行增强。该图像相关信息353可以包括有该图像351的拍 摄时间、拍摄天气、地理位置信息等等。例如，拍摄时间可以为“2019年12月3日8：00AM”，拍摄天气可以为“多云”，拍摄地点可以为“上海·迪士尼”等等。该菜单354可以包括分享按钮、收藏按钮、编辑按钮、删除按钮、更多按钮。分享按钮可用于触发对该图像351的分享。该收藏按钮可用于触发收藏该图像351到图片收藏文件夹。编辑按钮可用于触发对图像351的旋转、修剪、增加滤镜、虚化等编辑功能。删除按钮可用于触发删除该图像351。更多按钮可用于触发打开更多与该图像351相关的功能。该图库控件355可用于触发终端100打开图库应用。
在一些实施例中，由于终端100借助云服务器200的处理能力对图像增强时，需要先将图像和图像的环境信息发送给云服务器200。因此，当终端100打开相机应用开启“云增强”模式后，终端100可以检测当前的网络状态，若当前终端100的网络断开无法连接至云服务器，则终端100可以输出网络断开提示，该网络断开提示可用于提示用户在连接网络后再使用云增强模式进行拍照。
示例性的，如图4A所示，终端100显示有拍摄界面320。其中，针对拍摄界面320的文字描述可以参考前述实施例，在此不再赘述。当终端100检测到网络断开，无法与云服务器200建立连接时，终端100可以显示出提示411，该提示411可用于提示用户当前终端100的网络已经断开。例如，该提示411可以是文字提示“当前网络断开，请先连接网络，再借助云端优化图片”等等。
当终端100打开相机应用开启云增强模式后，终端100的网络状态由断开切换至重新连接时，终端100可以输出网络重连提示，该网络重连提示可用于提示用户当前网络已恢复连接，可正常使用云增强模式拍摄图片。
示例性的，如图4B所示，终端100显示有拍摄界面320。其中，针对拍摄界面320的文字描述可以参考前述实施例，在此不再赘述。当终端100检测到网络由断开切换至恢复连接时，终端100可以显示出提示412，该提示412可用于提示用户当前终端100的网络已经恢复，可以正常使用云增强模式进行拍照。例如，该提示412可以是文字提示“当前网络已连接，您可以使用云增强模式拍摄图片。
在一些实施例中，由于云服务器200可以处理多个终端上传的图片进行增强，云服务器200的对图片增强的任务过多时，云服务器200会对这些任务排队。当终端100请求图片增强任务需要排队时，终端100可以输出排队提示，并在后台保持与云服务器200的通信，监测排队进度。该图片增强任务。该排队提示可用于提示当前的图片增强任务正在排队，需要等待。
示例性的，如图4C所示，终端100显示有拍摄界面320。其中，针对拍摄界面320的文字描述可以参考前述实施例，在此不再赘述。当终端100将拍摄图像和拍摄图像的环境信息发送给云服务器200后，终端100可以周期性从云服务器200上获取增强该拍摄图像的处理进度。当云服务器200上增强图片的任务达到指定数量时，云服务器200可以让来自于终端100的任务请求进行排队，并周期性向终端100发送排队进度。终端100可以输出提示413，该提示413用于提示用户当前终端100请求增强拍摄图像的任务正在排队。例如，该提示413可以文字提示“当前通过云端增强图片的用户角度，已为您排队，可在 通知栏查看排队进度”。
当终端100请求增强拍摄图像的任务处于排队时，终端100可以保持与云服务器200的通信，并在后台监测排队进度，并可以在终端100的通知栏中显示排队进度。
示例性的，如图4D所示，终端100显示有拍摄界面320。终端100可以接收用户的输入操作，调出该拍摄界面320上方的状态栏311。终端100可以接收用户针对该状态栏311的输入操作(例如向下滑动)，响应于该输入操作，终端100可以显示如图4E所示的菜单窗口420和通知栏430。
如图4E所示，该菜单窗口420中可以显示有一些功能的开关控件(例如，无线局域网(WLAN)开关控件、蓝牙开关控件、手电筒开关控件、响铃开关控件、自动旋转开关控件、华为分享(Huawei Share)开关控件，等等)。该通知栏430可以显示有排队通知(例如，“借助云端增强图片效果的用户较多，当前正在排队”)，增强拍摄图片的排队时间提示432(例如，“预计等待时间1分钟”)和取消等待控件431。该取消等待控件431可用于触发终端100向云服务器200发送取消请求，该取消请求用于请求云服务器200取消对终端100的上传拍摄图像的增强。
当终端100借助云服务器200的处理能力完成对拍摄图像的增强后，终端100可以显示如图4F所示的完成提示433。该提示433可用于提示用户当前终端100已完成对拍摄图像的增强。终端100可以接收用户针对该提示433的输入操作(例如单击)，响应于该输入操作，终端100可以显示上述图3H所示的照片浏览界面350。针对照片浏览界面350的文字说明可以参考前述实施例，在此不再赘述。
本申请实施例通过提供的一种图像处理方法，可以实现终端100在“云增强模式”下，通过借助云服务器200的处理能力对终端100拍摄到的画面进行增强优化。可以使终端100能够突破终端自身成像质量极限，使得终端上的普通摄像头也能拍出专业单反设备的效果。
如图4G所示，图4G中的(a)示出了终端100借助云服务器200的处理能力增强之前的拍摄图像，图4G中的(b)示出了终端100借助云服务器200的处理能力增强拍摄图像后得到的图像。图4G中的(c)示出了上述图4G的(a)中拍摄图像的局部图，图4G中的(d)示出了上述图4G的(b)中增强后图像的局部图。对比图4G中(a)和(b)，图4G中的(c)和(d)，可以看出，终端100通过借助云服务器200的处理能力对拍摄图像增强之后，提升了图像整体和细节的清晰度，也提升了图像的对比度、动态范围、色彩等等。
在一些应用场景中，终端100本地可以保存有很多图片，这些图片可能是终端100拍摄的，或者其他设备发送过来的，或者是从网络上下载的。其中，部分图片由于拍摄设备本身拍摄能力的限制，无法呈现高质量的图像效果。因此，本申请实施例提供一种图像处理方法，终端100借助云服务器200的处理能力对图库中已存储的图像进行增强，这样，可以提升已拍摄完成的图像的图像效果，也方便用户在拍摄完图像后借助云服务器200的处理能力增强已拍摄图像的效果。
如图5A所示，终端100可以显示有主屏幕的界面510，该界面510中显示了一个放置 有应用图标的页面，该页面包括多个应用图标(例如，天气应用图标、股票应用图标、计算器应用图标、设置应用图标、邮件应用图标、支付宝应用图标、脸书应用图标、浏览器应用图标、图库应用图标512、音乐应用图标、视频应用图标、微信应用图标等等)。多个应用图标下方还显示包括有页面指示符，以表明当前显示的页面与其他页面的位置关系。页面指示符的下方有多个托盘图标(例如拨号应用图标、信息应用图标、联系人应用图标、相机应用图标513)，该界面510的上方部分区域显示有状态栏511。其中，由于该界面510与上述图3A所示的界面310相同，因此，针对该界面510的详细的文字描述可以参考前述图3A所示实施例，在此不再赘述。
终端100可以接收用户作用于图库应用图标512的输入操作，响应于该输入操作，终端100可以显示如图5B所示的图库相册界面520。
如图5B所示，该图库相册界面520可以显示包括有一个或多个相册(例如，所有照片相册526、视频相册、相机相册、云增强相册527、微信相册、微博相册等等)。终端100可以在在该图库相册界面520的下方显示有图库菜单521。其中，该图库菜单521中包括有照片控件522、相册控件523、时刻控件524、发现控件525。其中，该照片控件522用于触发终端100以图片缩略图的形式显示本地所有的图片。该相册控件523用于触发终端100显示本地图片归属的相册。如图5B所示，当前相册控件523处于选中状态，终端100显示该图库相册界面520。该时刻控件524可用于触发终端100显示本地存储的精选图片。该发现控件525可用于触发终端100显示图片的分类相册。例如，终端100可以根据人物、地点、景点和商圈对图片进行自动分类。
终端100可以接收用户针对该发现控件525的输入操作(例如单击)，响应于该输入操作，终端100可以显示如图5C所示的图库发现界面530。
如图5C所示，该图库发现界面530中显示有人像分类531、地点分类532、景点和商圈分类533。其中，人像分类531中包括有一个或多个包含有人像内容的图片。地点分类532中包括有一个或多个地点相册(例如“厦门”相册、“上海”相册、“深圳”相册等等)。景点和商圈533中包括有一个或多个景点和商圈相册(例如，“上海·迪士尼”相册、“上海·外滩”相册、“北京·故宫”相册等等)。
终端100可以接收用户针对景点和商圈分类533的输入操作(例如单击)，响应于该输入操作，终端100可以显示如图5D所示的景点和商圈分类界面540。
如图5D所示，该景点和商圈分类界面540显示包括有一个或多个相册(例如，相册541、相册542、相册543、相册544等等)和云增强控件545。其中，每个相册下方都有该相册对应的景点或商圈名称以及该相册中的图片数量。例如，相册541对应的景点或商圈名称为“上海·迪士尼”，该相册541中有3张图片。相册542对应的景点或商圈名称为“上海·外滩”，该相册542中有2张图片。相册543对应的景点或商圈名称为“北京·故宫”，该相册543中有2张图片。该相册544对应的景点或商圈名称为“厦门·鼓浪屿”，相册544中有2张图片。以上示例仅仅用于解释本申请，不应构成限定。
终端100可以接收用户作用于云增强控件545的输入操作(例如单击)，响应于该输入操作，终端100可以显示如图5E所示的云增强界面550。
如图5E所示，该云增强界面550中显示包括有图片添加控件551和了解详情控件552。 该图片添加控件551可用于触发终端100选择要借助云服务器200的处理能力进行增强的图片。该了解详情控件552可用于触发终端100显示云增强功能的效果以及相关信息。
终端100可以接收用户针对该了解详情控件552的输入操作(例如单击)，响应于该输入操作，终端100可以显示如图5F所示的云增强详情页560。
如图5F所示，该云增强详情页560中显示有云增强功能的效果对比图。例如，该云增强详情页560可以显示云增强处理之前的图片561以及该图片561的细节图562，和云增强处理之后的图片563以及该图片563的细节图564，和对比结果(例如“增强前细节模糊，增强后细节更清晰”)。其中，该云增强详情页560还可以显示有关于云增强功能的文字说明565(例如，“借助云端能力对照片进行增强处理，可以突显细节、提升清晰度和颜色表现，增强图片的质量”)。该云增强详情页560还可以显示有页面指示符566，该页面指示符566可用于表明当前效果对比图所在页面与其他效果对比图所在页面的位置关系。
终端100可以接收用户针对该云增强详情页560的输入操作(例如向左滑动)，响应于该输入操作，如图5G所示，终端100可以云增强详情页560中切换显示另一组效果对比图。
如图5G所示，终端100可以在云增强页560中显示云增强处理之前的图片571和云增强处理之后的图片572，和对比结果573(例如“增强后的图片的色彩更加丰富均衡，富有质感”)。
终端100在开启增强功能后，可以接收用户的操作，在图库应用中选择图片。终端100可以获取图片以及图片的环境信息(例如地理位置信息、天气信息、时间信息、拍摄参数、拍摄姿态信息等等)，上传给云服务器200，借助云服务器200的处理能力对用户选中的图片进行增强。
如图6A所示，终端100显示有云增强界面550。该云增强界面550中显示包括有图片添加控件551和了解详情控件552。其中，针对该云增强界面550的详细文字描述可以参考前述图5E所示实施例，在此不再赘述。终端100可以接收用户针对图片添加控件551的输入操作(例如单击)，响应于该操作，终端100可以显示如图6B所示的图片选择界面610。
如图6B所示，该图片选择界面610中显示包括有一个或多个的可以进行云增强的相册(例如，“上海·迪士尼”相册、“上海·外滩”相册611、“北京·故宫”相册、“厦门·鼓浪屿”相册等等)。其中，这些相册中的图片的环境信息齐全，可以通过借助云服务器200的处理能力进行增强。图片选择界面610中的每个相册上还可以显示有选择框(例如，“上海·外滩”相册611上显示的选择框612)，该选择框可用于选中该相册中的所有图片。每个相册的下方显示有相册中的图片数量。
终端100可以接收用户作用于该“上海·外滩”相册611的输入操作(例如单击)，响应该输入操作，终端100可以如图6C所示的相册界面620。
如图6C所示，该相册界面620的名称为“上海·外滩”。该相册界面620中显示包括有一张或多张环境信息齐全的图片的缩略图(例如缩略图621)和确定控件623。其中，该相册界面620的每张图片的缩略图上显示有选择框，该选择框可用于选中该图片。如图6C 所示，用户已选中缩略图621对应的图片。
终端100可以在用户选中图片后，接收用户针对确定控件623的输入操作(例如单击)，响应于该输入操作，终端100可以返回显示如图6D所示的图片选择界面610，并在该图片选择界面610中显示选中的图片数量，以及确定控件613。
如图6D所示，终端100在该图片选择界面610上显示有已选中的图片数量614(例如“共选中1张图片”)和确定控件613。
终端100可以接收用户针对该确定控件613的输入操作(例如单击)，响应于该输入操作，终端100可以在如图6E所示的云增强界面550上显示出已选中图片的缩略图(例如缩略图631)和上传控件633。其中，已选中图片的缩略图下方显示有删除控件632，该删除控件632可用于触发终端100删除用户选中的图片。该上传控件633上可以显示有文字描述(例如“上传云端增强效果”)。该上传控件633可用于终端100将用户选中的图片以及图片的环境信息上传至云服务器200，借助云服务器200的处理能力增强用户选中图片的效果。
在一些实施例中，由于终端100借助云服务器200的处理能力对用户选中的图片进行增强时，需要先将用户选中的图片和该图片的环境信息发送给云服务器200。因此，在用户触发终端100开始上传已选中图片和该图片的环境信息时，终端100可以检测当前的网络状态，若网络状态断开，无法连接至云服务器200时，终端100可以输出上传失败提示，该上传失败提示可用于提示用户在连接网络后再上传。
示例性的，如图6E，终端100接收用户针对上传控件633的输入操作(例如单击)，响应于该输入操作，终端100可以先检测当前的网络状态。当前终端100若断网，无法连接至云服务器200，则终端100显示如图6F所示上传失败提示634。
如图6F所示，该上传失败提示634显示有文字提示“上传失败，通过云端增强图片效果，需要先联网，请联网后再上传图片至云端增强效果”。
终端100在显示上述上传失败提示后，可以接收用户的输入，连接上网络，与云服务器200进行通信。当终端100连接上网络后，可以再次接收用户针对上传控件633的输入操作，将用户选中的图片和该图片的环境信息发送给云服务器200。
示例性的，如图6F所示，终端100可以接收用户作用于状态栏511的输入操作(例如向下滑动)，响应于该输入操作，终端100可以显示如图6G所示的菜单窗口640。
如图6G所示，该菜单窗口420中可以显示有一些功能的开关控件(例如，无线局域网(WLAN)开关控件641、蓝牙开关控件、手电筒开关控件、响铃开关控件、自动旋转开关控件、华为分享(Huawei Share)开关控件，等等)。其中，当前无线局域网控件641处于关闭状态。
终端100可以接收用户针对该无线局域网控件641的输入操作(例如单击)，响应于该输入操作，终端100可以打开无线局域网开关，并连接可连接的Wi-Fi网络。如图6H所示，终端100在打开无线局域网开关后，连接上了Wi-Fi网络，并在该无线局域网控件641下方显示有连接Wi-Fi网络的名称(例如“HW-ZL”)，以及在状态栏511中显示Wi-Fi信号 指示符642。该Wi-Fi信号指示符642，用于指示当前终端100网络状态正常，可以连接至云服务器200。
如图6I所示，终端100在网络连接之后，可以接收用户作用于云增强界面550中上传控件633的输入操作(例如单击)，响应于该输入操作，终端100可以将用户选中的图片以及图片的环境信息上传给云服务器200。
在一些实施例中，由于云服务器200可以处理多个终端上传的图片进行增强，云服务器200的对图片增强的任务过多时，云服务器200会对这些任务排队。当终端100请求图片增强任务需要排队时，终端100可以输出排队提示，并在后台保持与云服务器200的通信，监测排队进度。该图片增强任务。该排队提示可用于提示当前的图片增强任务正在排队，需要等待。
示例性的，如图6J所示，当终端100将已选中的图片和图片的环境信息发送给云服务器200后，终端100可以周期性从云服务器200上获取增强该已选中图片的处理进度。当云服务器200上增强图片的任务达到指定数量时，云服务器200可以让来自于终端100的任务请求进行排队，并周期性向终端100发送排队进度。终端100可以输出进度提示窗口650。其中，进度提示窗口650中显示有当前任务进度缓慢的原因(例如，“借助云端增强图片效果的用户较多，当前正在排队”)，等待时间651(例如，“共1张图，预计等待时间1分钟”)以及取消等待控件652。该取消等待控件652可用于触发终端100云服务器200发送取消请求，该取消请求用于请求云服务器200取消对终端100的上传图片的增强。
当云服务器200开始处理终端100的图片增强任务时，云服务器200可以向终端100发送开始处理响应，终端100接收到该开始处理响应后，可以显示正在处理提示，该正在处理提示可用于提示用户当前云服务器200正在处理图片增强任务。
示例性的，如图6K所示，终端100在云服务器200处理终端100的图片增强任务时，可以显示正在处理提示653。例如，该正在处理提示653可以是文字提示“云端正在处理，请等待，共1张图”等等。示例仅仅用于解释本申请，不应构成限定。
在一些实施例中，若终端100在上传已选中的图片和该图片的环境信息到云服务器200上后，若终端100断开连接与云服务器200断开了网络连接，终端100可以输出网络断开提示，该网络断开提示可用于提示用户当前网络已断开，需将连接上网络，以获取图片增强结果。
示例性的，如图6L所示，当终端100在上传已选中的图片和该图片的环境信息到云服务器200上后，终端100的网络断开，无法连接至云服务器200。终端100可以在状态栏511中显示网络断开标识符643，用于提示用户当前网络断开。终端100还可以显示网络断开提示654。例如，该网络断开提示654可以是文字提示“检测到您当前的网络已经断开，请重新连接网络，以获取云端增强后的图片”。示例仅仅用于解释本申请，不应构成限定。
如图6M所示，当终端100借助云服务器200的处理能力完成对用户选中图片的增强后，终端100可以在云增强界面550上显示增强成功提示窗口660。该增强成功提示窗口 660可以显示包括有对已选中图片增强的效果描述661。例如该效果描述661对应的文字内容可以为“您选择的图片地点为‘上海·外滩’，已通过云端使图片中大楼的细节更突出、更清晰，提升图片动态范围以及画面色彩，尽显都市风貌！”。该增强成功提示窗口660中显示包括有增强后的图片对应的缩略图662、查看控件663、下载控件664，全部下载控件665。其中，该查看控件663可用于查看增强后的图片。下载控件664可用于将增强后的图片保存到终端100本地指定存储路径。该全部下载控件665可用于在增强后的图片有多张时，同时保存这多张增强后的图片到终端100本地指定存储路径。
终端100可以接收用户针对该查看控件663的输入操作(例如单击)，响应于该输入操作，终端100可以显示如图6N所示的图片浏览界面670。
如图6N所示，该图片浏览界面670中显示包括有增强后的图片671，标记672，下载控件673，效果对比控件674，图片相关信息675，菜单676。其中，该标记672用于提示用户该图片671已借助云服务器200的处理能力进行增强。该图片相关信息675可以包括有该图片671的拍摄时间、拍摄天气、地理位置信息等等。例如，拍摄时间可以为“2019年12月1日8：00AM”，拍摄天气可以为“多云”，拍摄地点可以为“上海·外滩”等等。该菜单676可以包括分享按钮、收藏按钮、编辑按钮、删除按钮、更多按钮。分享按钮可用于触发对该图片671的分享。该收藏按钮可用于触发收藏该图片671到图片收藏文件夹。编辑按钮可用于触发对图片671的旋转、修剪、增加滤镜、虚化等编辑功能。删除按钮可用于触发删除该图片671。更多按钮可用于触发打开更多与该图片671相关的功能。该效果对比控件674可用于触发在图片671的位置上显示该图片671增强前的图片(即用户选中的图片)。
终端100可以接收用户针对效果对比控件674的输入操作(例如单击)，响应于该输入操作，如图6O所示，终端100可以将用户选中的图片677显示在图片671的显示位置区域。
终端100可以再次接收用户针对效果对比控件674的输入操作(例如单击)，响应于该输入操作，如图6P所示，终端100可以显示增强后的图片671。
如图6P所示，终端100可以接收用户针对下载控件673的输入操作(例如单击)，响应于该输入操作，终端100可以将该增强后的图片671保存至到终端100本地指定存储路径，并在保存完成后显示保存完成提示678，该保存完成提示678用于提示用户已保存该增强后的图片671。
在一些实施例中，终端100在拍照后，可以将拍摄到的图片存到本地。用户可以在终端100的图库应用中查看终端100拍摄的图片以及从其他设备或网络上获取到的图片。终端100可以标记出图库应用中的待增强的图片、已增强过的图片、和增强后的图片。其中，当图片的环境信息(例如，地理位置信息、天气信息、时间信息、拍摄参数、拍摄姿态信息等等)齐全时，且未通过云服务器200增强过，则该图片可以归为上述待增强的图片。这样，可以方便用户快速选择出需要借助云服务器200的处理能力进行增强的图片，节约用户的时间。
示例性的，如图7A所示，终端100可以显示图库相册界面520。其中，该图库相册界 面520可以显示包括有一个或多个相册(例如，所有照片相册526、视频相册、相机相册、云增强相册527、微信相册、微博相册等等)。针对该图库相册界面520的详细文字说明可以参考前述图5B所示实施例，在此不再赘述。
终端100可以接收用户作用于所有照片相册526的输入操作(例如单击)，响应于该输入操作，终端100可以显示如图7B所示的所有照片界面710。
如图7B所示，该所有照片界面710显示包括有一张或多张图片的缩略图。其中，终端100可以标记出该一张或多张图片中待增强的图片、已增强过的图片和增强后的图片。例如，缩略图711对应的图片已增强过，终端100借助云服务器200的处理能力对缩略图711对应的图片增强后得到缩略图713对应的图片。缩略图715对应的图片为待增强的图片。终端100可以在缩略图711上显示标记712，该标记712用于提示用户该缩略图711对应的图片已经借助云服务器200的处理能力增强过。终端100可以在缩略图713上显示标记714，该标记714用于提示用户该缩略图713对应的图片为借助云服务器200的处理能力增强后的图片。终端100可以在缩略图715上显示标记716，该标记716用于提示用户该缩略图715对应的图片等待借助云服务器200的处理能力进行增强。上述示例仅仅用于解释本申请，不应构成限定。
终端100可以接收用户作用于缩略图715的输入操作(例如单击)，响应于该输入操作，终端100可以显示如图7C所示的图片浏览界面720。由于缩略图715对应的图片等待借助云服务器200的处理能力进行增强，终端100可以在该图片浏览界面720中显示该缩略图715对应的图片以及云增强控件，该云增强控件用于触发终端100借助云服务器200的处理能力对该缩略图715对应的图片进行增强。
如图7C所示，由于缩略图715对应的图片等待借助云服务器200的处理能力进行增强，图片浏览界面720可以显示包括有该缩略图715对应的图片721，云增强控件722，该图片相关信息723，菜单724。该图片相关信息723可以包括有该图片721的拍摄时间、拍摄天气、地理位置信息等等。例如，拍摄时间可以为“2019年12月1日8：00AM”，拍摄天气可以为“多云”，拍摄地点可以为“上海·外滩”等等。该菜单724可以包括分享按钮、收藏按钮、编辑按钮、删除按钮、更多按钮。分享按钮可用于触发对该图片721的分享。该收藏按钮可用于触发收藏该图片721到图片收藏文件夹。编辑按钮可用于触发对图片721的旋转、修剪、增加滤镜、虚化等编辑功能。删除按钮可用于触发删除该图片721。更多按钮可用于触发打开更多与该图片721相关的功能。该效果对比控件722可用于触发终端100借助云服务器200的处理能力对该图片721进行增强。
终端100可以接收用户针对该云增强控件722的输入操作(例如单击)，响应于该输入操作，终端100可以显示如图7D所示的云增强界面730。
如图7D所示，该云增强界面730可以显示图片添加控件731、已选中图片的缩略图(例如缩略图732)、删除控件733、了解详情控件734、上传控件735。其中，由于云增强界面730的与上述图6E所示的云增强界面550相同，因此，针对该云增强界面730的详细文字描述可以参考前述图6E所示实施例，在此不再赘述。
终端100可以接收用户作用于上传控件735的输入操作，响应于该操作，终端100可以将用户选中的图片以及图片的环境信息上传至云服务器200，借助云服务器200的处理 能力增强用户选中图片(即图7C中所示的图片721)的效果。
下面介绍本申请实施例中终端100如何借助云服务器200的处理能力对图片进行增强。
请参考图8，图8示出了本申请实施例提供的一种图像处理方法的流程示意图。如图8所示，该方法包括：
S801、终端100接收用户的第一输入。
S802、终端100响应于第一输入，获取第一图像以及第一图像的第一环境信息。
其中，第一环境信息可以包括地理位置信息、天气信息、季节信息、时间信息、拍摄姿态信息、拍摄参数等等中的一个或多个。其中，拍摄参数可以包括变焦倍数等等。拍摄姿态信息可以包括拍摄设备的拍摄镜头在拍摄图像时的朝向信息。例如，拍摄镜头的俯仰角以及方向角度等等，或者自然坐标下的三轴旋转角等等。
其中，在用户打开终端100的相机应用通过云增强模式进行拍照的场景中，该第一输入可以是上述图3E所示实施例中，用户在拍摄界面320中开启云增强模式后，针对拍摄控件322的输入操作(例如单击)。该第一图像可以是上述图3E中所示的摄像头捕捉的画面324(也即拍摄图像)。终端100可以从一个或多个传感器上以及网络上获取到该第一图像的第一环境信息。
示例性的，当地理位置信息为GPS位置信息时，终端100以通过GPS定位模块获取到拍摄图像的GPS位置信息。当地理位置信息为地理名称时，终端100可以通过GPS定位模块获取到拍摄图像的GPS位置信息后，可以通过离线地图或者网络在线地图，查询到该GPS位置信息对应的地理名称。终端100还可以从网络上，结合拍摄时间和地理位置信息获取到该拍摄图像的天气信息、以及季节信息等。终端100还可以通过陀螺仪和加速传感器等传感器，获取到该拍摄图像的拍摄姿态信息。该拍摄参数可以包括变焦倍率等等，终端100可以通过相机应用获取到用户设置的拍摄参数等等。
在用户打开终端100的图库应用选择图片进行增强的场景中，该第一输入可以是上述图6I所示实施例中，用户在云增强界面550中选择缩略图631对应的图片后，针对上传控件633的输入操作。或者，第一输入可以是上述图7D所示实施例中，用户在云增强界面730中选择缩略图732对应的图片后，针对上传控件735的输入操作。该第一图像可以为上述图6I所示实施例中缩略图631对应的图片或者上述图7D所示实施例中缩略图732对应的图片。
其中，由于拍摄设备(终端100或者其他设备)在拍摄完图片在保存时，可以将该图片的第一环境信息(例如，地理位置信息、天气信息、时间信息、季节信息、拍摄参数、拍摄姿态信息等等)保存至图片的可交换图像文件格式(exchange image file format，Exif)数据中。因此，当上述第一图像为用户选中终端100本地已保存的图片时，终端100可以从用户所选中图片对应的Exif数据中获取到该用户选中图片的第一环境信息。
在一种可能实现中，终端100可以在图库应用中标记出待增强的图片、已增强过的图片和增强后的图片。其中，当图片的第一环境信息齐全时，且未通过云服务器200增强过，则该图片可以归为上述待增强的图片。当图片已经被用户选中借助云服务器200的处理能力进行优化，并生成有对应的高质量图片时，该图片归为上述已增强过的图片。上述增强 后的图片可以指用户选中的图片借助云服务器200的处理能力进行优化后生成的高质量图片。这样，可以方便用户快速选择出需要借助云服务器200的处理能力进行增强的图片，节约用户的时间。
S803、终端100发送第一图像以及第一图像的第一环境信息给云服务器200。
其中，在终端100获取到第一图像以及第一环境信息时，终端100可以检测当前的网络状态，若网络状态断开，无法连接至云服务器200时，终端100可以输出提示，以提示用户在连接网络后才能借助云服务器200对第一图像进行增强。具体内容就可以参考前述图4A所示实施例或图6F所示实施例，在此不再赘述。
S804、终端100根据第一环境信息，从图像处理模型库中确定与第一环境信息匹配的第一图像处理模型。
其中，由于在不同拍摄环境下，图像的成像效果以及风格会有不同，因此，可以对针对不同的拍摄环境训练出不同的图像处理模型。这样，可以针对特定的拍摄环境，使用特定的图像处理模型对图像进行优化，提升图像的质量。
该图像处理模型库中存储有训练好的多种拍摄环境对应的图像处理模型。该拍摄环境可以通过地理位置信息、天气信息、季节信息、时间信息等等中的一种或多种进行区分。
示例性的，例如，图像处理模型库中可以根据拍摄地点和拍摄天气，划分有多个图像处理模型。该图像处理模型库中存储的多种拍摄环境对应的图像处理模型可以如下表1所示：
表1
Figure PCTCN2020136178-appb-000001
由上表1可以看出，拍摄地点为“上海·外滩”、拍摄天气为“晴天”的拍摄环境对应图像处理模型1。拍摄地点为“上海·外滩”、拍摄天气为“多云”的拍摄环境对应图像处理模型2。拍摄地点为“上海·外滩”、拍摄天气为“其他天气”的拍摄环境对应图像处理模型3。拍摄地点为“北京·故宫”、拍摄天气为“晴天”的拍摄环境对应图像处理模型4。拍摄地点为“北京·故宫”、拍摄天气为“雪天”的拍摄环境对应图像处理模型5。拍摄地点为“北京·故宫”、拍摄天气为“其他天气”的拍摄环境对应图像处理模型6，等等。上述表1所示示例仅仅用于解释本申请，不应构成限定。
例如，结合表1所示的图像处理模型库中包括的多种拍摄环境对应的图像处理模型， 当第一图像的第一环境信息中的地理位置信息为“上海·外滩”，天气信息为“多云”时，云服务器200从图像处理模型库中确定出与第一环境信息的第一图像处理模型可以为“图像处理模型2”。示例仅仅用于解释本申请，不应构成限定。
S805、云服务器200将第一图像处理模型发送给终端100。
云服务器200在确定出第一环境信息匹配的第一图像处理模型后，可以将第一图像处理模型发送给终端100。
S806、终端100将根据第一图像处理模型对第一图像进行优化处理，得到第二图像。
其中，终端100在接收到云服务器200发送的第一图像处理模型后，终端100可以将第一图像输入到该第一图像处理模型中，经过第一图像处理模型处理，可以得到第二图像。该第二图像与第一图像相比，第二图像的细节清晰度、动态范围/对比度、噪声和色彩等有所提升。
S807、云服务器200根据第一图像以及第一环境信息，从高清图像库中检索出第一高清图像。其中，第一高清图像的第二环境信息与第一图像的第一环境信息之间的相似度在第一预设范围内，第一高清图像中的拍摄内容与所述第一图像中的拍摄内容之间的相似度在第二预设范围内。
其中，该高清图像库中存储有多张高清图像以及高清图像的第二环境信息。云服务器200在接收到终端100发送的第一图像和第一环境信息后，云服务器200可以根据第一环境信息从高清图像库中先选择出第二环境信息与第一环境信息的相似度在第一预设范围的高清图像。然后，云服务器200可以将第一图像与该拍摄环境相似的高清图像进行对比，从该拍摄环境相似的高清图像中筛选出与第一图像中拍摄内容的相似度在第二预设范围内的第一高清图像。
示例性的，高清图像库中存储的高清图像以及高清图像的第二环境信息可以如图下表2所示：
表2
Figure PCTCN2020136178-appb-000002
Figure PCTCN2020136178-appb-000003
由上表2可以看出，高清图像库中高清图像1的拍摄地点为“上海·外滩”，天气为“晴”，拍摄时间为“下午”。高清图像2的拍摄地点为“上海·外滩”，天气为“多云”，拍摄时间为“早上”。高清图像3的拍摄地点为“上海·外滩”，天气为“多云”，拍摄时间为“早上”。高清图像4的拍摄地点为“上海·外滩”，天气为“多云”，拍摄时间为“晚上”。高清图像5的拍摄地点为“北京·故宫”，天气为“晴”，拍摄时间为“早上”。高清图像6的拍摄地点为“北京·故宫”，天气为“小雨”，拍摄时间为“下午”。等等。上述表2所示示例仅仅用于解释申请，不应构成限定。
例如，第一图像的第一环境信息中拍摄地点可以为“上海·外滩”，天气可以为“多云”，拍摄时间可以为“早上”。由于上述表2中所示，高清图像2的拍摄地点为“上海·外滩”，天气为“多云”，拍摄时间为“早上”。高清图像3的拍摄地点为“上海·外滩”，天气为“多云”，拍摄时间为“早上”。因此，云服务器200可以确定出与第一图像拍摄环境相同的高清图像有高清图像2和高清图像3。然后，云服务器200可以从高清图像2和高清图像3中筛选出拍摄内容与第一图像中拍摄内容的相似度在预设范围内的第一高清图像。例如，高清图像2中的拍摄内容与第一图像中的拍摄内容相似度为90％，高清图像3中的拍摄内容与第一图像中的拍摄内容相似度为40％，预设范围为80％，因此，云服务器200可以将该高清图像3确定为第一高清图像。上述示例仅仅用于解释申请，不应构成限定。
S808、云服务器200根据第一图像对所述第一高清图像进行图像配准，得到第二高清图像。
其中，云服务器200可以提取第一图像和第一高清图像中匹配的特征点，然后，通过匹配特征点对，得到图像空间坐标变换参数。然后，云服务器200可以根据该图像空间坐标变换参数进行图像配准。
S809、云服务器200可以将第二高清图片发送给终端100。
S810、终端100可以将第二图像与第二高清图像进行图像融合，得到第三图像。
终端100在接收到云服务器200发送的第二高清图像后，可以通过图像融合算法对比第二图像和第二高清图像中的特征信息，并将第二高清图像中的具有一定优势的特征信息融合到第二图像上，得到第三图像(即对第一图像增强后的图像)。例如，可以将第二高清图像中清晰度高的细节、色彩丰富的特征点等等融合到第二图像中，以得到第三图像。这样，可以提升图像的细节清晰度、色彩、白平衡等等。
在一种可能的实现方式中，云服务器200中可以包括有3D世界模型和渲染引擎。云服务器200在接收到终端100发送的第一图像和第一环境信息后，可以根据第一环境信息从3D世界模型中确定出第一图像的拍摄环境，并通过渲染引擎，渲染3D世界模型中与第一图像的拍摄环境相同的高清图像，并将该渲染出的高清图像作为上述第一高清图像，用于后续的图像配准以及图像配准后的图像融合。在另一种可能的实现方式中，云服务器200还可以预先渲染出3D世界模型中每个地理区域中每个拍摄角度的高清图像，当云服务器200接收到终端100发送的第一图像和第一环境信息后，云服务器200可以根据该第一环 境信息，从渲染出的多张高清图像中查找到与第一图像拍摄角度一致的高清图像，并将其确定为上述第一高清图像，用于后续的图像配准以及图像配准后的图像融合。
在一些实施例中，在云服务器200根据第一图像的第一环境信息从图像处理模型库中确定出第一图像处理模型后，云服务器200可以不用将第一图像处理模型发送给终端100，而是直接根据第一图片处理模型对高清图像进行优化处理，得到第二图像。在云服务器200根据第一图像以及第一环境信息从高清图像库中检索出第一高清图像，并对第一高清图像进行图像配准，得到第二高清图像后，不用发送第二高清图像给终端100。云服务器200可以将第二图像与第二高清图像进行图像融合，得到第三图像(即对第一图像增强后的图像)后，再将该第三图像发送给终端100。这样，可以极大节约终端100的计算资源，使终端100能够突破终端自身成像质量极限，使得终端上的普通摄像头也能拍出专业单反设备的效果。
在一些实施例中，终端100可以在响应用户的第一输入后，只将第一图像的第一环境信息发送给云服务器200。云服务器200在接收到终端100发送的第一环境信息后，可以从图像处理模型库中确定出与第一环境信息匹配的第一图像处理模型，并将第一图像处理模型发送给终端100。终端100在接收到第一图像处理模型后，可以通过第一图像处理模型对第一图像进行优化处理得到第二图像，作为增强后的图像保存至本地。
在一些实施例中，终端100可以响应用户的第一输入，将第一图像以及第一图像的第一环境信息发送给云服务器200。云服务器200在接收到终端100发送的第一图像以及第一环境信息后，可以从图像处理模型库中确定出与第一环境信息匹配的第一图像处理模型，并通过第一图像处理模型对第一图像进行优化处理，得到第二图像。云服务器200可以将该第二图像发送给终端100，作为增强后的图像保存至终端100本地。
在一些实施例中，终端100可以在响应用户的第一输入后，将第一图像以及第一图像的第一环境信息发送给云服务器200。云服务器200可以根据第一环境信息从高清图像库中检索出与第一高清图像，并将根据第一图像对第一高清图像进行图像配准，得到第二高清图像。云服务200可以将第二高清图像发送至终端100上，由终端100将第二高清和第一图像进行图像融合，得到增强后的图像保存至终端100的本地。在一种可能的实现方式中，云服务器200还可以在得到第二高清图像后，直接与第一图像进行图像融合，并将图像融合后的图像发送给终端100。
通过本申请实施例提供一种图像处理方法，可以结合终端和云服务器的处理能力，充分利用高质量图像数据和图像处理算法，赋能终端拍照成像系统。云服务器可以通过终端上传的图像以及图像的环境信息(例如地理位置信息、天气信息、季节信息、时间信息、拍摄参数、拍摄姿态信息等等)，匹配出适合该图像的图像处理算法以及与该图像拍摄内容相同，拍摄环境相近的高质量图像数据。通过该图像处理算法以及该高质量图像数据，可以增强终端上传图像的质量和效果。这样，可以让不同环境下拍摄的图片，呈现出不同的 风格和效果。这样也使终端能够突破终端自身成像质量极限，使得终端上的普通摄像头也能拍出专业单反设备的效果，从而在降低手机成本的同时，极大的提升终端的拍照质量和用户拍照体验。
下面介绍本申请涉及的一种图像处理系统900。
请参考图9，图9示出了本申请实施例中提供的一种图像处理系统900的架构示意图。如图9所示，该图像处理系统900可以包括终端100和云服务器200。
其中，该终端100可以包括摄像头911、图像信号处理器912、同源处理模块913、异源融合模块914。
该摄像头911可用于在用户拍照时，捕获光信号，并将光信号转换为电信号，发送给图像信号处理器912。
该图像信号处理器912可以将摄像头911发送的电信号转换成数字图像信号，也称为第一图像，其中，该第一图像的格式可以是Raw图，YUV图或RGB图等等。该图像信号处理器912还可以从终端100的一个或多个传感器(图中未示出)以及终端100连接的网络上获取到该第一图像的第一环境信息(包括地理位置信息、天气信息、季节信息、时间信息、拍摄参数、拍摄姿态信息等等中的一种或多种)。
该同源处理模块913，可用于根据云服务器200发送的第一图像处理模型对图像处理器912生成的第一图像进行优化处理，得到第二图像。
该异源融合模块914，可用于将同源处理模块913处理得到的第二图像以及云服务器200发送的第二高清图像进行图像融合，得到第三图像。
其中，该云服务器200可以包括图像处理模型库921、模型选择模块922、高清图像库923、图像检索模块924、图像配准模块925。
该图像处理模型库921，可用于存储有根据拍摄环境划分的多个图像处理模型。
该模型选择模块922，可用于根据终端100发送的第一环境信息中的数据1，从图像处理模型库921中确定出与数据1匹配的第一图像处理模型。其中，数据1可以为第一环境信息中的全部或部分信息。例如，当第一环境信息包括有地理位置信息、天气信息、季节信息、时间信息、拍摄参数和拍摄姿态信息时，数据1可以包括有地理位置信息和天气信息。
该高清图像库923，可用于存储多张高清图像以及高清图像的第二环境信息。
该图像检索模块924，可用于根据第一图像和第一环境信息中的数据2，从该高清图像库923中检索出第一高清图像。其中，第一高清图像的拍摄环境与第一图像的拍摄环境相似度在第一预设范围内，第一高清图像中的拍摄内容与所述第一图像中的拍摄内容之间的相似度在第二预设范围内。其中，数据2可以为第一环境信息中的全部或部分信息。例如，当第一环境信息包括有地理位置信息、天气信息、季节信息、时间信息、拍摄参数和拍摄姿态信息时，数据2可以包括有地理位置信息、季节信息、时间信息和拍摄姿态信息。
该图像配准模块925，可用于根据第一图像对所述第一高清图像进行图像配准，得到第二高清图像。云服务器200可以将该第二高清图像发送给终端100。
通过本申请实施例提供一种图像处理系统900，可以结合终端和云服务器的处理能力，充分利用高质量图像数据和图像处理算法，赋能终端拍照成像系统。云服务器可以通过终端上传的图像以及图像的环境信息(例如地理位置信息、天气信息、时间信息、拍摄参数、拍摄姿态信息等等)，匹配出适合该图像的图像处理算法以及与该图像拍摄内容相同，拍摄环境相近的高质量图像数据。通过该图像处理算法以及该高质量图像数据，可以增强终端上传图像的质量和效果。这样，可以让不同环境下拍摄的图片，呈现出不同的风格和效果。这样也使终端能够突破终端自身成像质量极限，使得终端上的普通摄像头也能拍出专业单反设备的效果，从而在降低手机成本的同时，极大的提升终端的拍照质量和用户拍照体验。
下面介绍本申请涉及的一种图像处理系统1000。
请参考图10，图10示出了本申请实施例中提供的一种图像处理系统1000的架构示意图。如图10所示，该图像处理系统1000可以包括终端100和云服务器200。
其中，该终端100可以包括摄像头1011、图像信号处理器1012、同源处理模块1013、异源融合模块1014。
该摄像头1011可用于在用户拍照时，捕获光信号，并将光信号转换为电信号，发送给图像信号处理器1012。
该图像信号处理器1012可以将摄像头1011发送的电信号转换成数字图像信号，也称为第一图像，其中，该第一图像的格式可以是Raw图，YUV图或RGB图等等。该图像信号处理器1012还可以从终端100的一个或多个传感器(图中未示出)以及终端100连接的网络上获取到该第一图像的第一环境信息(包括地理位置信息、天气信息、季节信息、时间信息、拍摄参数、拍摄姿态信息等等中的一种或多种)。
该同源处理模块1013，可用于根据云服务器200发送的第一图像处理模型对图像处理器1012生成的第一图像进行优化处理，得到第二图像。
该异源融合模块1014，可用于将同源处理模块1013处理得到的第二图像以及云服务器200发送的第二高清图像进行图像融合，得到第三图像。
其中，该云服务器200可以包括图像处理模型库1021、模型选择模块1022、3D世界模型1023、渲染引擎1024、图像配准模块1025。
该图像处理模型库1021，可用于存储有根据拍摄环境划分的多个图像处理模型。
该模型选择模块1022，可用于根据终端100发送的第一环境信息的数据1，从图像处理模型库1021中确定出与数据1匹配的第一图像处理模型。其中，数据1可以为第一环境信息中的全部或部分信息。例如，当第一环境信息包括有地理位置信息、天气信息、季节信息、时间信息、拍摄参数和拍摄姿态信息时，数据1可以包括有地理位置信息和天气信息。
该3D世界模型1023，包括有现实世界的3D地图信息。
该渲染引擎1024，可用于根据第一图像的第一环境信息中的数据2，从该3D世界模型1023中渲染出第一高清图像。其中，第一高清图像中的拍摄内容与所述第一图像中的拍摄内容之间的相似度在预设范围内，且第一高清图像的拍摄环境与第一图像的拍摄环境相同。其中，数据2可以为第一环境信息中的全部或部分信息。例如，当第一环境信息包括 有地理位置信息、天气信息、季节信息、时间信息、拍摄参数和拍摄姿态信息时，数据2可以包括有地理位置信息、季节信息、时间信息和拍摄姿态信息。
该图像配准模块1025，可用于根据第一图像对所述第一高清图像进行图像配准，得到第二高清图像。云服务器200可以将该第二高清图像发送给终端100。
通过本申请实施例提供一种图像处理系统1000，可以结合终端和云服务器的处理能力，充分利用高质量图像数据和图像处理算法，赋能终端拍照成像系统。云服务器可以通过终端上传的图像以及图像的环境信息(例如地理位置信息、天气信息、时间信息、拍摄参数、拍摄姿态信息等等)，匹配出适合该图像的图像处理算法以及与该图像拍摄内容相同，拍摄环境相近的高质量图像数据。通过该图像处理算法以及该高质量图像数据，可以增强终端上传图像的质量和效果。这样，可以让不同环境下拍摄的图片，呈现出不同的风格和效果。这样也使终端能够突破终端自身成像质量极限，使得终端上的普通摄像头也能拍出专业单反设备的效果，从而在降低手机成本的同时，极大的提升终端的拍照质量和用户拍照体验。
下面介绍本申请涉及的一种图像处理系统1100。
请参考图11，图11示出了本申请实施例中提供的一种图像处理系统1100的架构示意图。如图11所示，该图像处理系统1100可以包括终端100和云服务器200。
其中，该终端100可以包括摄像头1111、图像信号处理器1112。
该摄像头1111可用于在用户拍照时，捕获光信号，并将光信号转换为电信号，发送给图像信号处理器1112。
该图像信号处理器1112可以将摄像头1111发送的电信号转换成数字图像信号，也称为第一图像，其中，该第一图像的格式可以是Raw图，YUV图或RGB图等等。该图像信号处理器1112还可以从终端100的一个或多个传感器(图中未示出)以及终端100连接的网络上获取到该第一图像的第一环境信息(包括地理位置信息、天气信息、季节信息、时间信息、拍摄参数、拍摄姿态信息等等中的一种或多种)。
其中，该云服务器200可以包括图像处理模型库1121、模型选择模块1122、高清图像库1123、图像检索模块1124、图像配准模块1125、同源处理模块1126、异源融合模块1127。
该图像处理模型库1121，可用于存储有根据拍摄环境划分的多个图像处理模型。
该模型选择模块1122，可用于根据终端100发送的第一环境信息中的数据1，从图像处理模型库1121中确定出与数据1匹配的第一图像处理模型。其中，数据1可以为第一环境信息中的全部或部分信息。例如，当第一环境信息包括有地理位置信息、天气信息、季节信息、时间信息、拍摄参数和拍摄姿态信息时，数据1可以包括有地理位置信息和天气信息。
该高清图像库1123，可用于存储多张高清图像以及高清图像的第二环境信息。
该图像检索模块1124，可用于根据第一图像和第一图像的第一环境信息中的数据2，从该高清图像库1123中检索出第一高清图像。其中，第一高清图像的拍摄环境与第一图像的拍摄环境的相似度在第一预设范围内，且第一高清图像中的拍摄内容与所述第一图像中的拍摄内容之间的相似度在预设范围内。其中，数据2可以为第一环境信息中的全部或部 分信息。例如，当第一环境信息包括有地理位置信息、天气信息、季节信息、时间信息、拍摄参数和拍摄姿态信息时，数据2可以包括有地理位置信息、季节信息、时间信息和拍摄姿态信息。
该图像配准模块1125，可用于根据第一图像对所述第一高清图像进行图像配准，得到第二高清图像。
该同源处理模块1126，可用于根据模型选择模块1122选择出的第一图像处理模型对终端100发送的第一图像进行优化处理，得到第二图像。
该异源融合模块1127，可用于将同源处理模块1126处理得到的第二图像以及第二高清图像进行图像融合，得到第三图像。云服务器200可以将该第三图像发送给终端100。
通过本申请实施例提供一种图像处理系统1100，可以结合终端和云服务器的处理能力，充分利用高质量图像数据和图像处理算法，赋能终端拍照成像系统。云服务器可以通过终端上传的图像以及图像的环境信息(例如地理位置信息、天气信息、时间信息、拍摄参数、拍摄姿态信息等等)，匹配出适合该图像的图像处理算法以及与该图像拍摄内容相同，拍摄环境相近的高质量图像数据。通过该图像处理算法以及该高质量图像数据，可以增强终端上传图像的质量和效果。这样，可以让不同环境下拍摄的图片，呈现出不同的风格和效果。这样也使终端能够突破终端自身成像质量极限，使得终端上的普通摄像头也能拍出专业单反设备的效果，从而在降低手机成本的同时，极大的提升终端的拍照质量和用户拍照体验。
下面介绍本申请涉及的一种图像处理系统1200。
请参考图12，图12示出了本申请实施例中提供的一种图像处理系统1200的架构示意图。如图12所示，该图像处理系统1200可以包括终端100和云服务器200。
其中，该终端100可以包括摄像头1211、图像信号处理器1212。
该摄像头1211可用于在用户拍照时，捕获光信号，并将光信号转换为电信号，发送给图像信号处理器1212。
该图像信号处理器1212可以将摄像头1211发送的电信号转换成数字图像信号，也称为第一图像，其中，该第一图像的格式可以是Raw图，YUV图或RGB图等等。该图像信号处理器1212还可以从终端100的一个或多个传感器(图中未示出)以及终端100连接的网络上获取到该第一图像的第一环境信息(包括地理位置信息、天气信息、季节信息、时间信息、拍摄参数、拍摄姿态信息等等中的一种或多种)。
其中，该云服务器200可以包括图像处理模型库1221、模型选择模块1222、3D世界模型1223、渲染引擎1224、图像配准模块1225、同源处理模块1226、异源融合模块1227。
该图像处理模型库1221，可用于存储有根据拍摄环境划分的多个图像处理模型。
该模型选择模块1222，可用于根据终端100发送的第一环境信息中的数据1，从图像处理模型库1221中确定出与第一环境信息中的数据1匹配的第一图像处理模型。其中，数据1可以为第一环境信息中的全部或部分信息。例如，当第一环境信息包括有地理位置信息、天气信息、季节信息、时间信息、拍摄参数和拍摄姿态信息时，数据1可以包括有地理位置信息和天气信息。
该3D世界模型1223，包括有现实世界的3D地图信息。
该渲染引擎1224，可用于根据第一图像的第一环境信息的数据2，从该3D世界模型1223中渲染出第一高清图像。其中，第一高清图像中的拍摄内容与所述第一图像中的拍摄内容之间的相似度在预设范围内，且第一高清图像的拍摄环境与第一图像的拍摄环境相同。其中，数据2可以为第一环境信息中的全部或部分信息。例如，当第一环境信息包括有地理位置信息、天气信息、季节信息、时间信息、拍摄参数和拍摄姿态信息时，数据2可以包括有地理位置信息、季节信息、时间信息和拍摄姿态信息。
该图像配准模块1225，可用于根据第一图像对所述第一高清图像进行图像配准，得到第二高清图像。
该同源处理模块1226，可用于根据模型选择模块1222选择出的第一图像处理模型对终端100发送的第一图像进行优化处理，得到第二图像。
该异源融合模块1227，可用于将同源处理模块1226处理得到的第二图像以及第二高清图像进行图像融合，得到第三图像。云服务器200可以将该第三图像发送给终端100。
通过本申请实施例提供一种图像处理系统1200，可以结合终端和云服务器的处理能力，充分利用高质量图像数据和图像处理算法，赋能终端拍照成像系统。云服务器可以通过终端上传的图像以及图像的环境信息(例如地理位置信息、天气信息、时间信息、拍摄参数、拍摄姿态信息等等)，匹配出适合该图像的图像处理算法以及与该图像拍摄内容相同，拍摄环境相近的高质量图像数据。通过该图像处理算法以及该高质量图像数据，可以增强终端上传图像的质量和效果。这样，可以让不同环境下拍摄的图片，呈现出不同的风格和效果。这样也使终端能够突破终端自身成像质量极限，使得终端上的普通摄像头也能拍出专业单反设备的效果，从而在降低手机成本的同时，极大的提升终端的拍照质量和用户拍照体验。
下面介绍本申请实施例中涉及数据构建系统。
请参考图13，图13示出了本申请实施例中提供的一种数据构建系统1300的架构示意图。如图13所示，该数据构建系统1300可以包括数据质量评估模块1301、高清图像库1302、待增强的图像数据库1303、图像检索模块1304、图像配准模块1305、图像配对模块1306、验证模块1307、图像处理模型库1308。
其中，专业摄影师或者摄影爱好者可以上传图像及其环境信息到数据质量评估模块1301。
该数据质量评估模块1301可用于评估专业摄影师或者摄影爱好者上传图像的图像质量，去除掉其中低质量的图像，例如，低清晰度或环境信息不齐全的图像，筛选出高清图像及其环境信息上传到高清图像库1302中。
该待增强的图像数据库1303中可以存储有多张待增强的图像及其环境信息。
图像检索模块1304可用于从待增强的图像数据库1303取出待增强的图像及其环境信息，并从高清图像数据1302中检索出与该待增强的图像环境信息相同，拍摄内容相近的高清图像。
图像检索模块1305，可用于根据待增强的图像对高清图像进行图像配准，得到高清训 练图。
图像配对模块1306，可用于将该待增强的图像和高清训练图绑定为训练图像对。
验证模块1307，可用于过滤掉对准度偏差较大的图像对，并将对准度偏差较小的图像对，按照环境信息，分类相应的训练集类别对图像处理模型库1308中的图像处理模型进行训练。
通过本申请实施例提供一种数据构建系统1300，可以实现对云服务器200中的图像处理模型库的优化和更新，提高了图像处理模型的训练效率。
下面介绍本申请实施例涉及的云服务器200的结构示意图。请参阅图14，图14是本申请实施例提供的一种云服务器200的结构示意图。该云服务器200应用于上述图9所描述的图像处理系统900或上述图10所描述的图像处理系统1000或上述图11所描述的图像处理系统1100或上述图12所描述的图像处理系统1200。
如图14所示，该云服务器200包括一个或多个处理器201A、通信接口202A、存储器203A，处理器201A、通信接口202A、存储器203A可通过总线或者其它方式连接，本申请实施例以通过总线204 A连接为例。其中：
处理器201A可以由一个或者多个通用处理器构成，例如CPU。处理器201A可用于运行设备控制方法的相关的程序代码。
通信接口202A可以为有线接口(例如以太网接口)或无线接口(例如蜂窝网络接口或使用无线局域网接口)，用于与其他节点进行通信。本申请实施例中，通信接口202A具体可用于与电子设备100进行通信。
存储器203A可以包括易失性存储器(volatile memory)，例如RAM；存储器也可以包括非易失性存储器(non-vlatile memory)，例如ROM、快闪存储器(flash memory)、HDD或固态硬盘SSD；存储器203A还可以包括上述种类的存储器的组合。存储器203A可用于存储一组程序代码，以便于处理器201A调用存储器203A中存储的程序代码以实现本申请实施例的在服务器的实现方法。
需要说明的，图14所示的服务器仅仅是本申请实施例的一种实现方式，实际应用中，服务器还可以包括更多或更少的部件，这里不作限制。
下面结合图15至图17对本申请实施例的图像处理方法进行示例性说明。
请参照图15所示，为本申请实施例提供的图像处理方法的一种示意性流程图，在该实施例中，终端100将图像A和图像A的拍摄信息发送给云服务器200，云服务器200从高清图像库中确定参考图像1，该参考图像1不仅拍摄信息与图像A的拍摄信息相似，拍摄内容也与图像A的拍摄内容一致。云服务器200根据参考图像1对图像A进行细节恢复处理，得到图像C，并向终端100返回该图像C。如图15所示，该图像处理方法包括但不限于以下步骤：
S1501，终端100检测拍照指令。
其中，为了便于终端100获取拍摄的图像A的拍摄信息，可以打开该终端100的GPS， 以及将终端100连接网络。
可选的，终端100可安装用于上传图像A和图像A的拍摄信息的应用程序(application，APP)。可选的，该应用程序可以包括但不限于拍照应用、修图应用等等。终端100上的该应用程序还可以登陆账号。
用户可以通过触屏操作发起拍照指令，或者，用户也可以通过按键发起拍照指令。终端100检测拍照指令，并拍摄图像A。终端100还可以通过GPS获取该图像A的拍摄地理位置信息。可选的，终端100还可以通过网络获取该拍摄地理位置信息的天气状况信息，作为该图像A的拍摄天气状况信息。可选的，终端100还可以获取该图像A的拍摄倍率。
S1502，终端100拍摄图像A。
其中，由于成像设备的限制，在高倍率拍摄时，该图像A的分辨率比较低，细节比较模糊。
S1503，终端100将图像A和图像A的拍摄信息发送给云服务器。
在一些实施例中，终端100可以通过拍照应用或者修图应用将图像A和图像A的拍摄信息发送给云服务器，该云服务器与该拍照应用或者修图应用对应。
其中，该云服务器中可以配置高清图像库，该高清图像库中存储多个高清图像、该多个高清图像中每个高清图像的拍摄信息以及该多个高清图像中每个高清图像对应的第一特征和第二特征。其中，第一特征可以是通过检索网络提取的，第二特征可以是通过重排序网络提取的，该重排序网络可以是比检索网络更复杂的特征提取网络，因此通过重排序网络提取的第二特征比通过检索网络提取的第一特征更加精细，即第二特征可以表达图像的更多细节特征。在本申请实施例中，可以通过检索网络提取的第一特征作一个粗略的快速检索，而进一步通过重排序网络提取的第二特征作一个更加精确的检索，以确定拍摄内容最相似的参考图像1，具体请参照后续实施例的描述，暂不赘述。
示例性的，基于多个地理区域，分别利用高清拍照设备采集多个高清图像，从而将所采集的多个高清图像构成高清图像库。地理区域可以是以大的地理区域进行划分，例如，可以将深圳、广州、重庆等作为大的地理区域。
采用检索网络分别提取该多个高清图像中每个高清图像对应的第一特征，例如，可以将该多个高清图像中的每个高清图像分别输入该检索网络，从而分别得到每个高清图像对应的第一特征。
采用重排序网络分别提取该多个高清图像中每个高清图像对应的第二特征，例如，可以将该多个高清图像中的每个高清图像分别输入该重排序网络，从而分别得到每个高清图像对应的第二特征。
根据每个高清图像的拍摄信息(拍摄信息包括但不限于拍摄地理位置信息、拍摄天气状况信息以及拍摄倍率等等)，对该多个高清图像和该多个高清图像分别对应的第一特征和第二特征进行分类存储。其中，拍摄天气状况信息可以包括拍摄该高清图像时是晴天、阴天、雾天或者雨天等等。
其中，为了便于检索，可以根据拍摄各个高清图像的地理区域，将该多个高清图像分为多个大类别，例如，将地理区域属于深圳的所有高清图像划分在一个大类别中，将地理区域属于广州的所有高清图像划分在另一个大类别中，等等。可以理解的是，地理区域的 划分还可以是其他，例如，可以根据景点名称划分，本申请实施例不作限定。
按照各个高清图像的拍摄信息，将属于同一个地理区域的所有高清图像进一步分类。例如，可以将拍摄倍率相同或者相近的、GPS信息相同或者相近的以及拍摄时的天气状况信息相同的多个高清图片归为同一个类别。当然，在划分类别时，可以根据实际情况选择参考的拍摄信息，例如，可以将拍摄倍率相同或者相近的、GPS信息相同或者相近的多个高清图片归为同一个类别，而不关注拍摄时的天气状况信息，本申请实施例对此不作限定。
其中，上述拍摄倍率相同或者相近的可以是指，拍摄倍率之差的绝对值小于或者等于设置的倍率阈值。上述GPS信息相同或者相近可以是指，GPS信息所指示的地理位置之间的距离可以小于或者等于设置的距离阈值。
可以理解的是，本申请实施例的GPS信息均是指拍摄图像时，拍照设备(比如终端100或者多个地理区域部署的高清拍照设备)所处地理位置的GPS信息。
云服务器在分类存储各个高清图像时，可以将该高清图像对应的第一特征和第二特征一并存储，便于后续检索使用。需要说明的是，本申请实施例中，高清图像库中存储的高清图像的分辨率大于终端100拍摄的图像A的分辨率。
S1504，云服务器200根据图像A的拍摄信息，确定至少一个图像B。
在一些实施例中，云服务器200接收终端100发送的图像A和该图像A的拍摄信息，并根据图像A的拍摄信息，从高清图像库中确定至少一个图像B。该至少一个图像B中每个图像B的拍摄信息与图像A的拍摄信息之间的匹配度大于第一阈值。
其中，拍摄信息可以包括以下信息中的一种或者多种：拍摄倍率、拍摄地理位置信息或拍摄天气状况信息等。
可选的，在拍摄信息包括拍摄地理位置信息时，该至少一个图像B中每个图像B的拍摄地理位置信息与图像A的拍摄地理位置信息之间的匹配度大于第一阈值，例如，拍摄该图像B的地理位置与拍摄该图像A的地理位置之间的距离小于或者等于距离阈值。通过地理位置信息的匹配，可以保证后续检索得到的与图像A的拍摄内容最相似的参考图像1与图像A的拍摄角度的一致性。
可选的，在拍摄信息包括拍摄倍率时，该至少一个图像B中每个图像B的拍摄倍率与图像A的拍摄倍率之间的匹配度大于第一阈值，例如，拍摄该图像B的拍摄倍率与拍摄该图像A的拍摄倍率之差的绝对值小于或者等于倍率阈值。通过拍摄倍率的匹配，可以保证图像A和该至少一个图像B的放大倍率相差不大，相应的，也能保证后续检索得到的与图像A的拍摄内容最相似的参考图像1与图像A的放大倍率的一致性。
可选的，在拍摄信息包括拍摄天气状况信息时，该至少一个图像B中每个图像B的拍摄天气状况信息与图像A的拍摄天气状况信息之间的匹配度大于第一阈值，例如，都是晴天拍摄的，或者都是雨天拍摄的，或者都是雾天拍摄的。通过天气状况信息的匹配，可以保证图像A和该至少一个图像B的透明度的一致性，相应的，也能保证后续检索得到的与图像A的拍摄内容最相似的参考图像1与图像A的透明度的一致性。
可选的，高清图像库中的高清图像可以是根据拍摄信息进行分类存储的，具体请参照前述实施例的描述，在此不再赘述。云服务器200在根据图像A的拍摄信息，确定至少一个图像B时，可以根据图像A的拍摄信息，确定相应的类别，从而从该确定的类别中获取 该至少一个图像B。
例如，图像A的拍摄信息包括拍摄地理位置信息，则根据拍摄地理位置信息，确定该拍摄地理位置信息所属地理区域，再从属于该地理区域的高清图像中，确定拍摄信息高度匹配的至少一个图像B。其中，属于该地理区域的高清图像也可以是根据拍摄信息进行分类存储的，比如，将拍摄倍率相同或者相近的、GPS信息相同或者相近的以及拍摄时的天气状况信息相同的多个高清图片归为同一个类别，并将该拍摄倍率、GPS信息以及天气状况信息作为分类标签，云服务器200在确定至少一个图像B时，可以根据图像A的拍摄信息与各个分类标签之间的匹配，确定对应的类别，从而从该类别中确定该至少一个图像B，以及获取所存储的该至少一个图像B中每个图像B对应的第一特征和第二特征。
S1505，云服务器200从该至少一个图像B中确定拍摄内容与图像A的拍摄内容最相似的参考图像1。
在一些实施例中，云服务器200可以通过检索网络提取该图像A对应的第一特征，并分别计算该图像A对应的第一特征与该至少一个图像B中每个图像B对应的第一特征之间的相似度，得到相似度最大的M个图像B。可以理解的是，每个图像B对应的第一特征可以是预先通过该检索网络提取并存储在高清图像库中的，具体请参照前述实施例的描述，在此不再赘述。
示例性的，云服务器200可以将该图像A对应的第一特征与该至少一个图像B中每个图像B对应的第一特征之间的相似度，按照从大到小的顺序进行排序，并将排序在最前面的M个相似度对应的图像B作为相似度最大的M个图像B。
可以理解的是，通过检索网络提取的第一特征进行相似度的度量，可以得到与图像A的拍摄内容大致相似的M个图像B。
为了更加精准的从该M个图像B中确定拍摄内容与图像A的拍摄内容最相似的图像B，本申请实施例中，云服务器200可以进一步通过重排序网络提取该图像A对应的第二特征，并分别计算该图像A对应的第二特征与该M个图像B中每个图像B对应的第二特征之间的相似度，得到相似度最大的图像B，作为参考图像1。可以理解的是，每个图像B对应的第二特征可以是预先通过该重排序网络提取并存储在高清图像库中的，具体请参照前述实施例的描述，在此不再赘述。
示例性的，云服务器200可以将该图像A对应的第二特征与该M个图像B中每个图像B对应的第二特征之间的相似度，按照从大到小的顺序进行重排序，并将排序在最前面的相似度对应的图像B作为参考图像1。
需要说明的是，由于第二特征比第一特征更加精细，第二特征能表达图像的更多细节特征。因此，根据第一特征进行的相似度排序结果，与根据第二特征进行的相似度排序结果可能不同。可以理解为，可以通过第一特征的相似度匹配，得到大致相似的M个图像B，再通过第二特征的相似度匹配，得到最相似的参考图像1。该参考图像1不仅拍摄信息与图像A的拍摄信息匹配度比较高，而且拍摄内容也非常接近，因此，通过本申请实施例的方法，可以得到拍摄角度、拍摄内容以及图片透明度等非常相似的且分辨率比较高的参考图像1，从而更好的对分辨率较低的图像A进行细节恢复。
S1506，云服务器200根据参考图像1，对图像A进行细节恢复处理，获得图像C。
在一些实施例中，图像A的分辨率比较低，相应的，清晰度比较低。云服务器200在根据参考图像1，对图像A进行细节恢复处理获得图像C，该图像C的分辨率大于图像A的分辨率，相应的，图像C的清晰度大于图像A的清晰度。
可选的，可以将参考图像1和图像A输入深度学习神经网络，通过深度学习神经网络提取该参考图像1的高频信息，并通过该高频信息对图像A进行细节恢复，从而得到清晰度更高的图像C。
示例性的，在根据参考图像1，对图像A进行细节恢复处理之前，还可以确定该参考图像1的拍摄内容占图像A的拍摄内容的比例，其中，该比例可以反映所检索的参考图像1包含的拍摄内容是否覆盖全图像A所包含的拍摄内容。
若参考图像1的拍摄内容占图像A的拍摄内容的比例小于第二阈值，从该图像A的拍摄内容中去除与参考图像1的拍摄内容匹配的拍摄内容，获得去除后的图像A。其中，该第二阈值可以根据实际需要进行设定，例如，该第二阈值可以设置为80％，若参考图像1的拍摄内容占图像A的拍摄内容的比例小于80％，则说明参考图像1包含的拍摄内容未覆盖全图像A所包含的拍摄内容，可以进一步检索图像A包含的拍摄内容中除参考图像1包含的拍摄内容以外的拍摄内容，即去除后的图像A。
进一步，从拍摄信息高度匹配的至少一个图像B中确定拍摄内容与去除后的图像A的拍摄内容最相似的参考图像2。
其中，从至少一个图像B中确定拍摄内容与去除后的图像A的拍摄内容最相似的参考图像2的确定方法可以参照前述实施例中确定参考图像1的确定方法，在此不再赘述。
在获得参考图像1和参考图像2后，可以根据参考图像1和参考图像2，对图像A进行细节恢复处理，获得图像C。
可以理解的是，在获得参考图像1和参考图像2后，还可以进一步确定参考图像1的拍摄内容和参考图像2的拍摄内容占图像A的拍摄内容的比例，若该比例仍然小于第二阈值，还可以进一步从该图像A的拍摄内容中去除与参考图像1的拍摄内容以及与参考图像2的拍摄内容匹配的拍摄内容，获得去除后的图像A，再进一步从至少一个图像B中检索得到与该去除后的图像A的拍摄内容最相似的参考图像3。以此循环，直至所获得的至少一个参考图像的拍摄内容占图像A的拍摄内容的比例大于或者等于第二阈值，再根据所获得的至少一个参考图像，对图像A进行细节恢复处理，获得分辨率更高的图像C。
可以理解的是，上述至少一个参考图像的拍摄内容占图像A的拍摄内容的比例，可以理解为，该至少一个参考图像的拍摄内容与图像A的拍摄内容的交集占该图像A的拍摄内容的比例。
S1507，云服务器200将图像C发送给终端100。
S1508，终端100显示该图像C。
在一些实施例中，终端100接收图像C，并显示该图像C。可选的，为了便于用户比较，终端100可以显示该图像A和图像C，其中，图像C和图像A的拍摄内容相同，但是，图像C的清晰度更高。
通过实施图15所述实施例，终端仅需将拍摄的分辨率比较低的图像A发送给云服务器，由云服务器检索得到拍摄信息高度匹配、且拍摄内容高度相似的参考图像，并由云服 务器根据参考图像对图像A进行细节恢复处理，得到图像C，返回至终端，该实施方式中，可以降低对终端的配置要求，兼容性强。
请参照图16所示，为本申请实施例提供的图像处理方法的另一种示意性流程图，在该实施例中，终端100将图像A的拍摄信息发送给云服务器200，云服务器200根据图像A的拍摄信息从高清图像库中确定至少一个图像B，该图像B的拍摄信息与图像A的拍摄信息之间的匹配度大于第一阈值。云服务器200向终端100发送该至少一个图像B，并由终端100从该至少一个图像B中确定拍摄内容与图像A的拍摄内容最相似的参考图像1。如图16所示，该图像处理方法包括但不限于以下步骤：
S1601，终端100检测拍照指令。
S1602，终端100拍摄图像A。
其中，步骤S1601-步骤S1602请参照图15实施例的步骤S1501-S1502，在此不再赘述。
S1603，终端100将图像A的拍摄信息发送给云服务器200。
在一些实施例中，终端100可以通过拍照应用或者修图应用将图像A的拍摄信息发送给云服务器，该云服务器与该拍照应用或者修图应用对应。
其中，该云服务器中可以配置高清图像库，该高清图像库中存储多个高清图像、该多个高清图像中每个高清图像的拍摄信息。示例性的，基于多个地理区域，分别利用高清拍照设备采集多个高清图像，从而将所采集的多个高清图像构成高清图像库。地理区域可以是以大的地理区域进行划分，例如，可以将深圳、广州、重庆等作为大的地理区域。
根据每个高清图像的拍摄信息(拍摄信息包括但不限于拍摄地理位置信息、拍摄天气状况信息以及拍摄倍率等等)，对该多个高清图像进行分类存储。其中，拍摄天气状况信息可以包括拍摄该高清图像时是晴天、阴天、雾天或者雨天等等。
其中，为了便于检索，可以根据拍摄各个高清图像的地理区域，将该多个高清图像分为多个大类别，例如，将地理区域属于深圳的所有高清图像划分在一个大类别中，将地理区域属于广州的所有高清图像划分在另一个大类别中，等等。可以理解的是，地理区域的划分还可以是其他，例如，可以根据景点名称划分，本申请实施例不作限定。
按照各个高清图像的拍摄信息，将属于同一个地理区域的所有高清图像进一步分类。例如，可以将拍摄倍率相同或者相近的、GPS信息相同或者相近的以及拍摄时的天气状况信息相同的多个高清图片归为同一个类别。当然，在划分类别时，可以根据实际情况选择参考的拍摄信息，例如，可以将拍摄倍率相同或者相近的、GPS信息相同或者相近的多个高清图片归为同一个类别，而不关注拍摄时的天气状况信息，本申请实施例对此不作限定。
其中，上述拍摄倍率相同或者相近的可以是指，拍摄倍率之差的绝对值小于或者等于设置的倍率阈值。上述GPS信息相同或者相近可以是指，GPS信息所指示的地理位置之间的距离可以小于或者等于设置的距离阈值。
可以理解的是，本申请实施例的GPS信息均是指拍摄图像时，拍照设备(比如终端100或者多个地理区域部署的高清拍照设备)所处地理位置的GPS信息。
本申请实施例中，高清图像库中存储的高清图像的分辨率大于终端100拍摄的图像A的分辨率。
S1604，云服务器200根据图像A的拍摄信息，确定至少一个图像B。
其中，云服务器200可以确定拍摄信息与图像A的拍摄信息高度匹配的至少一个图像B，具体确定方法可以参照图15实施例的步骤S1504，在此不再赘述。
S1605，云服务器200将该至少一个图像B发送给终端100。
S1606，终端100从至少一个图像B中确定拍摄内容与图像A的拍摄内容最相似的参考图像1。
在一些实施例中，终端100接收云服务器200发送的至少一个图像B。
进一步可选的，终端100可以采用检索网络分别提取该至少一个图像B中每个图像B对应的第一特征以及图像A对应的第一特征，例如，可以将该至少一个图像B中每个图像B分别输入该检索网络，从而分别得到每个图像B对应的第一特征。将该图像A输入该检索网络，从而得到该图像A对应的第一特征。
终端100分别计算该图像A对应的第一特征与该至少一个图像B中每个图像B对应的第一特征之间的相似度，得到相似度最大的M个图像B。
示例性的，云服务器200可以将该图像A对应的第一特征与该至少一个图像B中每个图像B对应的第一特征之间的相似度，按照从大到小的顺序进行排序，并将排序在最前面的M个相似度对应的图像B作为相似度最大的M个图像B。
可以理解的是，通过检索网络提取的第一特征进行相似度的度量，可以得到与图像A的拍摄内容大致相似的M个图像B。
为了更加精准的从该M个图像B中确定拍摄内容与图像A的拍摄内容最相似的图像B，本申请实施例中，终端100可以采用重排序网络分别提取该M个图像B中每个图像B对应的第二特征以及图像A对应的第二特征，例如，可以将该M个图像B中每个图像B分别输入该重排序网络，从而分别得到每个图像B对应的第二特征。将该图像A输入该重排序网络，从而得到该图像A对应的第二特征。
终端100分别计算该图像A对应的第二特征与该M个图像B中每个图像B对应的第二特征之间的相似度，得到相似度最大的图像B，作为参考图像1。
示例性的，云服务器200可以将该图像A对应的第二特征与该M个图像B中每个图像B对应的第二特征之间的相似度，按照从大到小的顺序进行重排序，并将排序在最前面的相似度对应的图像B作为参考图像1。
S1607，终端100根据参考图像1，对图像A进行细节恢复处理，获得图像C。
其中，终端100对图像A进行细节恢复处理的具体步骤可以参照图15实施例中的步骤S1506，在此不再赘述。
S1608，终端100显示图像C。
其中，步骤S1608请参照图15实施例步骤S1608，在此不再赘述。
通过实施图16实施例，云服务器向终端返回拍摄信息高度匹配的至少一个图像B，并由终端根据自身需求提取该至少一个图像B对应的特征以及图像A对应的特征，而不需要将整个高清图像库中的所有图像进行特征提取，避免处理资源的浪费。
请参照图17所示，为本申请实施例提供的图像处理方法的又一种示意性流程图，在该 实施例中，终端100将图像A和图像A的拍摄信息发送给云服务器200，云服务器200根据图像A的拍摄信息从高清图像库中确定至少一个图像B，该图像B的拍摄信息与图像A的拍摄信息之间的匹配度大于第一阈值。云服务器200进一步提取图像A对应的第一特征和第二特征，并将图像A对应的第一特征和第二特征和每个图像B对应的第一特征和第二特征发送给终端100，并由终端100从该至少一个图像B中确定拍摄内容与图像A的拍摄内容最相似的参考图像1。如图所示，该图像处理方法包括但不限于以下步骤：
S1701，终端100检测拍照指令。
S1702，终端100拍摄图像A。
S1703，终端100将图像A和图像A的拍摄信息发送给云服务器200。
S1704，云服务器200根据图像A的拍摄信息，确定至少一个图像B。
其中，步骤S1701-步骤S1704请参照图17实施例步骤S1701-步骤S1704，在此不再赘述。
S1705，云服务器将图像A对应的第一特征和第二特征以及每个图像B对应的第一特征和第二特征发送给终端100。
在一些实施例中，云服务器200可以通过检索网络提取该图像A对应的第一特征，以及通过重排序网络提取该图像A对应的第二特征。
云服务器200将图像A对应的第一特征和第二特征，以及高清图像库中存储的该至少一个图像B中每个图像B对应的第一特征和第二特征发送给终端100。
S1706，终端100从至少一个图像B中确定拍摄内容与图像A的拍摄内容最相似的参考图像1。
在一些实施例中，终端100根据接收的图像A对应的第一特征和第二特征以及该至少一个图像B中每个图像B对应的第一特征和第二特征，确定拍摄内容与图像A的拍摄内容最相似的参考图像1。其中，具体确定方法可以参照图15实施例中的步骤S1505，在此不再赘述。
S1707，终端100根据参考图像1，对图像A进行细节恢复处理，获得图像C。
在一些实施例中，终端100根据参考图像1，对图像A进行细节恢复处理的流程可以参照图15实施例的步骤S1506，在此不再赘述。
S1708，终端100显示图像C。
其中，步骤S1708请参照图15实施例的步骤S1508，在此不再赘述。
通过实施图17所述实施例，通过云服务器提取图像对应的特征以及存储图像对应的特征，而由终端根据云服务器返回的图像A对应的特征以及至少一个图像B对应的特征进行匹配，得到参考图像1，可以避免占用终端的大量内存存储图像特征。
请参照图18，是本申请实施例提供的一种图像处理的方案框架图；如图所示，云服务器的高清图像库中的高清图像，通过检索网络离线提取第一特征。
云服务器不仅会通过检索网络在线提取手机拍摄的图像对应的第一特征(即特征向量)，还会根据手机拍摄的图像的拍摄信息(包括实时GPS信息、天气状况信息、拍摄倍率等)，从高清图像库中筛选得到特征向量集合，该特征向量集合中的高清图像的拍摄信息与手机 拍摄的图像的拍摄信息之间的匹配度大于第一阈值。其中，在线提取第一特征可以是指手机与云服务器之间网络连接，并通过该网络连接得到手机拍摄的图像，以及提取该图像的第一特征。
进一步根据特征向量集合中的第一特征与手机拍摄的图像对应的第一特征之间进行相似性度量，得到相似度最大的M个图像，例如M为15，即得到相似度最大的15个高清图像。
将该相似度最大的15个高清图像通过重排序网络提取每个高清图像对应的第二特征，以及通过重排序网络提取手机拍摄的图像对应的第二特征。再根据15个高清图像中每个高清图像对应的第二特征与手机拍摄的图像对应的第二特征之间的相似性度量，得到相似度最大的参考图像。该参考图像可以是与手机拍摄的图像拍摄角度高度一致、拍摄倍率相似以及拍摄天气状况信息一致的高清图像。
根据该参考图像对手机拍摄的图像进行细节恢复，从而得到清晰度比手机拍摄的图像更高的图像。
请参照图19，为本申请实施例提供的一种图像处理装置的结构示意图，如图所示，该图像处理装置可以包括获取模块1901、第一确定模块1902以及第二确定模块1903。
获取模块1901，用于获取拍摄的图像A和所述图像A的拍摄信息；
第一确定模块1902，用于根据所述图像A的拍摄信息，确定至少一个图像B，其中，所述图像B的拍摄信息与所述图像A的拍摄信息之间的匹配度大于第一阈值，所述图像B的分辨率大于所述图像A的分辨率；
第二确定模块1903，用于从所述至少一个图像B中确定拍摄内容与所述图像A的拍摄内容最相似的参考图像1。
可选的，所述拍摄信息包括以下信息中的一种或者多种：拍摄倍率、拍摄地理位置信息或拍摄天气状况信息。
可选的，所述图像处理装置还可以包括：
恢复处理模块，用于根据所述参考图像1，对所述图像A进行细节恢复处理，获得图像C，所述图像C的分辨率大于所述图像A的分辨率；
输出模块，用于输出所述图像C。
可选的，所述第二确定模块1903可以包括：
获取单元，用于分别获取所述至少一个图像B中每个图像B对应的第一特征和所述图像A对应的第一特征，其中，一个图像B对应一个第一特征；
第一确定单元，用于根据所述至少一个图像B中每个图像B对应的第一特征与所述图像A对应的第一特征之间的相似度，从所述至少一个图像B中确定拍摄内容与所述图像A的拍摄内容最相似的M个图像B，所述M为大于或者等于1的整数；
第二确定单元，用于从所述M个图像B中确定拍摄内容与所述图像A的拍摄内容最相似的参考图像1。
可选的，所述第二确定单元具体用于：
分别获取所述M个图像B中每个图像B对应的第二特征和所述图像A对应的第二特 征，其中，一个图像B对应一个第二特征；
根据所述M个图像B中每个图像B对应的第二特征与所述图像A对应的第二特征之间的相似度，从所述M个图像B中确定拍摄内容与所述图像A的拍摄内容最相似的图像B，作为参考图像1；
其中，所述第二特征比所述第一特征更加精细。
可选的，所述图像处理装置还可以包括：
第三确定模块，用于确定所述参考图像1的拍摄内容占所述图像A的拍摄内容的比例；
去除处理模块，用于若所述比例小于第二阈值，从所述图像A的拍摄内容中去除与所述参考图像1的拍摄内容匹配的拍摄内容，获得去除后的图像A；
第四确定模块，用于从所述至少一个图像B中确定拍摄内容与所述去除后的图像A的拍摄内容最相似的参考图像2；
所述恢复处理模块具体用于根据所述参考图像1和所述参考图像2，对所述图像A进行细节恢复处理，获得图像C。
可以理解的是，上述各个模块和/或单元可以是软件，也可以是硬件，或者是软件和硬件结合。关于各个步骤的具体描述可以参照方法实施例的描述，在此不再赘述。
下面介绍本申请另一实施例提供的一种图像处理方法。
由于目前，终端100在拍照或录视频为了获取较大的视场角(filedofview，FOV)，通常使用较低的变焦倍率进行拍摄，使用的变焦倍率越小，终端100拍摄到画面的视场角越大，画面中包含的图像内容越多。例如，终端100可以用广角镜头或超广角镜头进行拍摄，从而减小变焦倍率，增大拍摄画面的FOV。但是，拍摄画面的视场角与其拍摄画面的画质质量负相关。单独依靠终端100上的摄像器件，无法兼具拍摄画面的大视场角和高清画质。例如，在使用广角摄像头拍摄照片或视频时，拍摄画面的视场角虽然较大，但是其画质较为模糊，所拍摄的物体的细节纹理也不清晰，尤其远景的细节纹理更加模糊。
因此，本申请实施例通过提供一种图像处理方法，借助云服务器200上的高清图像数据库，将终端100上拍摄的低变焦倍率(也即，大视场角)的低清图像，发送给云服务器200。云服务器200基于高清图像库中高清图像的变焦倍率对低变焦倍率的低清图像进行切块，得到多个图像块。针对每一个图像块，云服务器都可以在高清图像库中进行检索匹配，得到一张或多张高清参考图，再经过基于高清参考图的图像增强网络进行图像质量提升，得到多张高清图像块。最后，云端的服务器再将这多张高清图像块拼接为一张分辨率更大，清晰度更高的超清图像，返回给终端100。这样，可以借助云服务器200的处理能力，使得终端100拍摄照片或视频时的拍摄画面不仅具有较广视场角，还可以兼具高倍变焦时所能拍到的丰富细节纹理以及清晰度。
图20示出了本申请实施例中提供的另一种图像处理方法的流程示意图。
如图20所示，该方法可以包括但不限于以下步骤：
S2001、终端100检测到拍照指令。
终端100可安装用于上传低清图像的应用程序(application，APP)。可选的，该应用程序可以包括但不限于拍照应用、修图应用等等。终端100上的该应用程序还可以登陆账 号。
用户可以通过触屏操作发起拍照指令，或者，用户也可以通过按键发起拍照指令。终端100检测拍照指令，并拍摄低清图像。
S2002、终端100在检测到拍照指令后，拍摄低清图像并获取低清图像的拍摄信息。
其中，由于终端100上成像装置的限制，在低清图像的分辨率比较低，细节比较模糊。低清图像的拍摄信息包括变焦倍率、以及以下一种或多种：地理位置信息、天气信息、季节信息、时间信息、拍摄参数(例如、光圈、曝光时间等等)和拍摄姿态信息等。例如，终端100可以以1倍变焦倍率拍摄该低清图像。
其中，变焦倍率越小，低清图像的视场角越大，低清图像中的细节越少，低清图像中的画面越模糊。
S2003、终端100将低清图像和低清图像的拍摄信息发送给云服务器200。
其中，终端100可以在拍摄完低清图像后，可以自动传该低清图像和该低清图像的拍摄信息给云服务器200。终端100也可以在拍摄完低清图像后，通过接收用户的操作，将该低清图像上传给云服务器200。
在一种可能的实现方式中，终端100也可以将图库中保存的低清图像上传给云服务器200。
S2004、云服务器200识别出低清图像的语义信息。
云服务器200在接收到终端100发送的低清图像后，通过语义分割模型识别出低清图像中不同类别的图像内容所在的区域，获取到该低清图像的语义分割掩码(mask)图。该语义分割掩码(mask)图可标记出低清图像中不同类别的图像内容所在的区域。其中，云服务器200可以标记出低清图像中图像内容的类别。
示例性的，如图21A所示，低清图像中图像内容的类别可以包括有天空、建筑物、地面等。云服务器200可以通过语义分割模型识别出天空、建筑物和地面分别在低清图像中的区域，得到语义分割mask图。
如图21B所示，该语义分割mask图中包括有区域2101、区域2102、和区域2103。该区域2101用于表示低清图像中天空所在的区域，该区域2102用于表示建筑物在低清图像中所在的区域，该区域2103用于表示地面在低清图像中所在的区域。
S2005、云服务器200基于语义信息以及高清图像库的变焦倍率信息，将低清图像分割成指定放大倍率的N个图像块。
其中，该云服务器200中可以配置高清图像库，该高清图像库中存储有多个高清图像、以及该多个高清图像中每个高清图像的拍摄信息。其中，高清图像的拍摄信息可以包括以下一种或多种：变焦倍率、地理位置信息、天气信息、季节信息、时间信息、拍摄参数(例如、光圈、曝光时间等等)和拍摄姿态信息等。在给云服务器200配置高清图像库时，可以让高清图像库中的高清图像库都在固定变焦倍率(例如，10倍的变焦倍率)。
云服务器200可以基于高清图像库中高清图像的变焦倍率以及低清图像的变焦倍率，确定出图像块的个数。其中，高清图像的变焦倍率大于等于低清图像的变焦倍率，高清图像的变焦倍率与低清图像的变焦倍率相差越大，图像块的个数越多。每个图像块的变焦倍 率与高清图像的变焦倍率相应或相近或满足预设比例关系。
示例性的，高清图像的变焦倍率可以为10倍的变焦倍率、低清图像的变焦倍率可以为1倍的变焦倍率，云服务器200可以将低清图像按照10倍变焦倍率的窗口，将低清图像划分成N个图像块，其中，每个图像块的大小和10倍变焦倍率的窗口大小相同。
如图21C所示，低清图像可以划分为N个图像块。图21C中只是示例性的示出了该N个图像块中的部分图像块。其中，图像块2131具体可以如图21D所示。
其中，云服务器200可以将该低清图像的语音分割mask图也用同样的窗口大小进行分割，得到每个图像块对应的mask块。云服务器200可以标记出每个图像块的类别。
S2006、云服务器200基于语义信息，从高清图像库中分被检索出每个图像块的T张高清检索图像，T为正整数。
语义信息可以包括有低清图像中每个图像块的类别。在一些实施例中，云服务器200可以针对不同类别的图像块，采用不同的检索策略。
例如，图像块的类别可以包括有天空、建筑和地面。由于天空区域的纹理、地面区域的纹理相对简单，云服务器200针对类别为“天空”和“地面”的图像块，可以不进行图像检索也不进行细节修复。由于建筑区域的纹理细节丰富，云服务器200可以针对类别为“建筑”的图像块，检索出与该“建筑”类别的图像块对应的高清检索图像。
其中，云服务器200可以根据低清图像的拍摄信息(其他实施例中也可被称为第一环境信息)以及图像块，从高清图像库中检索出该图像块对应的T张高清检索图像。
在一种可能的实现方式中，云服务器200基于低清图像的拍摄信息，从高清图像库中确定出至少一个第一匹配图像，其中，第一匹配图像的拍摄信息与低清图像的拍摄信息之间的匹配度大于第一阈值，第一匹配图像的分辨率大于低清图像的分辨率；云服务器分别获取至少一个第一匹配图像中每个第一匹配图像对应的第一特征和第一类型的图像块对应的第一特征，其中，一个第一匹配图像对应一个第一特征。云服务器基于至少一个第一匹配图像中每个第一匹配图像对应的第一特征与第一图像块对应的第一特征之间的相似度，从至少一个第一匹配图像中确定拍摄内容与第一类型的图像块的拍摄内容最相似的T张高清检索图像。
在一种可能的实现方式中，云服务器基于至少一个第一匹配图像中每个第一匹配图像对应的第一特征与第一类型的图像块对应的第一特征之间的相似度，确定出在第一特征上与第一类型的图像块最相似的至少一张第二匹配图像。云服务器分别获取至少一个第二匹配图像中每个第二匹配图像对应的第二特征和第一类型的图像块对应的第二特征，其中，一个第二匹配图像对应一个第二特征。云服务器基于至少一个第二匹配图像中每个第二匹配图像对应的第二特征与第一类型的图像块对应的第二特征之间的相似度，确定出在第二特征上与第一类型的图像块最相似的T张高清检索图像。其中，第二特征比第一特征更加精细。
具体有关云服务器200针对图像块进行图像检索的过程可以参考前述图18所示实施例，在此不再赘述。
S2007、云服务器200基于语义信息和低清图像，对每个图像块的T张高清检索图像进行匹配配准，得到T张高清配准图像。
在一些实施例中，云服务器200可以基于语义信息作为先验指导信息，针对不同类别的图像块，采用不同的匹配配准策略。
例如，图像块的类别可以包括有天空、建筑和地面。由于天空区域的纹理、地面区域的纹理相对简单，云服务器200针对类别为“天空”和“地面”的图像块，可以不进行图像匹配配准也不进行细节修复。由于建筑区域的纹理细节丰富，针对类别为“建筑”的图像块，云服务器200可以在检索出与该“建筑”类别的图像块对应的T张高清检索图像后，对高清图像进行图像配准，得到T张高清配准图像。
S2008、云服务器200基于语义信息和每个图像块的T张高清配准图像，对每个图像块的细节进行修复，得到N个高清图像块。
其中，云服务器200针对图像块对应的T张高清检索图像进行图像配准的过程可以如下：
如图22所示，云服务器200可以将基于低清图像的语义分割mask图，确定图像块的类别，进而判断该图像块是否需要进行配准。若需要进行图像配准，云服务器200可以利用传统匹配算法获取该图像块与每一张高清检索图像中匹配的特征点。然后，云服务器200可以通过该图像块与每一张高清检索图像之间的匹配特征点对，得到图像块与每一张高清检索图像之间的图像空间坐标变换参数。云服务器200可以根据图像空间坐标变换参数对每个高清检索图像进行图像配准，得到T张高清配准图像。
具体的，云服务器基于图像块与T张高清检索图像中每一张高清检索图像的图像空间坐标变换参数，将每一张高清检索图像中与所述第一类型的图像块中相匹配的特征点，调整到与图像块中相同的位置，得到T张高清配准图像。
其中，传统匹配算法可以包括但不限于如下两类算法中的任一种：基于灰度级模板匹配的算法和基于特征匹配的算法。其中，基于灰度级模板匹配的算法包括平均绝对差算法(mean absolute differences，MAD)、绝对误差和算法(sum of absolute differences，SAD)、误差平方和算法(sum of squared differences，SSD)、平均误差平方和算法(mean square differences，MSD)、归一化积相关算法(normalized cross correlation，NCC)、序贯相似性检测算法(sequential similiarity detection algorithm，SSDA)、hadamard变换算法(sum of absolute transformed difference，SATD)、局部灰度值编码算法等。基于特征匹配的算法可以包括但不限于如下一种或多种：FAST算法、ORB算法、SIFT算法、SURF算法、LBP算法、传统光流方法等。
在一种可能的实现方式中，在得到T张高清配准图像后，云服务器200可以基于选优策略，从T张高清配准图像中，筛选出最优的L张高清参考图像，其中，L为正整数且L小于等于T。
其中，选优策略可以是从T张高清配准图像中选出与图像块的匹配度最高的L张高清参考图像。在一些可能的实现方式中，选优策略可以是选出与图像块与T个高清配准图像 中匹配特征点对的图像空间位置最相似的L张高清参考图像。
在一些实施例中，云服务器200可以基于语义信息作为先验指导信息，针对不同类别的图像块，采用不同的细节修复策略。
例如，图像块的类别可以包括有天空、建筑和地面。由于天空区域的纹理、地面区域的纹理相对简单，云服务器200针对类别为“天空”和“地面”的图像块，可以不进行图像匹配配准也不进行细节修复。由于建筑区域的纹理细节丰富，针对类别为“建筑”的图像块，云服务器200可以在获取到图像块的L张高清参考图像后，对图像块进行细节修复。
其中，云服务器200用L张高清配准图像对图像块的细节修复过程可以如下：
如图23A所示，云服务器200可以将基于低清图像的语义分割mask图，确定图像块的类别，进而判断该图像块是否需要进行细节。若需要进行图像细节修复，则云服务器200可以将L张高清参考图像后，可以将L张高清参考图像与图像块输入到超分网络算法模型中，利用L张高清参考图像中的细节纹理信息，修复图像块中细节，提升图像块中的分辨率，得到高清图像块。其中，该超分网络算法模型中可以包括匹配度检测网络、深度配准网络和深度融合网络。该匹配度检测网络可用于检测出高清参考图像中与图像块中内容不一致的区域。该深度配准网络可用于将高清参考图像中与图像块中内容一致的区域进行像素级的配准。该深度融合网络可用于将高清参考图像中的高频细节融合到图像块上，提升图像块的分辨率，得到高清图像块。
示例性的，如图23B所示，云服务器200可以经过上述图22所示的图像配准过程，得到上述图像块2131对应的高清参考图像2331。云服务器200可以将上述图像块2131和该高清参考图像2331输入到超分网络算法模型中，将高清参考图像2331中的高频细节信息，融合到图像块2131中，得到如图23C所示的高清图像块2332。
S2009、云服务器200将N个高清图像块进行拼接，得到高清输出图。
在获取到每个图像块对应的高清图像块后，云服务器200可以将每个图像块对应的高清图像块按照图像块在低清图像中的位置进行拼接，得到高清输出图。
在一些实施例中，一些类别的图像块由于没有进行细节修复，云服务器200可以直接使用未经过细节修复的图像块与其他经过细节修复后得到的高清图像块进行拼接，得到高清输出图。
示例性的，云服务器200没有对低清图像中“天空”类别和“地面”类别的图像块进行细节修复，对低清图像中“建筑物”类别的图像块进行了细节修复。因此，云服务器200可以使用低清图像中“天空”类别和“地面”类别的图像块，与“建筑物”类别的图像块对应的高清图像块进行拼接，得到高清输出图。
S2010、云服务器200将高清输出图发送给终端100。
S2011、终端100显示高清输出图。
其中，该高清输出图在前述实施例中也可被称为第三图像或图像C。
终端100显示该高清输出图的时机以及应用界面可以参考前述界面交互实施例，在此不再赘述。
在一些实施例中，云服务器200在上述步骤S2004之前，可以先利用低清图像(在前述实施例中也可被称为第一图像或图像A)的拍摄信息(在前述实施例中也可被称为第一环境信息)匹配出第一图像处理模型，对该低清图像进行优化处理，并将优化处理后的低清图像依次进行后续步骤S2004至步骤S2009。其中，第一图像处理模型的匹配过程，以及利用第一图像处理模型对该低清图像进行优化处理，可以参考前述图8-图11所示实施例，在此不再赘述。
通过本申请实施例提供的图像处理方法，可以实现通过将较小变焦倍率的图像按照云端高清图像库相近的变焦倍率进行切块，放大到一定的尺度上，在云端高清图像库中进行检索匹配，再经过基于参考图的图像增强网络进行图像质量提升，最终的结果再拼接为一张分辨率更大，清晰度更高的超清图像。这样，在语义信息的帮助下，先局部增强再整体拼接融合，获取到一张更高分辨率的超清图像，有效解决了缺少特征信息区域的误检与错误匹配。
下面介绍本申请另一实施例提供的一种图像处理方法。
图24示出了本申请实施例中提供的一种图像处理方法的流程示意图。
其中，云服务器200上可以布置有高清图像库，其中，该高清图像库中可以存储有多张高清图像以及高清图像的拍摄信息(在上述实施例中也可被称为第二环境信息)，其中，该高清图像的拍摄信息可以包括变焦倍率、以及以下一种或多种：地理位置信息、天气信息、季节信息、时间信息、拍摄参数(例如、光圈、曝光时间等等)和拍摄姿态信息等。在构建该高清图像库时，构建装置可以针对同一地理位置的同一拍摄物体，获取到多张高变焦倍率的高清图像，然后，构建装置可以将多张高变焦倍率的高清图像根据图像内容的关联性拼接成全景高清图。接着，构建装置可以按照在多种变焦倍率对应的切割尺寸大小，分别对全景高清图进行切割，使得全景高清图在每一类变焦倍率下，都被划分割成多张高清图像。其中，变焦倍率越大，切割尺寸越小，切割后得到的高清图像的变焦倍率越大。
示例性的，如图25A所示，构建装置可以通过高变焦倍率的高清图像，拼接成全景高清图2500。其中，构建装置可以将分别在5倍的变焦倍率、10倍的变焦倍率、20倍的变焦倍率等变焦倍率时对全景图像2500进行分割，在每一种变焦倍率下都分割出多张高清图像。如图25A所示，当构建装置在10倍的变焦倍率下对全景高清图进行分割时，分割出的每一张高清图像的尺寸都可以如图25B中的高清图像2511所示。如图25C所示，当构建装置在5倍的变焦倍率下对全景高清图进行分割时，分割出的每一张高清图像的尺寸都可以如图25D中的高清图像2521所示。其中，对全景高清图2500分割时使用的变焦倍率越大，分割得到高清图像的数量越多，每一张高清图像中包括的图像信息越少。上述图25A和图25C都只是示出了对全景图像的切割。
如图24所示，该图像处理方法可以包括但不限于以下步骤：
S2401、终端100检测拍照指令。
具体内容，可以参考前述图20所示实施例中的步骤S2001，在此不再赘述。
S2402、终端100在检测到拍照指令后，拍摄低清图像。
具体内容，可以参考前述图20所示实施例中的步骤S2002，在此不再赘述。
S2403、终端100将低清图像和低清图像的拍摄信息发送给云服务器200。
具体内容，可以参考前述图20所示实施例中的步骤S2003，在此不再赘述。
S2404、云服务器200识别出低清图像的语义信息。
具体内容，可以参考前述图20所示实施例中的步骤S2004，在此不再赘述。
S2405、云服务器200基于语义信息，从高清图像库中检索出低清图像的T张高清检索图像。
具体的，云服务器200可以基于低清图像的语义信息，从高清图像库中检索出与低清图像内容相近的T张高清检索图像。
在一些实施例中，云服务器200可以基于低清图像的语义信息以及低清图像的拍摄信息(其他实施例中也可被称为第一环境信息)，从高清图像库中检索出与低清图像的内容相似的T张高清检索图像。具体有关云服务器200针对图像块进行图像检索的过程可以参考前述图18所示实施例，在此不再赘述。
S2406、云服务器200基于语义信息和低清图像，对低清图像的T张高清检索图像进行配准，得到低清图像的T张高清配准图像。
其中，利用T张高清检索图像对低清图像的配准过程，可以参考前述图22所示实施例，在此不再赘述。
S2407、云服务器200基于语义信息和低清图像的T张高清配准图像，对低清图像进行修复，得到高清输出图。
其中，利用T张高清配准图像对低清图像的配准过程，可以参考前述图23A所示实施例，在此不再赘述。
S2408、云服务器200将高清输出图发送给终端100。
S2409、终端100显示高清输出图。
其中，该高清输出图在前述实施例中也可被称为第三图像或图像C。
终端100显示该高清输出图的时机以及应用界面可以参考前述界面交互实施例，在此不再赘述。
在一种可能的实现方式中，云服务器200在基于语义信息和低清图像，对低清图像的T张高清检索图像进行配准，得到低清图像的T张高清配准图像之后，可以将T张高清配准图像发送给终端100。终端100基于低清图像的T张高清配准图像，对低清图像进行修复，得到高清输出图。
通过本申请实施例，可以利用大量的高变焦倍率的高清图像对云服务器200上高清图像库进行全景图的构建拼接，然后，根据不同变焦倍率的图像尺寸对全景图进行切割，得到不同变焦倍率的高清图像。可以让云服务器200在高清图像库中为终端100拍摄到的低清图像检索到变焦倍率相同或相近的高清参考图像，提升了利用高清参考图像对低清图像 的细节修复效果。
以上所述，以上实施例仅用以说明本申请的技术方案，而非对其限制；尽管参照前述实施例对本申请进行了详细的说明，本领域的普通技术人员应当理解：其依然可以对前述各实施例所记载的技术方案进行修改，或者对其中部分技术特征进行等同替换；而这些修改或者替换，并不使相应技术方案的本质脱离本申请各实施例技术方案的范围。

Claims (11)

  1. 一种图像处理方法，其特征在于，所述方法包括：
    云服务器获取终端发送的低清图像以及拍摄所述低清图像时的拍摄信息，所述拍摄信息包括所述低清图像的变焦倍率；
    所述云服务器识别所述低清图像的语义信息；
    所述云服务器基于高清图像库中高清图像的变焦倍率以及所述低清图像的变焦倍率，确定出图像块的数量N，所述N为正整数；
    所述云服务器基于图像块的数量N将所述低清图像分割成N个图像块，并基于所述低清图像的语义信息标记出所述N个图像块中每个图像块中图像内容的类型；
    所述云服务器从所述高清图像库中检索出与第一类型的图像块的拍摄内容匹配的T张高清检索图像，所述T为正整数；
    所述云服务器基于所述T张高清检索图像，对所述第一类型的图像块中的细节进行修复，得到第一类型的高清图像块，所述第一类型的高清图像块的分辨率大于所述第一类型的图像块；
    所述云服务器基于第一类型的高清图像块替换所述低清图像中的第一类型的图像块，得到高清输出图，并将所述高清输出图发送给所述终端。
  2. 根据权利要求1所述的方法，其特征在于，所述低清图像的拍摄信息还包括以下一种或多种：地理位置信息、天气信息、季节信息、时间信息、拍摄参数和拍摄姿态信息。
  3. 根据权利要求1或2所述的方法，其特征在于，所述云服务器从所述高清图像库中检索出与第一类型的图像块的拍摄内容匹配的T张高清检索图像，具体包括：
    所述云服务器基于所述低清图像的拍摄信息，从所述高清图像库中确定出至少一个第一匹配图像，其中，所述第一匹配图像的拍摄信息与所述低清图像的拍摄信息之间的匹配度大于第一阈值，所述第一匹配图像的分辨率大于所述低清图像的分辨率；
    所述云服务器分别获取所述至少一个第一匹配图像中每个第一匹配图像对应的第一特征和所述第一类型的图像块对应的第一特征，其中，一个第一匹配图像对应一个第一特征；
    所述云服务器基于所述至少一个第一匹配图像中每个第一匹配图像对应的第一特征与所述第一图像块对应的第一特征之间的相似度，从所述至少一个第一匹配图像中确定拍摄内容与所述第一类型的图像块的拍摄内容最相似的T张高清检索图像。
  4. 根据权利要求3所述的方法，其特征在于，所述云服务器基于所述至少一个第一匹配图像中每个第一匹配图像对应的第一特征与所述第一类型的图像块对应的第一特征之间的相似度，从所述至少一个第一匹配图像中确定拍摄内容与所述第一类型的图像块的拍摄内容最相似的T张高清检索图像，具体包括：
    所述云服务器基于所述至少一个第一匹配图像中每个第一匹配图像对应的第一特征与所述第一类型的图像块对应的第一特征之间的相似度，确定出在第一特征上与所述第一类 型的图像块最相似的至少一张第二匹配图像；
    所述云服务器分别获取所述至少一个第二匹配图像中每个第二匹配图像对应的第二特征和所述第一类型的图像块对应的第二特征，其中，一个第二匹配图像对应一个第二特征；
    所述云服务器基于所述至少一个第二匹配图像中每个第二匹配图像对应的第二特征与所述第一类型的图像块对应的第二特征之间的相似度，确定出在第二特征上与所述第一类型的图像块最相似的T张高清检索图像；
    其中，所述第二特征比所述第一特征更加精细。
  5. 根据权利要求1或2所述的方法，其特征在于，所述云服务器利用所述T张高清检索图像，对所述第一类型的图像块中的细节进行修复，得到第一类型的高清图像块，具体包括：
    所述云服务器获取所述第一类型的图像块与所述T张高清检索图像中每一张高清检索图像的图像空间坐标变换参数；
    所述云服务器基于所述第一类型的图像块与所述T张高清检索图像中每一张高清检索图像中匹配特征点对的图像空间坐标变换参数，将每一张高清检索图像中与所述第一类型的图像块中相匹配的特征点，调整到与所述第一类型的图像块中相同的位置，得到T张高清配准图像；
    所述云服务器基于所述T张高清配准图像对所述第一类型的图像块的细节进行修复，得到第一类型的高清图像块。
  6. 根据权利要求5所述的方法，其特征在于，所述云服务器基于所述T张高清配准图像对所述第一类型的图像块的细节进行修复，得到第一类型的高清图像块，具体包括：
    所述云服务器基于所述第一类型的图像块与所述T张高清检索图像中每一张高清检索图像中匹配特征点对的图像空间坐标变换参数，从所述T张高清配准图像中确定出匹配特征点对的图像空间位置最相似的L张高清参考图像，L为正整数；
    所述云服务器基于所述L张高清参考图像对所述第一类型的图像块的细节进行修复，得到所述第一类型的高清图像块。
  7. 根据权利要求1所述的方法，其特征在于，所述云服务器基于第一类型的高清图像块替换所述低清图像中的第一类型的图像块，得到高清输出图，具体包括：
    当所述低清图像中只包括第一类型的图像块时，所述云服务器将所述第一类型的高清图像块拼接成所述高清输出图；
    当所述低清图像中包括第一类型的图像块和第二类型的图像块时，所述云服务器将所述第一类型的高清图像块与所述第二类型的高清图像块拼接成所述高清输出图。
  8. 根据权利要求1-7中任一项所述的方法，其特征在于，在所述云服务器基于图像块的数量N将所述低清图像分割成N个图像块之前，所述方法还包括：
    所述云服务器根据所述低清图像的拍摄信息，确定出与所述环境信息对应的第一图像 处理模型；
    所述云服务器通过所述第一处理模型对所述低清图像进行处理；
    所述云服务器基于图像块的数量N将所述低清图像分割成N个图像块，具体包括：
    所述云服务器基于图像块的数量N将经过所述第一图像处理模型处理后的所述低清图像分割成所述N个图像块。
  9. 根据权利要求8所述的方法，其特征在于，所述云服务器通过所述第一处理模型对所述低清图像进行处理，具体包括：
    所述云服务器通过所述第一图像处理模型对所述低清图像的细节清晰度、和/或动态范围、和/或对比度、和/或噪声、和/或色彩进行处理。
  10. 一种云服务器，其特征在于，包括：收发器、一个或多个处理器和一个或多个存储器；所述一个或多个存储器与一个或多个处理器耦合，所述一个或多个存储器用于存储计算机程序代码，计算机程序代码包括计算机指令，当所述一个或多个处理器执行所述计算机指令时，使得所述云服务器执行上述权利要求1-9中的任一项所述的图像处理方法。
  11. 一种可读存储介质，用于存储有计算机指令，当计算机指令在云服务器上运行时，使得所述云服务器执行上述权利要求1-9中的任一项所述的图像处理方法。
PCT/CN2020/136178 2019-12-13 2020-12-14 一种图像处理方法及相关装置 WO2021115483A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
US17/784,342 US20230056332A1 (en) 2019-12-13 2020-12-14 Image Processing Method and Related Apparatus
EP20900091.8A EP4060603A4 (en) 2019-12-13 2020-12-14 IMAGE PROCESSING METHOD AND IMAGE DEVICE

Applications Claiming Priority (6)

Application Number Priority Date Filing Date Title
CN201911285717.4 2019-12-13
CN201911285717 2019-12-13
CN202010357077 2020-04-29
CN202010357077.X 2020-04-29
CN202011446454.3 2020-12-11
CN202011446454.3A CN112989092A (zh) 2019-12-13 2020-12-11 一种图像处理方法及相关装置

Publications (1)

Publication Number Publication Date
WO2021115483A1 true WO2021115483A1 (zh) 2021-06-17

Family

ID=76329638

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/136178 WO2021115483A1 (zh) 2019-12-13 2020-12-14 一种图像处理方法及相关装置

Country Status (3)

Country Link
US (1) US20230056332A1 (zh)
EP (1) EP4060603A4 (zh)
WO (1) WO2021115483A1 (zh)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114255187A (zh) * 2021-12-22 2022-03-29 中国电信集团系统集成有限责任公司 基于大数据平台的多层次多级图像优化方法及系统
CN115147703A (zh) * 2022-07-28 2022-10-04 广东小白龙环保科技有限公司 一种基于GinTrans网络的垃圾分割方法及系统

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
TWI784645B (zh) * 2021-07-29 2022-11-21 宏碁股份有限公司 擴增實境系統及其操作方法
CN117689890B (zh) * 2024-01-09 2024-07-02 哈尔滨工程大学 一种基于晴雾场景的语义分割方法、装置及存储介质
CN118134815A (zh) * 2024-01-25 2024-06-04 天创恒达数字视频技术开发(北京)有限公司 用于摄像装置的图像清晰度增强方法及系统

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20120328197A1 (en) * 2010-01-25 2012-12-27 Conrad Sanderson Identifying matching images
CN106998459A (zh) * 2017-03-15 2017-08-01 河南师范大学 一种连续变焦技术的单摄像头立体图像生成方法
CN107169928A (zh) * 2017-05-12 2017-09-15 武汉华大联创智能科技有限公司 一种基于深层线性映射学习的人脸超分辨率重建算法
CN107507158A (zh) * 2016-06-14 2017-12-22 中兴通讯股份有限公司 一种图像处理方法和装置
US20190005305A1 (en) * 2017-06-30 2019-01-03 Beijing Kingsoft Internet Security Software Co., Ltd. Method for processing video, electronic device and storage medium
CN109302600A (zh) * 2018-12-06 2019-02-01 成都工业学院 一种立体场景拍摄装置

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105205782B (zh) * 2015-09-06 2019-08-16 京东方科技集团股份有限公司 超解像方法和系统、服务器、用户设备及其方法
CN107967669B (zh) * 2017-11-24 2022-08-09 腾讯科技（深圳）有限公司 图片处理的方法、装置、计算机设备及存储介质

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20120328197A1 (en) * 2010-01-25 2012-12-27 Conrad Sanderson Identifying matching images
CN107507158A (zh) * 2016-06-14 2017-12-22 中兴通讯股份有限公司 一种图像处理方法和装置
CN106998459A (zh) * 2017-03-15 2017-08-01 河南师范大学 一种连续变焦技术的单摄像头立体图像生成方法
CN107169928A (zh) * 2017-05-12 2017-09-15 武汉华大联创智能科技有限公司 一种基于深层线性映射学习的人脸超分辨率重建算法
US20190005305A1 (en) * 2017-06-30 2019-01-03 Beijing Kingsoft Internet Security Software Co., Ltd. Method for processing video, electronic device and storage medium
CN109302600A (zh) * 2018-12-06 2019-02-01 成都工业学院 一种立体场景拍摄装置

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
HUANG, QUANLIANG ET AL.: "Single Image Super-Resolution Combined with Learning Algorithm", COMPUTER ENGINEERING AND APPLICATIONS, vol. 49, no. 23, 28 November 2012 (2012-11-28), pages 186 - 190, XP055820105, ISSN: 1002-8331 *
See also references of EP4060603A4

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114255187A (zh) * 2021-12-22 2022-03-29 中国电信集团系统集成有限责任公司 基于大数据平台的多层次多级图像优化方法及系统
CN115147703A (zh) * 2022-07-28 2022-10-04 广东小白龙环保科技有限公司 一种基于GinTrans网络的垃圾分割方法及系统
CN115147703B (zh) * 2022-07-28 2023-11-03 广东小白龙环保科技有限公司 一种基于GinTrans网络的垃圾分割方法及系统

Also Published As

Publication number Publication date
EP4060603A4 (en) 2023-05-10
EP4060603A1 (en) 2022-09-21
US20230056332A1 (en) 2023-02-23

Similar Documents

Publication Publication Date Title
WO2021115483A1 (zh) 一种图像处理方法及相关装置
US12096120B2 (en) Photographing method in telephoto scenario and mobile terminal
CN111866404B (zh) 一种视频编辑方法及电子设备
WO2021104485A1 (zh) 一种拍摄方法及电子设备
WO2021244455A1 (zh) 一种图像内容的去除方法及相关装置
WO2021129198A1 (zh) 一种长焦场景下的拍摄方法及终端
WO2020029306A1 (zh) 一种图像拍摄方法及电子设备
WO2022017261A1 (zh) 图像合成方法和电子设备
CN112989092A (zh) 一种图像处理方法及相关装置
WO2020173379A1 (zh) 一种图片分组方法及设备
CN112580400A (zh) 图像选优方法及电子设备
CN114697543B (zh) 一种图像重建方法、相关装置及系统
WO2021185296A1 (zh) 一种拍摄方法及设备
WO2022206764A1 (zh) 一种显示方法、电子设备和系统
CN113497890A (zh) 一种拍摄方法及设备
CN117425065A (zh) 一种拍摄方法及相关设备
CN114979457B (zh) 一种图像处理方法及相关装置
WO2023231697A1 (zh) 一种拍摄方法及相关设备
CN114842069A (zh) 一种位姿确定方法以及相关设备
CN115002333B (zh) 一种图像处理方法及相关装置
WO2022127609A1 (zh) 图像处理方法及电子设备
WO2021204103A1 (zh) 照片预览方法、电子设备和存储介质
WO2023036084A1 (zh) 一种图像处理方法及相关装置
WO2022228010A1 (zh) 一种生成封面的方法及电子设备
WO2024114785A1 (zh) 一种图像处理方法、电子设备及系统

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20900091

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2020900091

Country of ref document: EP

Effective date: 20220613

NENP Non-entry into the national phase

Ref country code: DE