WO2021212810A1 - 图像处理方法、装置、电子设备及存储介质 - Google Patents

图像处理方法、装置、电子设备及存储介质 Download PDF

Info

Publication number
WO2021212810A1
WO2021212810A1 PCT/CN2020/127564 CN2020127564W WO2021212810A1 WO 2021212810 A1 WO2021212810 A1 WO 2021212810A1 CN 2020127564 W CN2020127564 W CN 2020127564W WO 2021212810 A1 WO2021212810 A1 WO 2021212810A1
Authority
WO
WIPO (PCT)
Prior art keywords
sky
image
area
target
processed
Prior art date
Application number
PCT/CN2020/127564
Other languages
English (en)
French (fr)
Inventor
刘晓坤
Original Assignee
北京达佳互联信息技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 北京达佳互联信息技术有限公司 filed Critical 北京达佳互联信息技术有限公司
Priority to JP2022548804A priority Critical patent/JP2023513726A/ja
Publication of WO2021212810A1 publication Critical patent/WO2021212810A1/zh
Priority to US17/883,165 priority patent/US20220383508A1/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/194—Segmentation; Edge detection involving foreground-background segmentation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/77—Retouching; Inpainting; Scratch removal
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/60—Image enhancement or restoration using machine learning, e.g. neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/70—Denoising; Smoothing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/11—Region-based segmentation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/136—Segmentation; Edge detection involving thresholding
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/143—Segmentation; Edge detection involving probabilistic approaches, e.g. Markov random field [MRF] modelling
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/90—Determination of colour characteristics
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
    • G06V10/443—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components by matching or filtering
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/35—Categorising the entire scene, e.g. birthday party or wedding scene
    • G06V20/38—Outdoor scenes
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10024—Color image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20212—Image combination
    • G06T2207/20221—Image fusion; Image merging

Definitions

  • This application relates to the field of image processing technology, in particular to image processing methods, devices, electronic equipment, and storage media.
  • AI Artificial Intelligence matting technology
  • AI portrait blurring a typical scene where AI matting technology is applied, and it is also a more challenging task.
  • the outdoor photos taken by the user are all-encompassing, there will be branches, birds, buildings, wires, flags, windmills, glass, etc. in the sky. It is not easy to accurately separate the sky area from the non-sky area, especially on the branches. In a scene where there are subtle matters in the sky between the sky, the details of electric wires, etc., how to segment the sky area becomes extremely complicated.
  • the current "change of sky” function is mainly to first determine the sky area through AI matting and then perform post-processing.
  • segmentation effect of the traditional technology is poor, which leads to inaccurate determination of the sky area, and imperfect post-processing.
  • There are defects such as unnatural sky edge transition, resulting in a low filming rate.
  • the present application provides an image processing method, device, electronic device, and storage medium, so as to at least solve the problem that the sky area replacement rate is not high in the related art.
  • the technical solution of this application is as follows:
  • an image processing method which includes: performing image segmentation processing on a picture to be processed, and obtaining an initial mask image according to the result of the image segmentation processing; the initial mask image includes the to-be-processed image Each pixel in the picture belongs to the probability value of a pixel in the sky area; according to the initial mask image, it is determined whether the picture to be processed meets the preset sky area replacement condition, and if the sky area replacement condition is satisfied, the The grayscale image of the picture to be processed is a oriented image.
  • an image processing device including: a mask map determining unit configured to perform image segmentation processing on a picture to be processed, and obtain an initial mask map according to a result of the image segmentation processing;
  • the initial mask map contains the probability value that each pixel in the picture to be processed belongs to a pixel in the sky area;
  • the guiding filter unit is configured to perform a determination according to the initial mask map whether the picture to be processed satisfies a preset If the sky area replacement condition is satisfied, use the grayscale image of the picture to be processed as the guiding image to perform guided filtering processing on the initial mask image to obtain a target mask image;
  • sky scene acquisition unit Is configured to execute the acquisition of the target sky scene;
  • the target sky scene is selected from preset sky scene materials;
  • the sky area replacement unit is configured to execute according to the target mask map and the target sky The scene performs replacement processing on the sky area in the to-be-processed picture to obtain the first processed picture.
  • an electronic device including a processor; a memory for storing executable instructions of the processor; wherein the processor is configured to execute the instructions to achieve the above The described image processing method.
  • a storage medium when instructions in the storage medium are executed by a processor of an electronic device, the electronic device can execute the image processing method described above.
  • a computer program product includes a computer program, the computer program is stored in a readable storage medium, and at least one processor of the device reads from the readable storage medium The computer program is read and executed, so that the device executes the image processing method in the above embodiment.
  • Fig. 1 is a diagram showing an application environment of an image processing method according to an exemplary embodiment
  • Fig. 2 is a flowchart showing an image processing method according to an exemplary embodiment
  • Fig. 3 is a schematic diagram showing a probability map according to an exemplary embodiment
  • Fig. 4 is a picture to be processed according to an exemplary embodiment
  • Fig. 5 is a processed picture according to an exemplary embodiment
  • Fig. 6 is another processed picture according to an exemplary embodiment
  • Fig. 7 is a flowchart showing an image processing method according to another exemplary embodiment
  • Fig. 8 is a flowchart showing an image processing method according to another exemplary embodiment
  • Fig. 9 is a block diagram showing an image processing device according to an exemplary embodiment.
  • the image processing method provided by the embodiment of the present application can be applied to the electronic device as shown in FIG. 1.
  • the electronic device can be various personal computers, notebook computers, smart phones, tablet computers, and portable wearable devices.
  • the electronic device 100 may include one or more of the following components: a processing component 101, a memory 102, a power supply component 103, a multimedia component 104, an audio component 105, an input/output (I/O) interface 106, a sensor component 107, and Communication component 108.
  • the processing component 101 generally controls overall operations of the electronic device 100, such as operations associated with display, telephone calls, data communication, camera operations, and recording operations.
  • the processing component 101 may include one or more processors 109 to execute instructions to complete all or part of the steps of the foregoing method.
  • the processing component 101 may include one or more modules to facilitate the interaction between the processing component 101 and other components.
  • the processing component 101 may include a multimedia module to facilitate the interaction between the multimedia component 104 and the processing component 101.
  • the memory 102 is configured to store various types of data to support operations in the electronic device 100. Examples of these data include instructions for any application or method operating on the electronic device 100, contact data, phone book data, messages, pictures, videos, and the like.
  • the memory 102 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable Programmable read only memory (EPROM), programmable read only memory (PROM), read only memory (ROM), magnetic memory, flash memory, magnetic or optical disk.
  • SRAM static random access memory
  • EEPROM electrically erasable programmable read-only memory
  • EPROM erasable Programmable read only memory
  • PROM programmable read only memory
  • ROM read only memory
  • magnetic memory flash memory
  • flash memory magnetic or optical disk.
  • the power supply component 103 provides power for various components of the electronic device 100.
  • the power supply component 103 may include a power management system, one or more power supplies, and other components associated with the generation, management, and distribution of power for the electronic device 100.
  • the multimedia component 104 includes a screen that provides an output interface between the electronic device 100 and the user.
  • the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user.
  • the touch panel includes one or more touch sensors to sense touch, sliding, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure related to the touch or slide operation.
  • the multimedia component 104 includes a front camera and/or a rear camera. When the electronic device 100 is in an operation mode, such as a shooting mode or a video mode, the front camera and/or the rear camera can receive external multimedia data. Each front camera and rear camera can be a fixed optical lens system or have focal length and optical zoom capabilities.
  • the audio component 105 is configured to output and/or input audio signals.
  • the audio component 105 includes a microphone (MIC).
  • the microphone is configured to receive external audio signals.
  • the received audio signal can be further stored in the memory 102 or sent via the communication component 108.
  • the audio component 105 further includes a speaker for outputting audio signals.
  • the I/O interface 106 provides an interface between the processing component 101 and a peripheral interface module.
  • the above-mentioned peripheral interface module may be a keyboard, a click wheel, a button, and the like. These buttons may include, but are not limited to: home button, volume button, start button, and lock button.
  • the sensor component 107 includes one or more sensors for providing the electronic device 100 with various aspects of state evaluation.
  • the sensor component 107 can detect the on/off status of the electronic device 100 and the relative positioning of the components.
  • the component is the display and the keypad of the electronic device 100.
  • the sensor component 107 can also detect the electronic device 100 or the electronic device 100.
  • the position of the component changes, the presence or absence of contact between the user and the electronic device 100, the orientation or acceleration/deceleration of the electronic device 100, and the temperature change of the electronic device 100.
  • the sensor assembly 107 may include a proximity sensor configured to detect the presence of nearby objects when there is no physical contact.
  • the sensor component 107 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications.
  • the sensor component 107 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.
  • the communication component 108 is configured to facilitate wired or wireless communication between the electronic device 100 and other devices.
  • the electronic device 100 can access a wireless network based on a communication standard, such as WiFi, an operator network (such as 2G, 3G, 4G, or 5G), or a combination thereof.
  • the communication component 108 receives a broadcast signal or broadcast related information from an external broadcast management system via a broadcast channel.
  • the communication component 108 further includes a near field communication (NFC) module to facilitate short-range communication.
  • the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.
  • RFID radio frequency identification
  • IrDA infrared data association
  • UWB ultra-wideband
  • Bluetooth Bluetooth
  • the electronic device 100 may be implemented by one or more application specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPD), programmable logic devices (PLD), field-available A programmable gate array (FPGA), controller, microcontroller, microprocessor, or other electronic components are implemented to implement the above methods.
  • ASIC application specific integrated circuits
  • DSP digital signal processors
  • DSPD digital signal processing devices
  • PLD programmable logic devices
  • FPGA field-available A programmable gate array
  • controller microcontroller, microprocessor, or other electronic components are implemented to implement the above methods.
  • Fig. 2 is a flowchart showing an image processing method according to an exemplary embodiment. As shown in Fig. 2, the image processing method is used in the electronic device of Fig. 1 and includes the following steps.
  • step S201 perform image segmentation processing on the picture to be processed, and obtain an initial mask map according to the result of the image segmentation processing; the initial mask map contains the probability value of each pixel in the picture to be processed belonging to a pixel in the sky area .
  • the picture to be processed refers to the picture that needs to be replaced by the sky area, which can be a picture input by the user through the client (for example, a picture downloaded through the client, a picture taken by a camera on the client, etc.), which can be The picture obtained in advance can also be a picture obtained in real time, so it can also be called an input picture.
  • the pictures to be processed can be pictures in various formats and scenes containing sky regions (the sky regions can be large or small).
  • the sky area refers to an area belonging to the sky, and the sky can be a sky under a scene such as a sunny day, a cloudy day, after rain, evening, night, rainbow, and so on. In addition to the sky area, there can be other areas such as buildings and hills in the image to be processed.
  • the size and shape of the picture to be processed can be of multiple types.
  • the image segmentation processing of the picture to be processed may refer to the classification of the regions to which pixels in the picture belong, that is, the division of the image into disjoint regions.
  • image segmentation technology has developed by leaps and bounds.
  • the technology-related scene object segmentation, human body front background segmentation and other technologies have been obtained in unmanned driving, augmented reality, security monitoring and other industries. Wide range of applications.
  • the image to be processed may be segmented by means of deep learning. For example, a neural network model is used to determine the probability value of each pixel in the picture to be processed belonging to the sky area, thereby realizing the sky area segmentation of the picture to be processed.
  • the probability value can be selected according to the actual situation. The higher the value, the more likely the pixel is to belong to the sky area (that is, the pixel within the sky area). For example, a value of 1 indicates that the pixel definitely belongs to the sky area, a value of 0 indicates that the pixel definitely does not belong to the sky area, and a value of 0.9 indicates that the pixel is 90% likely to belong to the sky area.
  • the probability value may be understood as the mask information of the corresponding pixel. Therefore, the probability value may be referred to as a mask value, and correspondingly, the initial mask image may also be referred to as a probability mask image.
  • the probability value can be arranged according to the position of the pixel to obtain the initial mask map; in addition, the probability value of each pixel can also be converted into the corresponding gray scale For example, a probability value of 1 is converted into a gray value of 255, a probability value of 0.5 is converted to a gray value of 127, and a probability value of 0 is converted to a gray value of 0. In this way, the probability value can be output
  • the electronic device can obtain the corresponding probability value after obtaining the gray value of the initial mask image.
  • the initial mask image displayed in the interface can be as shown in Figure 3.
  • step S202 it is determined whether the picture to be processed meets a preset sky area replacement condition according to the initial mask image, and if the sky area replacement condition is satisfied, the grayscale image of the picture to be processed is used as a guide map A guided filtering process is performed on the initial mask image to obtain a target mask image.
  • the sky area replacement condition can be determined according to the size of the sky area, whether it is believed, the scene of the picture to be processed, and so on. In an optional implementation manner, it may be considered that the sky area replacement condition is not satisfied when the picture to be processed meets at least one of the sky area is too small, the picture to be processed is overwhelming, the night scene, and the foggy scene.
  • the quantization process may make the difference of the probability value between adjacent pixels inaccurate, which will easily cause a sense of separation between the various areas of the picture, and the initial mask image and grayscale image Input to the guided filtering algorithm, which can fuse the information of the initial mask image and grayscale image, for example, if there is a large color gradient between the building and the sky in the input image, and the building in the initial mask image The probability value difference between the object and the sky is small, and the problem in the initial mask image can be corrected after the guided filtering.
  • the guided filtering can also correct the color gradient in the initial mask image.
  • This kind of problem Therefore, the effect of feathering the initial mask image can be achieved through the guided filtering process, so that the output processed image is closer to the input image, and a more realistic and natural changing sky effect is achieved.
  • the initial mask image after the guided filtering process can be shown in the partial enlarged image on the right side of Figure 3.
  • the algorithm can fuse the information of the two images to achieve the effect of feathering and gradual transition (gradual transition from black area to white Area), to prevent the appearance of a sense of fragmentation, so that the resulting picture will not be so blunt.
  • step S203 a target sky scene is acquired; the target sky scene is selected from preset sky scene materials.
  • the target sky scene refers to the type of sky that needs to be replaced
  • the sky scene material can refer to scenes such as sunny, cloudy, after rain, rainbow, sunset, evening, night, etc.
  • the target sky scene is selected from the sky scene materials, so that the subsequent pictures to be processed can be processed in a targeted manner.
  • the target sky scene can be obtained according to scene selection information input by the user. For example, if the scene selection information is "sunset", the target sky scene can be a sunset scene.
  • step S204 the sky area in the to-be-processed picture is replaced according to the target mask image and the target sky scene to obtain a first processed picture.
  • the process of performing the replacement processing on the sky area can be as follows: determine the sky area in the target mask map, obtain the target sky scene, obtain the target sky map (that is, the sky picture used for replacement) according to the target sky scene, and pass the target sky
  • the picture replaces the sky area, and the replaced picture can be used as the picture that has undergone the sky area replacement process, that is, the first processed picture.
  • the target sky map may be a sky picture obtained from a sky material library according to the target sky scene. For example, if the target sky scene is sunset, the target sky map may be a sky picture in the sunset scene.
  • the shape and size of the target sky map can be determined according to the shape and size of the sky area, and the shapes and sizes of the two can be the same or inconsistent (for example, the distance between the target sky map and the sides of the sky area is 50 pixels, that is, the target sky map is larger than the sky area).
  • image segmentation is performed on the image to be processed, and an initial mask image containing the probability value of each pixel in the image to be processed belonging to the pixel point in the sky area is obtained according to the image segmentation processing result, and the initial mask image can be accurate Determine the sky area; if it is determined according to the initial mask image that the image to be processed satisfies the preset sky area replacement condition, use the gray image of the image to be processed as the guiding image to perform guided filtering on the initial mask image to obtain the target mask image,
  • the guided filtering achieves the feathering effect on the initial mask image, and can effectively correct the color gradient between adjacent image regions in the initial mask image; making the processed image change effect obtained from the target mask image real and natural.
  • the film rate is high.
  • the step of using the grayscale image of the picture to be processed as a oriented image to perform oriented filtering processing on the initial mask image to obtain a target mask image includes: acquiring the to-be-processed image The blue channel pixel value of each pixel in the picture; the pixel with the blue channel pixel value in the initial mask in the target distribution interval and the probability value greater than the first threshold is determined as the first pixel; wherein
  • the target distribution interval is the interval in which the number of blue channel pixel values of each pixel in the first evaluation area is the largest among a plurality of preset intervals, and the first evaluation area is that the probability value in the initial mask is greater than the first
  • the target blue channel pixel value is the minimum value of the blue channel pixel value of each pixel in the second evaluation area, and the
  • the first evaluation area can correspond to an area that is most likely to be a sky area in the initial mask. Therefore, the size of the second threshold can be a larger value to ensure the accuracy of the selected first evaluation area as much as possible. sex. In an optional implementation manner, the second threshold may be 0.9, 0.92, and so on. Further, after the first evaluation area is determined, the blue channel pixel value of each pixel in the first evaluation area can be classified into a set interval (the interval can be divided from 0 to 255 according to the actual situation), of which the number is the largest The interval of is determined as the target distribution interval.
  • the first pixel point can be determined in combination with the target distribution interval and the probability value.
  • the probability value in the initial mask image is greater than a preset threshold (which can be determined according to the actual situation, such as 0.5) and the blue channel pixel value is in the target
  • the area where the pixels in the distribution interval are located is determined as the first pixel.
  • a more accurate first pixel can be determined by combining the blue channel pixel value and the probability value, thereby improving the accuracy of the sky area replacement.
  • the blue channel pixel value is the value corresponding to the B (Blue) channel in the RGB value.
  • the probability value reduction processing may also be performed on the area that is highly likely to be not the sky area.
  • the second evaluation area can correspond to the area in the initial mask image that may be the sky area.
  • the third threshold can be set to a higher value (or even It can be greater than the second threshold.
  • the third threshold can be 0.93, 0.95, etc.), so as to take the area into consideration as much as possible to determine the target blue channel pixel value from it, and use the target blue channel The upper limit of the pixel value determines the second pixel point in the initial mask image.
  • the target blue channel pixel value corresponding to the second evaluation area can be understood as the lowest critical point of the blue channel pixel value in the sky area. If the blue channel pixel value of a certain pixel is less than the critical point, it is considered that the pixel does not belong to the sky area. Then the probability value reduction process is performed.
  • the second pixel point determined according to the target blue channel pixel value can effectively prevent the non-sky area from being missed and classified into the sky area.
  • the second threshold and the third threshold may also be the same size, and even the third threshold is smaller than the second threshold.
  • the processing of increasing the probability value can refer to assigning the probability value to a larger value, such as: 1, 0.99, 0.98, etc.
  • the processing of reducing the probability value can refer to assigning the probability value to a smaller value, for example: 0.5 , 0.6, etc.
  • a pixel with a probability value of 0.9 if it is determined that it is most likely to belong to the sky area according to the blue channel pixel value (for example, the value corresponding to the blue channel B is higher), then its probability value can be assigned to 1; and
  • the probability value can be halved or reduced to 0.
  • the process of halving the probability value of the first pixel can also be replaced by a uniform subtraction of a certain probability value (for example, uniform subtraction of 0.1, 0.2, etc.).
  • the probability value increase processing is performed on the pixels that are most likely to be the sky area
  • the probability value reduction processing is performed on the pixels that are most likely not the sky area, which can highlight the probability value of the sky area for subsequent replacement of the sky area. This part of the area can be accurately identified at the time, and then the sky can be changed to improve the accuracy of the sky.
  • the step of increasing the probability value of the first pixel and reducing the probability value of the second pixel includes: setting the probability value of the first pixel Is 1; the probability value of the second pixel is halved.
  • the implementation process of adding and reducing pixels can be: calculating the histogram of the blue channel pixel value of each pixel in the area of the probability value> 0.9 of the picture to be processed, and calculating according to the histogram
  • the blue channel pixel value is in which of the following intervals (Q1:0-63, Q2:64-127, Q3:128-191, Q4:192-255), the interval containing the most pixels is regarded as the target distribution interval, which is recorded as Qi, the probability value in the target distribution interval that the probability value is greater than 0.5 and the blue channel value belongs to the Qi interval is set to 1.0.
  • the way of dividing the interval can be adjusted according to the actual situation, for example, it can be divided into larger or smaller intervals.
  • the second pixel that is most likely not the sky area is halved, which can highlight the probability value of the sky area for subsequent sky areas. This part of the area can be accurately identified when replacing, and then the sky is changed to improve the accuracy of the sky.
  • the red channel pixel value can be used to implement it.
  • Such a processing manner can determine a more accurate scene for the scene in an optional implementation manner. Accurate sky area, and then achieve accurate sky area replacement operation, improve the filming rate.
  • the probability value reduction process may be performed on the second pixel first, and then the probability value increase process of the first pixel point is performed on the probability map after the probability value reduction process to obtain the candidate mask map; It is also possible to perform the process of increasing the probability value of the first pixel on the initial mask image, perform the process of reducing the probability value of the second pixel on the initial mask image, and integrate the probability maps obtained by the two processing methods (for example: For the pixels that have not changed in the initial mask image, the original probability value is retained; for those that have only been processed for increasing the probability value or processed with only a small probability value, the processed probability value is retained; for both the probability value If the value increase processing is performed with the probability value reduction processing, the average value of the probability values obtained by the two processing methods can be taken to obtain the candidate mask map. Afterwards, the candidate mask image is subjected to guided filtering processing using the gray image of the picture to be processed as the guided image to obtain the target mask image.
  • the step of performing replacement processing on the sky area in the picture to be processed according to the target mask map and the target sky scene to obtain the first processed picture includes: determining all The non-sky area in the picture to be processed is used as the foreground image; the sky material image is cropped according to the size of the target sky scene and the sky area, and the scene corresponds to the target sky scene and the size is the same as the sky area
  • the target sky map corresponding to the size of the target sky map; the foreground map and the target sky map are combined according to the target mask map to obtain the first processed picture; wherein the sky in the first processed picture The area is replaced by the target sky map.
  • determining the non-sky area in the picture to be processed can be to determine the pixels with a probability value lower than 0.85 (or other values or combined with the blue color channel value) as non-sky area pixels, and for these non-sky areas Integrating regional pixels (for example, integrating scattered points into a complete area) can obtain a non-sky area.
  • the process of obtaining the target sky map may be: determining the smallest rectangular bounding box of the sky area, obtaining candidate sky material images corresponding to the target sky scene from the sky material images, and maintaining the aspect ratio In the case of, crop out the target sky image in the candidate sky material image.
  • a center cutting method can be used (of course, other methods can also be used), for example: determining the center point of the candidate sky material map, and cutting the center point as the center to a rectangle surrounded by The box size gets the target sky map.
  • the candidate sky material image can be scaled and then cropped; of course, it can also be cropped in the candidate sky material image first, and then the cropped image can be scaled to obtain the target sky. picture.
  • the realization process of combining the foreground image and the target sky image according to the target mask image to obtain the processed image can be: in a blank area, determine an area consistent with the size of the image to be processed as the area to be filled; according to the target mask The map determines the sky area in the area to be filled, fills the target sky map into the sky area, and fills the foreground image into the remaining area to obtain the processed image.
  • the sky area is determined in the picture to be processed according to the target mask map, the target sky map is filled in the sky area, and the foreground image is filled in the remaining area to obtain the processed picture.
  • the probability values in the overlapping area can be integrated to obtain a final probability value, and then the area filling can be completed according to this final probability value, for example , If it is determined to be a sky area (for example, the probability value is greater than 0.5), the target sky map is filled, and if it is determined to be a non-sky area, the foreground map is filled. Assuming that the picture to be processed is shown in FIG. 4, the area formed by two rectangular frames represents the sky area 401, and the sky area 401 is currently a rainbow scene. After replacing it with a sunny scene, it can be shown in Figure 5.
  • the sky area has been replaced with the target sky map, that is, the sky area replacement process is realized, and the buildings (foreground map) in it are not replaced.
  • the above process combines the sky area and the foreground map. Differentiating processing can effectively prevent the aliasing of the picture content, and ensure the clarity of the processed pictures obtained while meeting the needs of users.
  • the step of performing replacement processing on the sky area in the to-be-processed picture according to the target mask map and the target sky scene to obtain the first processed picture includes : According to the target mask map, determine the first area, the second area, and the remaining area from the to-be-processed picture; wherein the probability value of the pixels contained in the first area is 1, and the first area The probability value of the pixel points contained in the second area is 0, and the remaining area is the area in the picture to be processed excluding the first area and the second area; replacing the first area with the Target sky map; replace the second area with the foreground image; compare the foreground image and the target according to the probability value, red channel pixel value, green channel pixel value, and blue channel pixel value corresponding to the remaining area
  • the sky map performs color channel information fusion of pixels; and obtains the first processed picture according to the target sky map after the color channel information fusion processing.
  • the first area may refer to an area that is most likely or definitely a sky area
  • the second area may refer to an area that is most likely or definitely not a sky area.
  • the probability value can also be replaced with other values, such as 0.98, 0.99, etc.; in the case of the above probability value of 0, the probability value can also be replaced with other values, for example: 0.02 , 0.01 and so on.
  • the color channel information can refer to the values corresponding to the three RGB channels of the picture, and the color of the corresponding pixel can be obtained through the color channel information. Further, the color channel information may be a value corresponding to a certain channel or multiple channels. In an optional implementation manner, it may include a red channel pixel value, a green channel pixel value, and a blue channel pixel value.
  • the fusion of the foreground image and the target sky image can be to perform arithmetic processing on the color channel information corresponding to the two parts, such as: multiply processing, gamma transformation processing, etc., and use the processed color channel information as the corresponding pixel
  • the color channel information can be all or part of the RGB value.
  • multiplying can be multiplying the RGB values of the corresponding pixels of the foreground image and the target sky image
  • the gamma transformation processing can be the exponentiation processing of the RGB values of the pixels corresponding to the foreground image and the target sky image.
  • the step of combining the foreground image and the target sky image according to the target mask image to obtain the first processed image includes: comparing all the images according to the target sky scene At least one of the brightness, contrast, and saturation of the foreground image is adjusted to obtain a target foreground image whose brightness, contrast, and saturation match the target sky scene; The foreground image and the target sky image are combined to obtain the first processed image.
  • the first processed picture obtained through such processing can adapt to the style of the target sky scene.
  • the foreground image can be filtered and beautified to obtain the target foreground image.
  • the implementation process of this embodiment may be:
  • the target sky map and the foreground map are fused according to the probability value, and the mixRGB of a certain pixel A obtained by fusion is as follows:
  • mixRGB src*(1.0-mask)+sky*mask; where src represents the RGB value of pixel A in the foreground image, mask represents the probability value of pixel A in the reference probability map, and sky represents the pixel in the target sky image.
  • the RGB value of point A is the RGB value of point A.
  • tempRGB sqrt(mixRGB*sky,0.5); among them, 0.5 is a preset parameter, which can also be set to other values as needed.
  • the other pixels are merged in the same way as above.
  • the area that is definitely the sky area is replaced with the target sky map, and the area that is definitely not the sky area is replaced with the foreground map.
  • the foreground map and the target sky map are If the area in between is not processed, there will be a sense of fragmentation that the foreground suddenly changes to the sky area.
  • the area in the middle part (the area that is neither the foreground image nor the target sky image) is merged, and the merged area is integrated
  • the color channel information of the sky part and the non-sky part can be ensured that the foreground transitions to the sky naturally, and a more realistic and natural picture can be obtained.
  • the step of correcting the processed image may be included, for example, comparing the processed image with the image to be processed, if the foreground image or the target sky image is If there is a position or angle deviation, you can adjust it. If there is an unnatural edge transition, you can adjust it. In this way, the final output processed image can have higher accuracy and sky area replacement effect, and the filming rate can be improved.
  • the step of performing image segmentation processing on the picture to be processed, and obtaining an initial mask image according to the result of the image segmentation processing includes: performing image segmentation processing on the picture to be processed through a pre-trained neural network model , Obtain the image segmentation processing result; determine the probability value of each pixel in the to-be-processed picture belonging to a pixel in the sky area according to the image segmentation processing result; obtain the probability value of each pixel in the to-be-processed picture The initial mask map.
  • the neural network model may be a CNN (Convolutional Neural Network) model or the like.
  • the neural network may be u-net, u-net variant, ic-net, deeplab series, and so on.
  • the step of performing image segmentation processing on the picture to be processed through a pre-trained neural network model includes: scaling the size of the picture to be processed to a preset size; The picture to be processed after the size scaling is subjected to normalization processing; the picture to be processed after the normalization processing is subjected to image segmentation processing through a pre-trained neural network model.
  • the preset size can be set according to the actual situation, for example: 512*512, 768*768, etc.
  • the normalization process can be: obtain picture samples, determine the mean and variance of the RGB value of each pixel of these picture samples, and subtract the mean and divide by the variance of the RGB value of each pixel of the picture to be processed for better performance
  • Machine learning and feature learning are used to classify whether each pixel in the image to be processed belongs to the sky area pixel to obtain a probability map.
  • the neural network model can be obtained through pre-determined training images.
  • the trained neural network model can be down-sampled to process the image to be processed, and then extract the feature information, analyze the feature information, and output the image segmentation processing result.
  • the computer can determine the probability value that each pixel belongs to the sky area according to the image segmentation processing result, and then obtain the initial mask image.
  • the neural network model fully analyzes the feature information in the image to be processed. Compared with the traditional graph cut image segmentation method, this machine learning method can more accurately segment the image to be processed, and then obtain an accurate probability map.
  • the picture to be processed when the picture to be processed is obtained, it is preprocessed first, and then the neural network model is used to segment the network to obtain the initial mask image, and then the sky is accurately distinguished from the image to be processed according to the initial mask image.
  • Region can be understood as a post-processing process
  • the segmentation network and post-processing are combined to retain the accurate segmentation effect of the neural network model.
  • accurate sky and non-sky areas are post-processed to make the final The sky area replacement processing is more accurate.
  • the step of determining whether the picture to be processed meets a preset sky area replacement condition according to the initial mask image includes: determining the picture to be processed according to the initial mask image Whether it meets at least one of the following, if it meets at least one of the following, it is determined that the to-be-processed picture does not meet the sky area replacement condition; if it does not meet any of the following, it is determined that the to-be-processed picture meets the sky area replacement condition: determine the to-be-processed picture The first proportion of the mid-sky area, if the first proportion is less than the preset fourth threshold, it is determined that the sky area is too small; the second proportion of the enormous area in the image to be processed is determined, if the If the second proportion is greater than the preset fifth threshold, it is determined to be enormous; wherein, the enormous area is the area where the probability value of each pixel is in the middle interval, and the middle interval is determined by the middle of the probability value.
  • the average brightness of the sky area in the picture to be processed determines the average brightness of the sky area in the picture to be processed, and if the average brightness is less than the preset sixth threshold, determine it as a night scene; determine the picture to be processed
  • the third proportion of the target dark channel area if the third proportion is greater than the preset seventh threshold, it is determined to be a foggy scene; wherein, the target dark channel area is the sky area where the dark channel value is less than eighth The area where the threshold pixel is located.
  • the size of the fourth, fifth, sixth, seventh, and eightth thresholds can be determined according to actual conditions.
  • the fourth threshold can be 0.9
  • the fifth threshold can be 0.1
  • the sixth threshold can be 0.3cd/m 2
  • the seventh The threshold may be 0.4
  • the eighth threshold may be 0.8.
  • a pixel with a probability value of 1 can be understood as definitely belonging to the sky area
  • a pixel with a probability value of 0 can be understood as definitely not belonging to the sky area
  • a pixel with an intermediate probability value (for example: 0.5) may belong to
  • the sky area may not belong to the sky area, and this part of the pixels belongs to the uncertain area, that is, the overwhelming area.
  • the disbelief area may be an area where the probability value falls between 0.3 and 0.7 (the boundary value of the interval may also be other values).
  • the area with a probability value greater than 0.9 can be extracted from the probability map obtained in the current step, as the sky area, different implementations
  • the sky area in the example can be the same or different.
  • the realization process of determining the average brightness of the sky area in the picture to be processed may be: calculating the average gray value of the area of the original image (picture to be processed) corresponding to the area with the probability value greater than 0.9 in the initial mask image.
  • the dark channel value of the sky area (for RGB images, the dark channel value is the minimum of the three values of RGB, for example, if R/G/B is 10/20/30 respectively, the dark channel value is 10)
  • a dark channel map can be obtained, and then based on the dark channel map, it is determined whether the picture to be processed is a foggy scene (if the proportion of the area with a dark channel value less than a certain threshold is greater than the seventh threshold, it is considered to be a foggy day).
  • the third proportion of the target dark channel area can be understood as the statistical information of the dark channel value.
  • the implementation process of the foregoing embodiment may be:
  • any one of the above conditions is met, it can be determined that the image to be processed is not suitable for the replacement of the sky area, that is, it is not suitable for changing the sky, otherwise, if the above conditions are not met (or most of them are not met), it can be considered suitable for changing Day, then the subsequent day change processing will be performed.
  • the method further includes: if the sky area replacement condition is not satisfied , Acquiring a target filter tool according to the target sky scene, and performing filter processing on the picture to be processed through the target filter tool to obtain a second processed picture.
  • the following processing can be performed: obtain a predetermined target sky scene; select a comprehensive filter corresponding to the target sky scene; The integrated filter processes the picture to be processed to obtain a second processed picture.
  • an image processing method including the following steps:
  • S703 Obtain the blue channel pixel value of each pixel in the picture to be processed.
  • S704 Determine a pixel with a blue channel pixel value in the initial mask image in the target distribution interval and with the probability value greater than a first threshold as the first pixel.
  • S705 Determine a pixel with a blue channel pixel value less than a target blue channel pixel value in the initial mask image as a second pixel.
  • Channel pixel values perform pixel color channel information fusion on the foreground image and the target sky map; obtain the first processed picture according to the target sky map after color channel information fusion processing.
  • S712 Obtain a target filter tool according to the target sky scene, and perform filter processing on the to-be-processed picture through the target filter tool to obtain a second processed picture.
  • image segmentation is performed on the picture to be processed to accurately obtain an initial mask map containing the probability value of each pixel in the picture to be processed belonging to the sky area; according to the blue channel pixel value of each pixel in the picture to be processed, The pixel points corresponding to the sky area in the initial mask image are increased by the probability value process to obtain a reference probability map.
  • the resulting reference probability map further expands the difference in probability values between the sky area and the non-sky area; according to the reference probability map Accurately identify the sky area in the picture to be processed. Therefore, the processed picture obtained from the replacement process of the sky area has high accuracy, and the effect of changing the sky is natural, and the rate of changing the sky is high.
  • an image processing method is provided.
  • the method is applied to a mobile terminal and includes the following steps:
  • Send to the segmentation network for processing send the preprocessed picture to be processed into the segmentation network to obtain a probability map.
  • Crop the sky material calculate the minimum rectangular bounding box of the sky area in the picture to be processed according to the feathering probability map, and cut and scale the sky material to the size of the rectangular bounding box at the center while maintaining the aspect ratio;
  • (4)Adaptively adjust the foreground adjust the brightness, contrast, and saturation of the foreground image in the image to be processed according to the target sky scene to match the style of the material. Then use the foreground filter corresponding to the target sky scene to beautify to obtain the final used foreground image;
  • Segmentation fusion fusion adjusted foreground image and cropped sky material according to the feathering probability map
  • the new sky area is merged with the original image background, using the method of segmented layer aliasing and segmented linear fusion, so that the transition of the synthesized image between the sky and the non-sky is real and natural;
  • the non-sky area namely the foreground image
  • the final overall color is adjusted to ensure the uniformity and beauty of the final rendering
  • Fig. 9 is a block diagram showing an image processing device 900 according to an exemplary embodiment.
  • the device includes a mask map determining unit 901, a guidance filtering unit 902, a sky scene acquisition unit 903, and a sky area replacement unit 904.
  • the mask map determining unit 901 is configured to perform image segmentation processing on the picture to be processed, and obtain an initial mask map according to the result of the image segmentation processing; the initial mask map includes that each pixel in the picture to be processed belongs to the sky area The probability value of the inner pixel.
  • the guiding filtering unit 902 is configured to determine whether the picture to be processed meets a preset sky area replacement condition according to the initial mask map, and if the sky area replacement condition is satisfied, use the gray scale of the picture to be processed
  • the figure is a directional graph that performs directional filtering processing on the initial mask graph to obtain a target mask graph.
  • the sky scene acquisition unit 903 is configured to perform acquisition of a target sky scene; the target sky scene is selected from preset sky scene materials.
  • the sky area replacement unit 904 is configured to perform replacement processing on the sky area in the to-be-processed picture according to the target mask map and the target sky scene, to obtain a first processed picture.
  • the guiding filtering unit includes: a pixel value obtaining subunit configured to perform obtaining the blue channel pixel value of each pixel in the picture to be processed; a first pixel determining subunit, Is configured to execute the determination of the pixel points with the blue channel pixel value in the initial mask image in the target distribution interval and the probability value greater than the first threshold as the first pixel point; wherein, the target distribution interval is a preset Among the multiple intervals, the interval with the largest number of blue channel pixel values of each pixel in the first evaluation area, where the first evaluation area is the area where the pixels with the probability value greater than the second threshold in the initial mask map are located The second threshold is greater than the first threshold; a second pixel point determining sub-unit is configured to perform the determination of a pixel with a blue channel pixel value less than the target blue channel pixel value in the initial mask as the second Pixel; the target blue channel pixel value is the minimum value of the blue channel pixel value of each pixel in the second evaluation area, and
  • the guided filtering subunit includes: a first probability value setting module configured to perform setting the probability value of the first pixel point to 1; and a second probability value setting module configured to It is configured to perform the halving of the probability value of the second pixel point.
  • the sky area replacement unit includes: a foreground image determining subunit, configured to perform determining a non-sky area in the picture to be processed, as a foreground image; and a sky material cropping subunit, which is It is configured to perform cropping of the sky material image according to the size of the target sky scene and the sky area to obtain a target sky image with a scene corresponding to the target sky scene and a size corresponding to the size of the sky area; a first foreground The combination subunit is configured to perform the combination of the foreground image and the target sky image according to the target mask image to obtain the first processed image; wherein the sky in the first processed image The area is replaced by the target sky map.
  • the sky area replacement unit further includes: an area determining subunit, configured to perform determining the first area and the second area from the to-be-processed picture according to the target mask map And the remaining area; wherein the probability value of the pixels contained in the first area is 1, the probability value of the pixels contained in the second area is 0, and the remaining area is the picture to be processed Excluding the first area and the second area; the sky map replacement sub-unit is configured to perform the replacement of the first area with the target sky map; the foreground image replacement sub-unit is configured to execute The second area is replaced with the foreground image; the channel information fusion subunit is configured to perform the matching of the probability value, the red channel pixel value, the green channel pixel value, and the blue channel pixel value corresponding to the remaining area.
  • the foreground image and the target sky map perform color channel information fusion of pixels; the processed picture acquiring subunit is configured to execute the target sky map processed according to the color channel information fusion process to obtain the first processed picture.
  • the sky area replacement unit further includes: a foreground image adjustment sub-unit configured to perform an adjustment of at least one of the brightness, contrast, and saturation of the foreground image according to the target sky scene. Adjustment to obtain a target foreground image with brightness, contrast, and saturation matching the target sky scene; a second foreground combination subunit configured to perform a comparison between the target foreground image and the target foreground image according to the target mask image The target sky images are combined to obtain the first processed image.
  • a foreground image adjustment sub-unit configured to perform an adjustment of at least one of the brightness, contrast, and saturation of the foreground image according to the target sky scene. Adjustment to obtain a target foreground image with brightness, contrast, and saturation matching the target sky scene
  • a second foreground combination subunit configured to perform a comparison between the target foreground image and the target foreground image according to the target mask image The target sky images are combined to obtain the first processed image.
  • the guided filtering unit is further configured to perform determining whether the picture to be processed meets at least one of the following preset conditions according to the initial mask image, and if it meets at least one of the following preset conditions, It is determined that the image to be processed does not meet the sky area replacement condition, and if it does not meet any of the following preset conditions, it is determined that the image to be processed meets the sky area replacement condition:
  • Preset condition 1 The first proportion of the sky area in the picture to be processed is less than a preset fourth threshold
  • Preset condition 2 The second proportion of the enormous area in the picture to be processed is greater than the preset fifth threshold; wherein, the overwhelming area is the area where the probability value of each pixel is in the middle interval, and the middle The interval is composed of the median value of the probability value and the adjacent value of the median value;
  • Preset condition 3 The average brightness of the sky area in the picture to be processed is less than a preset sixth threshold
  • Preset condition 4 The third proportion of the target dark channel area in the image to be processed is greater than the preset seventh threshold; wherein, the target dark channel area is the pixel point in the sky area whose dark channel value is less than the eighth threshold your region.
  • the image processing device further includes: a filter processing unit configured to perform, if the sky area replacement condition is not met, obtain a target filter tool according to the target sky scene, and pass the target The filter tool performs filter processing on the to-be-processed picture to obtain a second processed picture.
  • a filter processing unit configured to perform, if the sky area replacement condition is not met, obtain a target filter tool according to the target sky scene, and pass the target The filter tool performs filter processing on the to-be-processed picture to obtain a second processed picture.
  • non-transitory computer-readable storage medium including instructions, such as the memory 102 including instructions, and the foregoing instructions may be executed by the processor 109 of the device 100 to complete the foregoing method.
  • the non-transitory computer-readable storage medium may be ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, optical data storage device, etc.
  • a computer program product includes a computer program, the computer program is stored in a readable storage medium, and at least one processor of the device reads from the readable storage medium The computer program is read and executed, so that the device executes the image processing method described in the above embodiment.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Multimedia (AREA)
  • Software Systems (AREA)
  • Evolutionary Computation (AREA)
  • Probability & Statistics with Applications (AREA)
  • Artificial Intelligence (AREA)
  • Computing Systems (AREA)
  • Databases & Information Systems (AREA)
  • General Health & Medical Sciences (AREA)
  • Medical Informatics (AREA)
  • Health & Medical Sciences (AREA)
  • Image Processing (AREA)
  • Processing Or Creating Images (AREA)
  • Studio Devices (AREA)
  • Image Analysis (AREA)

Abstract

一种图像处理方法、装置、电子设备及存储介质。所述方法包括：对待处理图片进行图像分割处理，根据图像分割处理结果得到初始掩码图；初始掩码图包含有待处理图片中各个像素点属于天空区域内像素点的概率值；根据初始掩码图确定待处理图片是否满足预设的天空区域替换条件，若满足天空区域替换条件，以待处理图片的灰度图为导向图对初始掩码图进行导向滤波处理，得到目标掩码图；获取目标天空场景；目标天空场景从预设的天空场景素材中选择得到；根据目标掩码图和目标天空场景对待处理图片中的天空区域进行替换处理，得到第一已处理图片。根据该方案能准确确定出天空区域并进行天空区域的替换，换天效果真实自然，成片率高。

Description

图像处理方法、装置、电子设备及存储介质
相关申请的交叉引用
本申请要求在2020年04月23日提交中国专利局、申请号为202010327149.6、申请名称为“图像处理方法、装置、电子设备及存储介质”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及图像处理技术领域，尤其涉及图像处理方法、装置、电子设备及存储介质。
背景技术
目前AI(人工智能，Artificial Intelligence)抠图技术广泛落地到各个应用场景，如AI人像虚化、AI分区场景识别等等。其中AI换天(将图片中的天空区域替换为特定场景下的天空区域)是AI抠图技术落地的一个典型场景，也是一个比较有挑战性的任务。因为用户实际拍摄的室外照片包罗万象，天空中会有树枝、飞鸟、建筑、电线、彩旗、风车、玻璃等等，如何精准地将天空区域与非天空区域分割开来并非易事，特别是在树枝间的天空、包含电线细节等等类似的天空中存在细微事务的场景中，如何分割出天空区域变得异常复杂。
目前的“换天”功能主要是先通过AI抠图确定出天空区域然后进行后处理。但是传统技术的分割效果差，因此导致天空区域确定不准确，且后处理不完善存在天空边缘过渡不自然等瑕疵，导致成片率不高。
发明内容
本申请提供一种图像处理方法、装置、电子设备及存储介质，以至少解决相关技术中天空区域替换成片率不高的问题。本申请的技术方案如下：
根据本申请实施例的第一方面，提供一种图像处理方法，包括：对待处理图片进行图像分割处理，根据图像分割处理结果得到初始掩码图；所述初始掩码图包含有所述待处理图片中各个像素点属于天空区域内像素点的概率值；根据所述初始掩码图确定所述待处理图片是否满足预设的天空区域替换条件，若满足所述天空区域替换条件，以所述待处理图片的灰度图为导向图对所述初始掩码图进行导向滤波处理，得到目标掩码图；获取目标天空场景；所述目标天空场景为从预设的天空场景素材中选择得到的；根据所述目标掩码图和所述目标天空场景对所述待处理图片中的天空区域进行替换处理，得到第一已处理图片。
根据本申请实施例的第二方面，提供一种图像处理装置，包括：掩码图确定单元，被配置为执行对待处理图片进行图像分割处理，根据图像分割处理结果得到初始掩码图；所述初始掩码图包含有所述待处理图片中各个像素 点属于天空区域内像素点的概率值；导向滤波单元，被配置为执行根据所述初始掩码图确定所述待处理图片是否满足预设的天空区域替换条件，若满足所述天空区域替换条件，以所述待处理图片的灰度图为导向图对所述初始掩码图进行导向滤波处理，得到目标掩码图；天空场景获取单元，被配置为执行获取目标天空场景；所述目标天空场景为从预设的天空场景素材中选择得的到；天空区域替换单元，被配置为执行根据所述目标掩码图和所述目标天空场景对所述待处理图片中的天空区域进行替换处理，得到第一已处理图片。
根据本申请实施例的第三方面，提供一种电子设备，包括处理器；用于存储所述处理器可执行指令的存储器；其中，所述处理器被配置为执行所述指令，以实现如上所述的图像处理方法。
根据本申请实施例的第四方面，提供一种存储介质，当所述存储介质中的指令由电子设备的处理器执行时，使得电子设备能够执行如上所述的图像处理方法。
根据本申请实施例的第五方面，提供一种计算机程序产品，所述程序产品包括计算机程序，所述计算机程序存储在可读存储介质中，设备的至少一个处理器从所述可读存储介质读取并执行所述计算机程序，使得设备执行如上实施例中的图像处理方法。
应当理解的是，以上的一般描述和后文的细节描述仅是示例性和解释性的，并不能限制本申请。
附图说明
此处的附图被并入说明书中并构成本说明书的一部分，示出了符合本申请的实施例，并与说明书一起用于解释本申请的原理，并不构成对本申请的不当限定。
图1是根据一示例性实施例示出的一种图像处理方法的应用环境图；
图2是根据一示例性实施例示出的一种图像处理方法的流程图；
图3是根据一示例性实施例示出的一种概率图的显示示意图；
图4是根据一示例性实施例示出的一种待处理图片；
图5是根据一示例性实施例示出的一种已处理图片；
图6是根据一示例性实施例示出的另一种已处理图片；
图7是根据另一示例性实施例示出的一种图像处理方法的流程图；
图8是根据又一示例性实施例示出的一种图像处理方法的流程图；
图9是根据一示例性实施例示出的一种图像处理装置的框图。
具体实施方式
为了使本领域普通人员更好地理解本申请的技术方案，下面将结合附图，对本申请实施例中的技术方案进行清楚、完整地描述。
需要说明的是，本申请的说明书和权利要求书及上述附图中的术语“第一”、“第二”等是用于区别类似的对象，而不必用于描述特定的顺序或先后次序。应该理解这样使用的数据在适当情况下可以互换，以便这里描述的本 申请的实施例能够以除了在这里图示或描述的那些以外的顺序实施。以下示例性实施例中所描述的实施方式并不代表与本申请相一致的所有实施方式。相反，它们仅是与如所附权利要求书中所详述的、本申请的一些方面相一致的装置和方法的例子。
本申请实施例提供的图像处理方法可以应用于如图1所示的电子设备中。该电子设备可以是各种个人计算机、笔记本电脑、智能手机、平板电脑和便携式可穿戴设备。参照图1，电子设备100可以包括以下一个或多个组件：处理组件101、存储器102、电源组件103、多媒体组件104、音频组件105、输入/输出(I/O)接口106、传感器组件107以及通信组件108。
处理组件101通常控制电子设备100的整体操作，诸如与显示、电话呼叫、数据通信、相机操作和记录操作相关联的操作。处理组件101可以包括一个或多个处理器109来执行指令，以完成上述的方法的全部或部分步骤。此外，处理组件101可以包括一个或多个模块，便于处理组件101和其他组件之间的交互。例如，处理组件101可以包括多媒体模块，以方便多媒体组件104和处理组件101之间的交互。
存储器102被配置为存储各种类型的数据以支持在电子设备100的操作。这些数据的示例包括用于在电子设备100上操作的任何应用程序或方法的指令、联系人数据、电话簿数据、消息、图片、视频等。存储器102可以由任何类型的易失性或非易失性存储设备或者它们的组合实现，如静态随机存取存储器(SRAM)、电可擦除可编程只读存储器(EEPROM)、可擦除可编程只读存储器(EPROM)、可编程只读存储器(PROM)、只读存储器(ROM)、磁存储器、快闪存储器、磁盘或光盘。
电源组件103为电子设备100的各种组件提供电力。电源组件103可以包括电源管理系统，一个或多个电源，及其他与为电子设备100生成、管理和分配电力相关联的组件。
多媒体组件104包括在所述电子设备100和用户之间的提供一个输出接口的屏幕。在一些实施例中，屏幕可以包括液晶显示器(LCD)和触摸面板(TP)。如果屏幕包括触摸面板，屏幕可以被实现为触摸屏，以接收来自用户的输入信号。触摸面板包括一个或多个触摸传感器以感测触摸、滑动和触摸面板上的手势。所述触摸传感器可以不仅感测触摸或滑动动作的边界，而且还检测与所述触摸或滑动操作相关的持续时间和压力。在一些实施例中，多媒体组件104包括一个前置摄像头和/或后置摄像头。当电子设备100处于操作模式，如拍摄模式或视频模式时，前置摄像头和/或后置摄像头可以接收外部的多媒体数据。每个前置摄像头和后置摄像头可以是一个固定的光学透镜系统或具有焦距和光学变焦能力。
音频组件105被配置为输出和/或输入音频信号。例如，音频组件105包括一个麦克风(MIC)，当电子设备100处于操作模式，如呼叫模式、记录模式和语音识别模式时，麦克风被配置为接收外部音频信号。所接收的音频信号可以被进一步存储在存储器102或经由通信组件108发送。在一些实施例中，音频组件105还包括一个扬声器，用于输出音频信号。
I/O接口106为处理组件101和外围接口模块之间提供接口，上述外围接口模块可以是键盘，点击轮，按钮等。这些按钮可包括但不限于：主页按钮、音量按钮、启动按钮和锁定按钮。
传感器组件107包括一个或多个传感器，用于为电子设备100提供各个方面的状态评估。例如，传感器组件107可以检测到电子设备100的打开/关闭状态，组件的相对定位，例如所述组件为电子设备100的显示器和小键盘，传感器组件107还可以检测电子设备100或电子设备100一个组件的位置改变，用户与电子设备100接触的存在或不存在，电子设备100方位或加速/减速和电子设备100的温度变化。传感器组件107可以包括接近传感器，被配置用来在没有任何的物理接触时检测附近物体的存在。传感器组件107还可以包括光传感器，如CMOS或CCD图像传感器，用于在成像应用中使用。在一些实施例中，该传感器组件107还可以包括加速度传感器、陀螺仪传感器、磁传感器、压力传感器或温度传感器。
通信组件108被配置为便于电子设备100和其他设备之间有线或无线方式的通信。电子设备100可以接入基于通信标准的无线网络，如WiFi，运营商网络(如2G、3G、4G或5G)，或它们的组合。在一个示例性实施例中，通信组件108经由广播信道接收来自外部广播管理系统的广播信号或广播相关信息。在一个示例性实施例中，所述通信组件108还包括近场通信(NFC)模块，以促进短程通信。例如，在NFC模块可基于射频识别(RFID)技术，红外数据协会(IrDA)技术，超宽带(UWB)技术，蓝牙(BT)技术和其他技术来实现。
在示例性实施例中，电子设备100可以被一个或多个应用专用集成电路(ASIC)、数字信号处理器(DSP)、数字信号处理设备(DSPD)、可编程逻辑器件(PLD)、现场可编程门阵列(FPGA)、控制器、微控制器、微处理器或其他电子元件实现，用于执行上述方法。
图2是根据一示例性实施例示出的一种图像处理方法的流程图，如图2所示，图像处理方法用于图1的电子设备中，包括以下步骤。
在步骤S201中，对待处理图片进行图像分割处理，根据图像分割处理结果得到初始掩码图；所述初始掩码图包含有所述待处理图片中各个像素点属于天空区域内像素点的概率值。
其中，待处理图片指的是需要进行天空区域替换的图片，可以是用户通过客户端中输入的图片(例如，通过客户端下载的图片、通过客户端上的摄像头拍摄的图片等)，可以是预先获取到的图片，也可以是实时获取的图片，因此也可以称为输入图。待处理图片可以是各种格式、各种场景包含有天空区域(天空区域可大可小)的图片。该天空区域指的是属于天空的区域，该天空可以是晴天、阴天、雨后、傍晚、夜晚、彩虹等等场景下的天空。待处理图片中除了天空区域，还可以有建筑物、山丘等其他区域。另外，待处理图片的大小、形状可以是多种类型。
对待处理图片进行的图像分割(image segmentation)处理可以指对图片中像素所属区域的分类，即将图像划分成互不相交的区域。近些年来随着深 度学习技术的逐步深入，图像分割技术有了突飞猛进的发展，该技术相关的场景物体分割、人体前背景分割等技术已经在无人驾驶、增强现实、安防监控等行业都得到广泛的应用。进一步的，本申请中可以通过深度学习的方式对待处理图片进行图像分割处理。例如，通过神经网络模型确定待处理图片中各个像素点属于天空区域的概率值，进而实现对待处理图片的天空区域分割。其中，概率值可以根据实际情况进行取值，取值越高则表示该像素点越可能属于天空区域(即属于天空区域范围内的像素点)。例如，取值为1则表示该像素肯定属于天空区域，取值为0则表示该像素肯定不属于天空区域，取值为0.9则表示该像素有90％的可能属于天空区域。在一些示例性实施例中，概率值可以理解为对应像素点的掩码(mask)信息，因此，概率值可以称为mask值，对应的，初始掩码图也可以称为概率掩码图。
进一步的，在得到各个像素点的概率值之后，可以对各个概率值按照像素点的位置进行排布，得到初始掩码图；另外，也可以将各个像素点的概率值转化为对应的灰度值，例如，概率值为1则转化为灰度值255，概率值为0.5则转化为灰度值127，概率值为0则转化为灰度值0，通过这样的方式就可以输出与概率值对应的灰度图像，电子设备在获取到初始掩码图的灰度值后就能获知其对应的概率值，在界面中显示的初始掩码图可以如图3所示。
在步骤S202中，根据所述初始掩码图确定所述待处理图片是否满足预设的天空区域替换条件，若满足所述天空区域替换条件，以所述待处理图片的灰度图为导向图对所述初始掩码图进行导向滤波处理，得到目标掩码图。
其中，天空区域替换条件可以根据天空区域的大小、是否置信、待处理图片的场景等来确定。在一种可选的实施方式中，可以在待处理图片符合天空区域太小、待处理图片不置信、夜晚场景、雾天场景中的至少一项时认为不满足天空区域替换条件。
由于是初始掩码图经过图像分割处理得到的，而图像分割处理会对各个像素点进行量化处理(对于通过神经网络模型进行图像分割处理的情况，下采样会导致图片分辨率降低，例如一个像素点变为5个像素点)，量化处理可能会使得相邻像素点之间的概率值差别不准确，这样就容易使得图片各个区域之间存在割裂感，而将初始掩码图和灰度图输入到导向滤波的算法中，该算法能够融合初始掩码图和灰度图的信息，例如，如果输入图像中建筑物与天空之间的存在较大的颜色梯度，而初始掩码图中建筑物与天空之间的概率值差别较小，经过导向滤波后就可以纠正初始掩码图中的这种问题。相反，如果输入图像中树叶与天空之间的存在较小的颜色梯度，而初始掩码图中树叶与天空之间的概率值差别较大，经过导向滤波后同样可以纠正初始掩码图中的这种问题。因此经过导向滤波处理可以实现对初始掩码图羽化的效果，使得输出的已处理图片更为接近输入图像，实现更为真实自然的换天效果。导向滤波处理后的初始掩码图可以如图3右侧的局部放大图所示，从该图可以看出算法能融合两个图的信息，达到羽化渐变的效果(从黑色区域逐步过渡到白色区域)，防止出现割裂感，使得得到的图片不会那么生硬。
在步骤S203中，获取目标天空场景；所述目标天空场景从预设的天空场 景素材中选择得到。
其中，目标天空场景指的是所需要替换的天空类型，天空场景素材可以指晴天、阴天、雨后、彩虹、晚霞、傍晚、夜晚等等场景。本步骤从天空场景素材中选择目标天空场景，以便后续针对性地对待处理图片进行处理。进一步的，该目标天空场景可以根据用户输入的场景选择信息得到。例如：场景选择信息为“晚霞”，则目标天空场景可以为晚霞场景。
在步骤S204中，根据所述目标掩码图和所述目标天空场景对所述待处理图片中的天空区域进行替换处理，得到第一已处理图片。
对天空区域进行替换处理的实现过程可以为：确定目标掩码图中的天空区域，获取目标天空场景，根据该目标天空场景获取目标天空图(即用于替换的天空图片)，通过该目标天空图对该天空区域进行替换，替换后的图片就可以作为已经经过天空区域替换处理的图片，即第一已处理图片。其中，目标天空图可以是根据目标天空场景从天空素材库中获取的天空图片，例如，目标天空场景为晚霞，则目标天空图就可以是晚霞场景下的天空图片。
在一示例性实施例中，目标天空图的形状大小可以根据天空区域的形状大小确定，两者的形状大小可以一致，也可以不一致(例如，目标天空图和天空区域各条边的间隔都为50个像素点，即，目标天空图比天空区域大)。
上述图像处理方法中，对待处理图片进行图像分割处理，根据图像分割处理结果得到包含有待处理图片中各个像素点属于天空区域内像素点概率值的初始掩码图，根据该初始掩码图能准确确定出天空区域；若根据初始掩码图确定待处理图片满足预设的天空区域替换条件，以待处理图片的灰度图为导向图对初始掩码图进行导向滤波处理得到目标掩码图，该导向滤波实现了对初始掩码图的羽化效果，能有效纠正初始掩码图中相邻图片区域之间的颜色梯度；使得根据目标掩码图得到的已处理图片换天效果真实自然，成片率高。
在一示例性实施例中，所述以所述待处理图片的灰度图为导向图对所述初始掩码图进行导向滤波处理，得到目标掩码图的步骤，包括：获取所述待处理图片中各个像素点的蓝通道像素值；将所述初始掩码图中蓝通道像素值处于目标分布区间且所述概率值大于第一阈值的像素点确定为第一像素点；其中，所述目标分布区间为预设的多个区间中，第一评价区域中各个像素点的蓝通道像素值数量最多的区间，所述第一评价区域为所述初始掩码图中所述概率值大于第二阈值的像素点所在区域；所述第二阈值大于所述第一阈值；将所述初始掩码图中蓝通道像素值小于目标蓝通道像素值的像素点确定为第二像素点；所述目标蓝通道像素值为第二评价区域中各个像素点的蓝通道像素值的最小值，所述第二评价区域为所述初始掩码图中所述概率值大于第三阈值的像素点所在区域；所述第三阈值大于所述第二阈值；将所述第一像素点的概率值增加，并将所述第二像素点的概率值减少，得到参考掩码图；以所述待处理图片的灰度图为导向图对所述参考掩码图进行导向滤波处理，得到所述目标掩码图。
其中，第一评价区域可以对应初始掩码图中极有可能是天空区域的区域， 因此，第二阈值的大小可以为一个较大的值，以尽可能保证所选取的第一评价区域的准确性。在一种可选的实施方式中，第二阈值可以为0.9、0.92等。进一步的，在确定第一评价区域后，可以将第一评价区域中各个像素点蓝通道像素值归类到设定区间(可以根据实际情况从0到255中进行区间划分)中，其中数量最多的区间确定为目标分布区间。
更进一步的，第一像素点可以结合目标分布区间和概率值来确定，例如，将初始掩码图中概率值大于预设阈值(可以根据实际情况确定，例如0.5)且蓝通道像素值处于目标分布区间的像素点所在区域确定为第一像素点。通过这样的处理方式，能结合蓝通道像素值和概率值确定出更为准确的第一像素点，进而提高天空区域替换的准确性。其中，蓝通道像素值为RGB值中B(Blue)通道对应的值。
在一示例性实施例中,除了将极有可能是天空区域的区域进行概率值增加处理，还可以对极有可能不是天空区域的区域进行概率值降低处理。其中，第二评价区域可以对应初始掩码图中有可能是天空区域的区域，为防止将非天空区域的像素点归入天空区域中，第三阈值可以取值为一个较高的值(甚至可以大于第二阈值，在一种可选的实施方式中，第三阈值可以为0.93、0.95等)，以尽可能将区域纳入考虑范围进而从中确定出目标蓝通道像素值，并以目标蓝通道像素值为上限在初始掩码图中确定出第二像素点。第二评价区域对应的目标蓝通道像素值可以理解为天空区域的蓝通道像素值最低临界点，如果某个像素点的蓝通道像素值小于该临界点，则认为该像素点不属于天空区域，则对于进行概率值减少处理。根据该目标蓝通道像素值确定的第二像素点能有效防止遗漏非天空区域而将其归入天空区域中。
当然，在另一些示例性实施例中，第二阈值和第三阈值也可以一样大，甚至第三阈值小于第二阈值。
另一方面，概率值增加处理可以指将概率值赋值为一个更大的值，例如：1、0.99、0.98等；概率值降低处理可以指将概率值赋值为一个更小的值，例如：0.5、0.6等。例如，对于概率值为0.9的像素点，如果根据蓝通道像素值确定其极有可能属于天空区域(例如，蓝色通道B对应的值较高)，则可以将其概率值赋值为1；而对于概率值为0.5的像素点，如果根据颜色通道信息确定其极有可能不属于天空区域，则可以将其概率值减半或者降低为0。
在一些示例性实施例中，对第一像素点的概率值的减半处理也可以换成统一减去某个概率值(例如：统一减去0.1、0.2等)等。
上述实施例通过将极有可能是天空区域的像素点进行概率值增加处理，对极有可能不是天空区域的像素点进行概率值降低处理，能够突出天空区域的概率值，以便后续进行天空区域替换时能准确识别出这一部分区域，进而进行换天处理，以提高换天的准确性。
在一示例性实施例中，所述将所述第一像素点的概率值增加，并将所述第二像素点的概率值减少的步骤，包括：将所述第一像素点的概率值设置为1；将所述第二像素点的概率值减半。
在一种可选的实施方式中，对像素点进行增加减少处理的实现过程可以 为：计算待处理图片在概率值>0.9区域内各个像素点的蓝通道像素值的直方图，根据直方图计算蓝通道像素值在以下哪个区间最多(Q1:0-63、Q2:64-127、Q3:128-191、Q4:192-255)，将其中包含最多像素点的区间作为目标分布区间，记为Qi，将目标分布区间中概率值大于0.5且蓝通道值属于Qi区间的概率值置为1.0。其中对区间进行划分的方式可以根据实际情况调整，例如，可以划分为更大或更小的区间。
上述实施例通过将极有可能是天空区域的第一像素点设置为1，将极有可能不是天空区域的第二像素点进行减半处理，能够突出天空区域的概率值，以便后续进行天空区域替换时能准确识别出这一部分区域，进而进行换天处理，以提高换天的准确性。
在一示例性实施例中，除了根据蓝通道像素值确定第一像素点和第二像素点，也可以根据其他通道像素值来确定，如：红通道像素值、绿通道像素值等。在一种可选的实施方式中，对于主要进行傍晚天空区域替换处理的场景，可以采用红通道像素值来实现，这样的处理方式能够针对在一种可选的实施方式中场景确定出更为准确的天空区域，进而实现准确的天空区域替换操作，提高成片率。
另外，在一些示例性实施例中，可以先对第二像素点进行概率值减少处理，之后对概率值减少处理后的概率图进行第一像素点的概率值增加处理，得到候选掩码图；还可以是分别对初始掩码图进行第一像素点的概率值增加处理，对初始掩码图进行第二像素点的概率值减少处理，将两种处理方式得到的概率图进行整合(例如：对于初始掩码图中没有变化的像素点保留其原始的概率值；对于仅进行了概率值增加处理或仅进行了概率值较少处理的，则保留处理后的概率值；对于既进行了概率值增加处理又进行了概率值减少处理的，则可以取两种处理方式得到的概率值的平均值)，得到候选掩码图。之后，以待处理图片的灰度图为导向图对所述候选掩码图进行导向滤波处理，得到所述目标掩码图。
在一示例性实施例中，所述根据所述目标掩码图和所述目标天空场景对所述待处理图片中的天空区域进行替换处理，得到第一已处理图片的步骤，包括：确定所述待处理图片中的非天空区域，作为前景图；根据所述目标天空场景和所述天空区域的尺寸对天空素材图进行裁剪，得到场景与所述目标天空场景对应且尺寸与所述天空区域的尺寸对应的目标天空图；根据所述目标掩码图对所述前景图和所述目标天空图进行组合，得到所述第一已处理图片；其中，所述第一已处理图片中的天空区域被所述目标天空图替换。
其中，确定待处理图片中的非天空区域可以是将概率值低于0.85(也可以为其他值，或者结合蓝颜色通道值进行确定)的像素点确定为非天空区域像素点，对这些非天空区域像素点进行整合(例如：将零散的点整合为完整的区域)即可以得到非天空区域。
在一种可选的实施方式中，获取目标天空图的过程可以为：确定天空区域的最小矩形包围盒，从天空素材图中获取与目标天空场景对应的候选天空素材图，在保持长宽比的情况下，在候选天空素材图中裁剪出目标天空图。 进一步的，在裁剪目标天空图时可以采用中心裁切的方式(当然，也可以为其他方式)，例如：确定所述候选天空素材图的中心点，以该中心点为中心裁切至矩形包围盒大小得到目标天空图。更进一步的，可以对候选天空素材图进行缩放处理后再进行裁切；当然，也可以先在候选天空素材图中进行裁切，对裁切得到的图片再进行尺寸缩放处理，以得到目标天空图。
根据目标掩码图对前景图和目标天空图进行组合得到已处理图片的实现过程可以为：在一个空白区域内确定出一个与待处理图片尺寸一致的区域，作为待填充区域；根据目标掩码图在该待填充区域中确定出天空区域，将目标天空图填充到该天空区域内，将前景图填充到剩下的区域内，得到已处理图片。也可以为，根据目标掩码图在待处理图片中确定出天空区域，将目标天空图填充到该天空区域内，将前景图填充到剩下的区域内，得到已处理图片。如果完成区域填充后，前景图和目标天空图之间存在重合的区域，则可以对重合区域内的概率值进行整合，得到一个最终的概率值，进而根据这个最终的概率值完成区域填充，例如，如果确定是天空区域(例如：概率值大于0.5)则填充目标天空图，如果确定是非天空区域则填充前景图。假设待处理图片为图4所示，其中两个矩形框构成的区域表示天空区域401，天空区域401当前为彩虹场景。将其替换为晴天场景后可以如图5所示，由图5可知，彩虹场景的天空区域401被替换为了晴天场景的天空区域501；将其替换为多云场景后可以如图6所示，由图6可知，彩虹场景的天空区域401被替换为了多云场景的天空区域601。
从上述处理过程可以看出，天空区域已经被替换为了目标天空图，即实现了天空区域的替换处理，而其中的建筑物(前景图)则没有被替换，上述过程将天空区域和前景图进行区分处理，能有效防止图片内容的混叠，在满足用户需求的同时保证所得到的已处理图片的清晰度。
进一步的，在一示例性实施例中,所述根据所述目标掩码图和所述目标天空场景对所述待处理图片中的天空区域进行替换处理，得到第一已处理图片的步骤，包括：根据所述目标掩码图，从所述待处理图片中确定第一区域、第二区域以及剩余区域；其中，所述第一区域内部所包含的像素点的概率值为1，所述第二区域内部所包含的像素点的概率值为0，所述剩余区域为所述待处理图片中除去所述第一区域和所述第二区域的区域；将所述第一区域替换为所述目标天空图；将所述第二区域替换为所述前景图；根据所述剩余区域对应的概率值、红通道像素值、绿通道像素值以及蓝通道像素值对所述前景图和所述目标天空图进行像素点的颜色通道信息融合；根据颜色通道信息融合处理后的所述目标天空图得到所述第一已处理图片。
第一区域可以指极有可能或者肯定是天空区域的区域，第二区域可以指极有可能或者肯定不是天空区域的区域。需要说明的是，上述概率值为1的情况也可以将概率值替换为其他值，例如：0.98、0.99等等；上述概率值为0的情况也可以将概率值替换为其他值，例如：0.02、0.01等等。
颜色通道信息可以指图片在RGB三个通道对应的值，通过这些颜色通道信息可以获知对应像素点的颜色。进一步的，颜色通道信息可以是某一个通 道或者多个通道对应的值，在一种可选的实施方式中，可以包括红通道像素值、绿通道像素值以及蓝通道像素值。对前景图和目标天空图进行融合可以是对这两部分所对应的颜色通道信息进行运算处理，例如：进行正片叠底处理、伽玛变换处理等，将处理后得到的颜色通道信息作为对应像素点的颜色通道信息。该颜色通道信息可以是RGB值中的全部或部分信息。举例如下：正片叠底可以为将前景图和目标天空图对应像素点的RGB值进行相乘处理，伽玛变换处理可以是对前景图和目标天空图对应像素点的RGB值进行幂运算处理。
在一示例性实施例中，所述根据所述目标掩码图对所述前景图和所述目标天空图进行组合，得到所述第一已处理图片的步骤，包括：根据目标天空场景对所述前景图的亮度、对比度和饱和度中的至少一项进行调整，以得到亮度、对比度和饱和度与所述目标天空场景相匹配的目标前景图；根据所述目标掩码图对所述目标前景图和所述目标天空图进行组合，得到所述第一已处理图片。通过这样处理得到的第一已处理图片能够适应目标天空场景的风格。另外，还可以对前景图进行滤镜美化处理，得到该目标前景图。
在一种可选的实施方式中，本实施例的实现过程可以为：
对于概率值为1.0的区域直接使用目标天空图，对于概率值为0.0的区域直接使用前景图；需要说明的是，上述替换过程中图片的位置、角度要与对应的区域相匹配，例如：处于竖直状态的建筑物再替换到第一已处理图片中后也需要处于竖直状态。
对于概率值为0.0至1.0之间的区域：
首先根据概率值融合目标天空图和前景图，其中融合得到的某一像素点A的mixRGB如下：
即mixRGB＝src*(1.0-mask)+sky*mask；其中，src表示前景图中像素点A的RGB值，mask表示像素点A在参考概率图中的概率值，sky表示目标天空图中像素点A的RGB值。
然后通过正片叠底和伽玛变换处理mixRGB得到tempRGB：
即tempRGB＝sqrt(mixRGB*sky,0.5)；其中，0.5为预设参数，也可以根据需要设置为其他值。
最后，对于概率值介于0.5至1.0之间的像素点使用如下公式融合：
resRGB＝sky*(2.0*mask-1.0)+tempRGB*(2.0-2.0*mask)；
对于概率值介于0.0至0.5之间的使用如下公式融合：
resRGB＝tempRGB*(2.0*mask-1.0)+src*(2.0-2.0*mask)。
其他像素点按照上述同样的方式进行融合。
上述对前景图和目标天空图进行组合的过程，将肯定为天空区域的区域替换为目标天空图，将肯定不是天空区域的区域替换为前景图，经过上述处理过程，前景图和目标天空图之间的区域如果不进行处理则会存在前景突然换到天空区域的割裂感，本实施例将中间部分的区域(既不是前景图也不是目标天空图的区域)进行融合处理，融合后的区域整合了天空部分和非天空部分的颜色通道信息，能保证前景自然地过渡到天空，得到更为真实自然的 图片。
在一示例性实施例中，在完成天空区域替换处理后，可以包括对已处理图片进行纠正的步骤，例如，将已处理图片与待处理图片进行比对，如果其中的前景图或者目标天空图存在位置、角度偏差，则可以进行调整，如果存在边缘过渡不自然的情况，则可以进行调整。通过这样的方式，能使得最终输出的已处理图片具有较高的准确性和天空区域替换效果，提高成片率。
在一示例性实施例中,所述对待处理图片进行图像分割处理，根据图像分割处理结果得到初始掩码图的步骤，包括：通过预先训练的神经网络模型对所述待处理图片进行图像分割处理，得到图像分割处理结果；根据图像分割处理结果确定所述待处理图片中各个像素点属于天空区域内像素点的概率值；根据所述待处理图片中各个像素点的所述概率值，得到所述初始掩码图。
其中，神经网络模型可以是CNN(卷积神经网络)模型等。在一种可选的实施方式中，该神经网络可以为u-net、u-net变体、ic-net、deeplab系列等等。
进一步的，在一示例性实施例中，所述通过预先训练的神经网络模型对所述待处理图片进行图像分割处理的步骤，包括：将所述待处理图片的尺寸缩放至预设尺寸；对尺寸缩放后的待处理图片进行归一化处理；通过预先训练的神经网络模型对经过归一化处理的所述待处理图片进行图像分割处理。
其中，预设尺寸可以根据实际情况设置，例如为：512*512、768*768等。归一化处理可以是：获取图片样本，确定这些图片样本各个像素点RGB值的均值、方差，将待处理图片各个像素点的RGB值减去该均值、除以该方差，以便更好地进行机器学习和进行特征学习，进而对待处理图片中各个像素点是否属于天空区域像素点进行分类，得到概率图。
神经网络模型可以通过预先确定的训练图片训练得到，已训练的神经网络模型可以对待处理图片进行下采样处理，进而提取其中的特征信息，对特征信息进行分析，输出图像分割处理结果，此时计算机设备可以根据图像分割处理结果确定各个像素点属于天空区域的概率值，进而得到初始掩码图。神经网络模型充分分析待处理图片中的特征信息，相对于传统的graph cut的图像分割方法，这种机器学习的方式能更为准确地对待处理图片进行分割处理，进而得到准确的概率图。
上述实施例中，在获取到待处理图片时，先对其进行预处理，然后通过神经网络模型进行分割网络处理得到初始掩码图，进而根据初始掩码图从待处理图片中准确区分出天空区域(可以理解为后处理过程)，将分割网络以及后处理进行结合，保留了神经网络模型准确分割的效果，在此基础上又进行了准确的天空区域和非天空区域后处理，使得最终的天空区域替换处理更为准确。
在一示例性实施例中,所述根据所述初始掩码图确定所述待处理图片是否满足预设的天空区域替换条件的步骤，包括：根据所述初始掩码图确定所述待处理图片是否符合以下至少一个，若符合以下至少一个，判定所述待处理图片不满足天空区域替换条件，若不符合以下任意一个，判定所述待处理图片满足天空区域替换条件：确定所述待处理图片中天空区域的第一占比，若 所述第一占比小于预设的第四阈值，则判定为天空区域太小；确定所述待处理图片中不置信区域的第二占比，若所述第二占比大于预设的第五阈值，则判定为不置信；其中，所述不置信区域为各个像素点的概率值处于中部区间的区域，所述中部区间由所述概率值的中值以及所述中值的临近值构成；确定所述待处理图片中天空区域的平均亮度，若所述平均亮度小于预设的第六阈值，则判定为夜晚场景；确定所述待处理图片中目标暗通道区域的第三占比，若所述第三占比大于预设的第七阈值，则判定为雾天场景；其中，所述目标暗通道区域为天空区域中暗通道值小于第八阈值的像素点所在区域。
第四、五、六、七、八阈值的大小可以根据实际情况确定，例如，第四阈值可以为0.9，第五阈值可以取值为0.1，第六阈值可以为0.3cd/m 2，第七阈值可以取值为0.4，第八阈值可以取值为0.8。
概率值为1的像素点可以理解为肯定属于天空区域，概率值为0的像素点可以理解为肯定不属于天空区域，而对于处于中间概率值(例如：0.5)的像素点，则有可能属于天空区域也有可能不属于天空区域，这部分像素点属于不确定的区域，即不置信区域。在一种可选的实施方式中，不置信区域可以为概率值落在0.3至0.7之间的区域(区间的边界值也可以为其他值)。需要说明的是，在本申请实施例中，如果需要确定天空区域，则可以从当前步骤已经得到的概率图中提取概率值大于0.9(也可以为其他值)的区域，作为天空区域，不同实施例中的天空区域可以相同也可以不同。
确定待处理图片中天空区域的平均亮度的实现过程可以为：计算初始掩码图中概率值大于0.9的区域对应的原图(待处理图片)区域的平均灰度值。
在确定天空区域的暗通道值(对于RGB图，暗通道值为取RGB三个值中的最小值，例如，R/G/B分别为10/20/30，则暗通道取值为10)后，可以得到暗通道图，然后基于暗通道图判断待处理图片是否为雾天场景(如果暗通道值小于某个阈值的区域占比大于第七阈值，则认为是雾天)。其中，目标暗通道区域的第三占比可以理解为暗通道值的统计信息。
在一种可选的实施方式中，上述实施例的实现过程可以为：
(1)根据初始掩码图统计待处理图片中天空区域在整图中的第一占比，若第一占比小于0.9则判断为天空区域太小；
(2)根据初始掩码图统计待处理图片中天空区域的平均亮度，若平均亮度小于阈值0.3cd/m 2则判断为夜景；
(3)根据初始掩码图统计待处理图片中概率值落在0.3至0.7之间的区域在整图中的第二占比，若第二占比大于阈值0.1则判断为初始掩码图不置信；
(4)根据初始掩码图得到待处理图片中天空区域的暗通道图，然后基于暗通道图的统计信息判断是否为雾天，如果目标暗通道值的第三占比小于0.8，则判断为雾天场景。
如果满足上述任意一个条件，则可以判定待处理图片不适合进行天空区域的替换处理，即不适合换天，否则，如果上述条件均不满足(或者大部分都不满足)，则可以认为适合换天，则进行后续的换天处理。
进一步的，在一示例性实施例中，在所述根据所述初始掩码图确定所述 待处理图片是否满足预设的天空区域替换条件的步骤之后，还包括：若不满足天空区域替换条件，根据所述目标天空场景获取目标滤镜工具，通过所述目标滤镜工具对所述待处理图片进行滤镜处理，得到第二已处理图片。
在一种可选的实施方式中，如果待处理图片不符合天空区域替换条件，则可以进行以下处理：获取预先确定的目标天空场景；选择与所述目标天空场景对应的综合滤镜；通过所述综合滤镜对所述待处理图片进行处理，得到第二已处理图片。
在一示例性实施例中，如图7所示，提供一种图像处理方法，包括以下步骤：
S701、通过预先训练的神经网络模型对待处理图片进行图像分割处理；根据图像分割处理结果确定所述待处理图片中各个像素点属于天空区域内像素点的概率值；根据所述概率值，得到初始掩码图。
S702、根据所述初始掩码图确定所述待处理图片是否满足预设的天空区域替换条件，若符合天空区域替换条件，则执行S703，若不符合天空区域替换条件，则执行S712。
S703、获取所述待处理图片中各个像素点的蓝通道像素值。
S704、将所述初始掩码图中蓝通道像素值处于目标分布区间且所述概率值大于第一阈值的像素点确定为第一像素点。
S705、将所述初始掩码图中蓝通道像素值小于目标蓝通道像素值的像素点确定为第二像素点。
S706、将所述第一像素点的概率值设置为1，将所述第二像素点的概率值减半，得到参考掩码图。
S707、获取所述待处理图片的灰度图；以所述灰度图中各个像素点的灰度值为导向对所述参考掩码图进行导向滤波处理，得到目标掩码图。
S708、根据所述目标掩码图，从所述待处理图片中确定第一区域、第二区域以及剩余区域。
S709、根据所述目标天空场景和所述天空区域的尺寸对天空素材图进行裁剪，得到场景与所述目标天空场景对应且尺寸与所述天空区域的尺寸对应的目标天空图。
S710、确定所述待处理图片中的非天空区域，作为前景图。
S711、将所述第一区域替换为所述目标天空图；将所述第二区域替换为所述前景图；根据所述剩余区域对应的概率值、红通道像素值、绿通道像素值以及蓝通道像素值对所述前景图和所述目标天空图进行像素点的颜色通道信息融合；根据颜色通道信息融合处理后的所述目标天空图得到第一已处理图片。
S712、根据所述目标天空场景获取目标滤镜工具，通过所述目标滤镜工具对所述待处理图片进行滤镜处理，得到第二已处理图片。
上述图像处理方法中，对待处理图片进行图像分割处理，准确得到包含有待处理图片中各个像素点属于天空区域的概率值的初始掩码图；根据待处理图片中各个像素点的蓝通道像素值，对初始掩码图中与该天空区域对应的 像素点进行概率值增加处理，得到参考概率图，所得的参考概率图进一步扩大了天空区域和非天空区域之间的概率值差别；根据参考概率图准确识别出待处理图片中的天空区域，因此，根据天空区域替换处理所得的已处理图片具有较高的准确性，且换天效果自然，换天成片率高。
在一示例性实施例中，如图8所示，提供一种图像处理方法，该方法应用于移动端，包括以下步骤：
一、利用CNN分割网络抠图
1、进行图像预处理：将待处理图片缩放至固定大小，然后归一化。
2、送入分割网络处理：将预处理后的待处理图片送入分割网络得到概率图。
二、根据分割网络输出进行后处理：
1、根据概率图确定待处理图片是否适合换天。
2、若不适合换天，则根据目标天空场景选择相对应的综合滤镜作用于待处理图片直接得到已处理图片。
3、若适合换天则：
(1)基于统计信息优化概率图：统计待处理图片在概率值>0.95区域的蓝通道的最小值minBlue；计算待处理图片在概率值>0.9区域的蓝通道的直方图，根据直方图计算蓝通道像素值最多的区间Qi；将概率图中蓝通道小于minBlue的区域对应的概率值减半；将概率图中概率值大于0.5且蓝通道属于Qi区间的概率值置为1.0；
(2)导向滤波羽化：将待处理图片从RGB图转为灰度图，然后以灰度图为导向图对优化后的概率图进行导向滤波得到羽化概率图；
(3)裁剪天空素材：根据羽化概率图计算待处理图片中天空区域的最小矩形包围盒，在保持长宽比的情况下中心裁切和缩放天空素材至矩形包围盒大小；
(4)自适应调整前景：根据目标天空场景调整待处理图片中前景图的亮度、对比度、饱和度以符合素材的风格。然后使用与目标天空场景对应的前景滤镜进行美化得到最终使用的前景图；
(5)分段融合：根据羽化概率图融合调整后的前景图和裁剪后的天空素材；
(6)根据目标天空场景选取相对应的整体滤镜，通过该整体滤镜对融合后的图片进行处理，得到已处理图片。
上述实施例具有以下技术效果：
1)结合AI分割模型和后处理中的分割结果优化，能够更好地完成天空区域的精准抠图，有效避免因抠图错误导致的瑕疵，如天空中电线丢失、树洞中天空未换等；
2)后处理中将新的天空区域与原图背景融合采用分段图层混叠和分段线性融合的方式，使得合成图片在天空与非天空处的过渡真实自然；
3)根据素材类型(目标天空场景)对非天空区域即前景图进行针对性调整，以及最后整体调色，保证了最终效果图的统一和美观；
4)综合以上3点实现了一种高成片率的自然而真实的换天效果，极大提升用户在移动端上处理室外天空照片的一键P图体验。
应该理解的是，虽然图2、7、8的流程图中的各个步骤按照箭头的指示依次显示，但是这些步骤并不是必然按照箭头指示的顺序依次执行。除非本文中有明确的说明，这些步骤的执行并没有严格的顺序限制，这些步骤可以以其它的顺序执行。而且，图2、7、8中的至少一部分步骤可以包括多个步骤或者多个阶段，这些步骤或者阶段并不必然是在同一时刻执行完成，而是可以在不同的时刻执行，这些步骤或者阶段的执行顺序也不必然是依次进行，而是可以与其它步骤或者其它步骤中的步骤或者阶段的至少一部分轮流或者交替地执行。
图9是根据一示例性实施例示出的一种图像处理装置900的框图。参照图9，该装置包括掩码图确定单元901，导向滤波单元902，天空场景获取单元903和天空区域替换单元904。
掩码图确定单元901，被配置为执行对待处理图片进行图像分割处理，根据图像分割处理结果得到初始掩码图；所述初始掩码图包含有所述待处理图片中各个像素点属于天空区域内像素点的概率值。
导向滤波单元902，被配置为执行根据所述初始掩码图确定所述待处理图片是否满足预设的天空区域替换条件，若满足所述天空区域替换条件，以所述待处理图片的灰度图为导向图对所述初始掩码图进行导向滤波处理，得到目标掩码图。
天空场景获取单元903，被配置为执行获取目标天空场景；所述目标天空场景从预设的天空场景素材中选择得到。
天空区域替换单元904，被配置为执行根据所述目标掩码图和所述目标天空场景对所述待处理图片中的天空区域进行替换处理，得到第一已处理图片。
在一示例性实施例中,所述导向滤波单元，包括：像素值获取子单元，被配置为执行获取所述待处理图片中各个像素点的蓝通道像素值；第一像素点确定子单元，被配置为执行将所述初始掩码图中蓝通道像素值处于目标分布区间且所述概率值大于第一阈值的像素点确定为第一像素点；其中，所述目标分布区间为预设的多个区间中，第一评价区域中各个像素点的蓝通道像素值数量最多的区间，所述第一评价区域为所述初始掩码图中所述概率值大于第二阈值的像素点所在区域；所述第二阈值大于所述第一阈值；第二像素点确定子单元，被配置为执行将所述初始掩码图中蓝通道像素值小于目标蓝通道像素值的像素点确定为第二像素点；所述目标蓝通道像素值为第二评价区域中各个像素点的蓝通道像素值的最小值，所述第二评价区域为所述初始掩码图中所述概率值大于第三阈值的像素点所在区域；所述第三阈值大于所述第二阈值；概率值处理子单元，被配置为执行将所述第一像素点的概率值增加，并将所述第二像素点的概率值减少，得到参考掩码图；导向滤波子单元，被配置为执行以所述待处理图片的灰度图为导向图对所述参考掩码图进行导向滤波处理，得到所述目标掩码图。
在一示例性实施例中,所述导向滤波子单元，包括：第一概率值设置模块， 被配置为执行将所述第一像素点的概率值设置为1；第二概率值设置模块，被配置为执行将所述第二像素点的概率值减半。
在一示例性实施例中,所述天空区域替换单元，包括：前景图确定子单元，被配置为执行确定所述待处理图片中的非天空区域，作为前景图；天空素材裁剪子单元，被配置为执行根据所述目标天空场景和所述天空区域的尺寸对天空素材图进行裁剪，得到场景与所述目标天空场景对应且尺寸与所述天空区域的尺寸对应的目标天空图；第一前景组合子单元，被配置为执行根据所述目标掩码图对所述前景图和所述目标天空图进行组合，得到所述第一已处理图片；其中，所述第一已处理图片中的天空区域被所述目标天空图替换。
在一示例性实施例中,所述天空区域替换单元，还包括：区域确定子单元，被配置为执行根据所述目标掩码图，从所述待处理图片中确定第一区域、第二区域以及剩余区域；其中，所述第一区域内部所包含的像素点的概率值为1，所述第二区域内部所包含的像素点的概率值为0，所述剩余区域为所述待处理图片中除去所述第一区域和所述第二区域的区域；天空图替换子单元，被配置为执行将所述第一区域替换为所述目标天空图；前景图替换子单元，被配置为执行将所述第二区域替换为所述前景图；通道信息融合子单元，被配置为执行根据所述剩余区域对应的概率值、红通道像素值、绿通道像素值以及蓝通道像素值对所述前景图和所述目标天空图进行像素点的颜色通道信息融合；已处理图片获取子单元，被配置为执行根据颜色通道信息融合处理后的所述目标天空图得到所述第一已处理图片。
在一示例性实施例中,所述天空区域替换单元，还包括：前景图调整子单元，被配置为执行根据目标天空场景对所述前景图的亮度、对比度和饱和度中的至少一项进行调整，以得到亮度、对比度和饱和度与所述目标天空场景相匹配的目标前景图；第二前景组合子单元，被配置为执行根据所述目标掩码图对所述目标前景图和所述目标天空图进行组合，得到所述第一已处理图片。
在一示例性实施例中,所述导向滤波单元，还被配置为执行根据所述初始掩码图确定所述待处理图片是否符合以下至少一个预设条件，若符合以下至少一个预设条件，判定所述待处理图片不满足天空区域替换条件，若不符合以下任意一个预设条件，判定所述待处理图片满足天空区域替换条件：
预设条件1：所述待处理图片中天空区域的第一占比小于预设的第四阈值；
预设条件2：所述待处理图片中不置信区域的第二占比大于预设的第五阈值；其中，所述不置信区域为各个像素点的概率值处于中部区间的区域，所述中部区间由所述概率值的中值以及所述中值的临近值构成；
预设条件3：所述待处理图片中天空区域的平均亮度小于预设的第六阈值；
预设条件4：所述待处理图片中目标暗通道区域的第三占比大于预设的第七阈值；其中，所述目标暗通道区域为天空区域中暗通道值小于第八阈值的像素点所在区域。
在一示例性实施例中,所述图像处理装置，还包括：滤镜处理单元，被配置为执行若不满足天空区域替换条件，根据所述目标天空场景获取目标滤镜工具，通过所述目标滤镜工具对所述待处理图片进行滤镜处理，得到第二已处理图片。
关于上述实施例中的装置，其中各个模块执行操作的具体方式已经在有关该方法的实施例中进行了详细描述，此处将不做详细阐述说明。
在示例性实施例中，还提供了一种包括指令的非临时性计算机可读存储介质，例如包括指令的存储器102，上述指令可由设备100的处理器109执行以完成上述方法。例如，所述非临时性计算机可读存储介质可以是ROM、随机存取存储器(RAM)、CD-ROM、磁带、软盘和光数据存储设备等。
根据本申请实施例的第五方面，提供一种计算机程序产品，所述程序产品包括计算机程序，所述计算机程序存储在可读存储介质中，设备的至少一个处理器从所述可读存储介质读取并执行所述计算机程序，使得设备执行如上实施例中所述的图像处理方法。
本领域技术人员在考虑说明书及实践这里公开的发明后，将容易想到本申请的其它实施方案。本申请旨在涵盖本申请的任何变型、用途或者适应性变化，这些变型、用途或者适应性变化遵循本申请的一般性原理并包括本申请未公开的本技术领域中的公知常识或惯用技术手段。说明书和实施例仅被视为示例性的，本申请的真正范围和精神由下面的权利要求指出。
应当理解的是，本申请并不局限于上面已经描述并在附图中示出的精确结构，并且可以在不脱离其范围进行各种修改和改变。本申请的范围仅由所附的权利要求来限制。

Claims (18)

  1. 一种图像处理方法，其特征在于，包括：
    对待处理图片进行图像分割处理，根据图像分割处理结果得到初始掩码图；所述初始掩码图包含有所述待处理图片中各个像素点属于天空区域内像素点的概率值；
    根据所述初始掩码图确定所述待处理图片是否满足预设的天空区域替换条件，若满足所述天空区域替换条件，以所述待处理图片的灰度图为导向图对所述初始掩码图进行导向滤波处理，得到目标掩码图；
    获取目标天空场景；所述目标天空场景为从预设的天空场景素材中选择得到；
    根据所述目标掩码图和所述目标天空场景对所述待处理图片中的天空区域进行替换处理，得到第一已处理图片。
  2. 根据权利要求1所述的图像处理方法，其特征在于，所述以所述待处理图片的灰度图为导向图对所述初始掩码图进行导向滤波处理，得到目标掩码图，包括：
    获取所述待处理图片中各个像素点的蓝通道像素值；
    将所述初始掩码图中蓝通道像素值处于目标分布区间且所述概率值大于第一阈值的像素点确定为第一像素点；其中，所述目标分布区间为预设的多个区间中，第一评价区域中各个像素点的蓝通道像素值数量最多的区间，所述第一评价区域为所述初始掩码图中所述概率值大于第二阈值的像素点所在区域；所述第二阈值大于所述第一阈值；
    将所述初始掩码图中蓝通道像素值小于目标蓝通道像素值的像素点确定为第二像素点；所述目标蓝通道像素值为第二评价区域中各个像素点的蓝通道像素值的最小值，所述第二评价区域为所述初始掩码图中所述概率值大于第三阈值的像素点所在区域；所述第三阈值大于所述第二阈值；
    将所述第一像素点的概率值增加，并将所述第二像素点的概率值减少，得到参考掩码图；
    以所述待处理图片的灰度图为导向图对所述参考掩码图进行导向滤波处理，得到所述目标掩码图。
  3. 根据权利要求2所述的图像处理方法，其特征在于，所述将所述第一像素点的概率值增加，并将所述第二像素点的概率值减少，包括：
    将所述第一像素点的概率值设置为1；
    将所述第二像素点的概率值减半。
  4. 根据权利要求3所述的图像处理方法，其特征在于，所述根据所述目标掩码图和所述目标天空场景对所述待处理图片中的天空区域进行替换处理，得到第一已处理图片，包括：
    确定所述待处理图片中的非天空区域，作为前景图；
    根据所述目标天空场景和所述天空区域的尺寸对天空素材图进行裁剪，得到场景与所述目标天空场景对应且尺寸与所述天空区域的尺寸对应的目标 天空图；
    根据所述目标掩码图对所述前景图和所述目标天空图进行组合，得到所述第一已处理图片；其中，所述第一已处理图片中的天空区域被所述目标天空图替换。
  5. 根据权利要求4所述的图像处理方法，其特征在于，所述根据所述目标掩码图和所述目标天空场景对所述待处理图片中的天空区域进行替换处理，得到第一已处理图片，包括：
    根据所述目标掩码图，从所述待处理图片中确定第一区域、第二区域以及剩余区域；其中，所述第一区域内部所包含的像素点的概率值为1，所述第二区域内部所包含的像素点的概率值为0，所述剩余区域为所述待处理图片中除去所述第一区域和所述第二区域的区域；
    将所述第一区域替换为所述目标天空图；
    将所述第二区域替换为所述前景图；
    根据所述剩余区域对应的概率值、红通道像素值、绿通道像素值以及蓝通道像素值对所述前景图和所述目标天空图进行像素点的颜色通道信息融合；
    根据颜色通道信息融合处理后的所述目标天空图得到所述第一已处理图片。
  6. 根据权利要求4所述的图像处理方法，其特征在于，所述根据所述目标掩码图对所述前景图和所述目标天空图进行组合，得到所述第一已处理图片，包括：
    根据目标天空场景对所述前景图的亮度、对比度和饱和度中的至少一项进行调整，以得到亮度、对比度和饱和度与所述目标天空场景相匹配的目标前景图；
    根据所述目标掩码图对所述目标前景图和所述目标天空图进行组合，得到所述第一已处理图片。
  7. 根据权利要求1所述的图像处理方法，其特征在于，所述根据所述初始掩码图确定所述待处理图片是否满足预设的天空区域替换条件，包括：
    根据所述初始掩码图确定所述待处理图片是否符合以下至少一个预设条件，若符合以下至少一个预设条件，判定所述待处理图片不满足天空区域替换条件，若不符合以下任意一个预设条件，判定所述待处理图片满足天空区域替换条件：
    预设条件1：所述待处理图片中天空区域的第一占比小于预设的第四阈值；
    预设条件2：所述待处理图片中不置信区域的第二占比大于预设的第五阈值；其中，所述不置信区域为各个像素点的概率值处于中部区间的区域，所述中部区间由所述概率值的中值以及所述中值的临近值构成；
    预设条件3：所述待处理图片中天空区域的平均亮度小于预设的第六阈值；
    预设条件4：所述待处理图片中目标暗通道区域的第三占比大于预设的第 七阈值；其中，所述目标暗通道区域为天空区域中暗通道值小于第八阈值的像素点所在区域。
  8. 根据权利要求7所述的图像处理方法，其特征在于，在所述根据所述初始掩码图确定所述待处理图片是否满足预设的天空区域替换条件之后，还包括：
    若不满足天空区域替换条件，根据所述目标天空场景获取目标滤镜工具，通过所述目标滤镜工具对所述待处理图片进行滤镜处理，得到第二已处理图片。
  9. 一种图像处理装置，其特征在于，包括：
    掩码图确定单元，被配置为执行对待处理图片进行图像分割处理，根据图像分割处理结果得到初始掩码图；所述初始掩码图包含有所述待处理图片中各个像素点属于天空区域内像素点的概率值；
    导向滤波单元，被配置为执行根据所述初始掩码图确定所述待处理图片是否满足预设的天空区域替换条件，若满足所述天空区域替换条件，以所述待处理图片的灰度图为导向图对所述初始掩码图进行导向滤波处理，得到目标掩码图；
    天空场景获取单元，被配置为执行获取目标天空场景；所述目标天空场景为从预设的天空场景素材中选择得到；
    天空区域替换单元，被配置为执行根据所述目标掩码图和所述目标天空场景对所述待处理图片中的天空区域进行替换处理，得到第一已处理图片。
  10. 根据权利要求9所述的图像处理装置，其特征在于，所述导向滤波单元，包括：
    像素值获取子单元，被配置为执行获取所述待处理图片中各个像素点的蓝通道像素值；
    第一像素点确定子单元，被配置为执行将所述初始掩码图中蓝通道像素值处于目标分布区间且所述概率值大于第一阈值的像素点确定为第一像素点；其中，所述目标分布区间为预设的多个区间中，第一评价区域中各个像素点的蓝通道像素值数量最多的区间，所述第一评价区域为所述初始掩码图中所述概率值大于第二阈值的像素点所在区域；所述第二阈值大于所述第一阈值；
    第二像素点确定子单元，被配置为执行将所述初始掩码图中蓝通道像素值小于目标蓝通道像素值的像素点确定为第二像素点；所述目标蓝通道像素值为第二评价区域中各个像素点的蓝通道像素值的最小值，所述第二评价区域为所述初始掩码图中所述概率值大于第三阈值的像素点所在区域；所述第三阈值大于所述第二阈值；
    概率值处理子单元，被配置为执行将所述第一像素点的概率值增加，并将所述第二像素点的概率值减少，得到参考掩码图；
    导向滤波子单元，被配置为执行以所述待处理图片的灰度图为导向图对所述参考掩码图进行导向滤波处理，得到所述目标掩码图。
  11. 根据权利要求10所述的图像处理装置，其特征在于，所述导向滤波 子单元，包括：
    第一概率值设置模块，被配置为执行将所述第一像素点的概率值设置为1；
    第二概率值设置模块，被配置为执行将所述第二像素点的概率值减半。
  12. 根据权利要求11所述的图像处理装置，其特征在于，所述天空区域替换单元，包括：
    前景图确定子单元，被配置为执行确定所述待处理图片中的非天空区域，作为前景图；
    天空素材裁剪子单元，被配置为执行根据所述目标天空场景和所述天空区域的尺寸对天空素材图进行裁剪，得到场景与所述目标天空场景对应且尺寸与所述天空区域的尺寸对应的目标天空图；
    第一前景组合子单元，被配置为执行根据所述目标掩码图对所述前景图和所述目标天空图进行组合，得到所述第一已处理图片；其中，所述第一已处理图片中的天空区域被所述目标天空图替换。
  13. 根据权利要求12所述的图像处理装置，其特征在于，所述天空区域替换单元，还包括：
    区域确定子单元，被配置为执行根据所述目标掩码图，从所述待处理图片中确定第一区域、第二区域以及剩余区域；其中，所述第一区域内部所包含的像素点的概率值为1，所述第二区域内部所包含的像素点的概率值为0，所述剩余区域为所述待处理图片中除去所述第一区域和所述第二区域的区域；
    天空图替换子单元，被配置为执行将所述第一区域替换为所述目标天空图；
    前景图替换子单元，被配置为执行将所述第二区域替换为所述前景图；
    通道信息融合子单元，被配置为执行根据所述剩余区域对应的概率值、红通道像素值、绿通道像素值以及蓝通道像素值对所述前景图和所述目标天空图进行像素点的颜色通道信息融合；
    已处理图片获取子单元，被配置为执行根据颜色通道信息融合处理后的所述目标天空图得到所述第一已处理图片。
  14. 根据权利要求12所述的图像处理装置，其特征在于，所述天空区域替换单元，还包括：
    前景图调整子单元，被配置为执行根据目标天空场景对所述前景图的亮度、对比度和饱和度中的至少一项进行调整，以得到亮度、对比度和饱和度与所述目标天空场景相匹配的目标前景图；
    第二前景组合子单元，被配置为执行根据所述目标掩码图对所述目标前景图和所述目标天空图进行组合，得到所述第一已处理图片。
  15. 根据权利要求9所述的图像处理装置，其特征在于，所述导向滤波单元，还被配置为执行根据所述初始掩码图确定所述待处理图片是否符合以下至少一个预设条件，若符合以下至少一个预设条件，判定所述待处理图片不满足天空区域替换条件，若不符合以下任意一个预设条件，判定所述待处 理图片满足天空区域替换条件：
    预设条件1：所述待处理图片中天空区域的第一占比小于预设的第四阈值；
    预设条件2：所述待处理图片中不置信区域的第二占比大于预设的第五阈值；其中，所述不置信区域为各个像素点的概率值处于中部区间的区域，所述中部区间由所述概率值的中值以及所述中值的临近值构成；
    预设条件3：所述待处理图片中天空区域的平均亮度小于预设的第六阈值；
    预设条件4：所述待处理图片中目标暗通道区域的第三占比大于预设的第七阈值；其中，所述目标暗通道区域为天空区域中暗通道值小于第八阈值的像素点所在区域。
  16. 根据权利要求15所述的图像处理装置，其特征在于，所述图像处理装置，还包括：
    滤镜处理单元，被配置为执行若不满足天空区域替换条件，根据所述目标天空场景获取目标滤镜工具，通过所述目标滤镜工具对所述待处理图片进行滤镜处理，得到第二已处理图片。
  17. 一种电子设备，其特征在于，包括：
    处理器；
    用于存储所述处理器可执行指令的存储器；
    其中，所述处理器被配置为执行所述指令，以实现如权利要求1至8中任一项所述的图像处理方法。
  18. 一种存储介质，当所述存储介质中的指令由电子设备的处理器执行时，使得所述电子设备能够执行如权利要求1至8中任一项所述的图像处理方法。
PCT/CN2020/127564 2020-04-23 2020-11-09 图像处理方法、装置、电子设备及存储介质 WO2021212810A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
JP2022548804A JP2023513726A (ja) 2020-04-23 2020-11-09 画像処理方法、装置、電子機器及び記録媒体
US17/883,165 US20220383508A1 (en) 2020-04-23 2022-08-08 Image processing method and device, electronic device, and storage medium

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202010327149.6A CN113554658A (zh) 2020-04-23 2020-04-23 图像处理方法、装置、电子设备及存储介质
CN202010327149.6 2020-04-23

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US17/883,165 Continuation US20220383508A1 (en) 2020-04-23 2022-08-08 Image processing method and device, electronic device, and storage medium

Publications (1)

Publication Number Publication Date
WO2021212810A1 true WO2021212810A1 (zh) 2021-10-28

Family

ID=78129377

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/127564 WO2021212810A1 (zh) 2020-04-23 2020-11-09 图像处理方法、装置、电子设备及存储介质

Country Status (4)

Country Link
US (1) US20220383508A1 (zh)
JP (1) JP2023513726A (zh)
CN (1) CN113554658A (zh)
WO (1) WO2021212810A1 (zh)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116363148A (zh) * 2022-06-21 2023-06-30 上海玄戒技术有限公司 图像处理方法、装置、芯片及存储介质
CN116600210A (zh) * 2023-07-18 2023-08-15 长春工业大学 基于机器人视觉的图像采集优化系统
GB2619381A (en) * 2022-05-13 2023-12-06 Adobe Inc Object class inpainting in digital images utilizing class-specific inpainting neural networks

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114004834B (zh) * 2021-12-31 2022-04-19 山东信通电子股份有限公司 一种图像处理中的雾天情况分析方法、设备及装置
CN114494298A (zh) * 2022-01-28 2022-05-13 北京字跳网络技术有限公司 对象分割方法、装置、设备及存储介质
CN115150390B (zh) * 2022-06-27 2024-04-09 山东信通电子股份有限公司 一种图像显示方法、装置、设备及介质

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20020085243A1 (en) * 2000-12-21 2002-07-04 Tomotoshi Kanatsu Document processing apparatus and method
CN105761230A (zh) * 2016-03-16 2016-07-13 西安电子科技大学 基于天空区域分割处理的单幅图像去雾方法
CN110533616A (zh) * 2019-08-30 2019-12-03 福建省德腾智能科技有限公司 一种图像天空区域分割的方法
CN111047540A (zh) * 2019-12-27 2020-04-21 嘉应学院 一种基于天空分割的图像去雾方法及其应用系统

Family Cites Families (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10074161B2 (en) * 2016-04-08 2018-09-11 Adobe Systems Incorporated Sky editing based on image composition
CN106447638A (zh) * 2016-09-30 2017-02-22 北京奇虎科技有限公司 一种美颜处理方法及装置
CN107025457B (zh) * 2017-03-29 2022-03-08 腾讯科技（深圳）有限公司 一种图像处理方法和装置
CN108280809B (zh) * 2017-12-26 2021-07-30 浙江工商大学 一种基于大气散射物理模型的有雾图像天空区域估计方法
CN109255759B (zh) * 2018-08-02 2021-06-15 辽宁师范大学 基于天空分割和透射率自适应修正的图像去雾方法
CN110782407B (zh) * 2019-10-15 2021-10-19 北京理工大学 一种基于天空区域概率分割的单幅图像去雾方法

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20020085243A1 (en) * 2000-12-21 2002-07-04 Tomotoshi Kanatsu Document processing apparatus and method
CN105761230A (zh) * 2016-03-16 2016-07-13 西安电子科技大学 基于天空区域分割处理的单幅图像去雾方法
CN110533616A (zh) * 2019-08-30 2019-12-03 福建省德腾智能科技有限公司 一种图像天空区域分割的方法
CN111047540A (zh) * 2019-12-27 2020-04-21 嘉应学院 一种基于天空分割的图像去雾方法及其应用系统

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
GB2619381A (en) * 2022-05-13 2023-12-06 Adobe Inc Object class inpainting in digital images utilizing class-specific inpainting neural networks
CN116363148A (zh) * 2022-06-21 2023-06-30 上海玄戒技术有限公司 图像处理方法、装置、芯片及存储介质
CN116363148B (zh) * 2022-06-21 2024-04-02 上海玄戒技术有限公司 图像处理方法、装置、芯片及存储介质
CN116600210A (zh) * 2023-07-18 2023-08-15 长春工业大学 基于机器人视觉的图像采集优化系统
CN116600210B (zh) * 2023-07-18 2023-10-10 长春工业大学 基于机器人视觉的图像采集优化系统

Also Published As

Publication number Publication date
CN113554658A (zh) 2021-10-26
US20220383508A1 (en) 2022-12-01
JP2023513726A (ja) 2023-04-03

Similar Documents

Publication Publication Date Title
WO2021212810A1 (zh) 图像处理方法、装置、电子设备及存储介质
CN108764091B (zh) 活体检测方法及装置、电子设备和存储介质
EP3125158B1 (en) Method and device for displaying images
US10091409B2 (en) Improving focus in image and video capture using depth maps
CN108701439B (zh) 一种图像显示优化方法及装置
US20170053156A1 (en) Human face recognition method, apparatus and terminal
CN111402135A (zh) 图像处理方法、装置、电子设备及计算机可读存储介质
EP3057304B1 (en) Method and apparatus for generating image filter
CN109712177B (zh) 图像处理方法、装置、电子设备和计算机可读存储介质
US9195880B1 (en) Interactive viewer for image stacks
KR102525293B1 (ko) 촬영 방법 및 장치, 단말기, 저장 매체
CN109784327B (zh) 边界框确定方法、装置、电子设备及存储介质
EP3933675B1 (en) Method and apparatus for detecting finger occlusion image, and storage medium
CN112017137A (zh) 图像处理方法、装置、电子设备及计算机可读存储介质
CN113610865B (zh) 图像处理方法、装置、电子设备及计算机可读存储介质
CN111383166B (zh) 处理待显示图像的方法及装置、电子设备、可读存储介质
CN113658197A (zh) 图像处理方法、装置、电子设备及计算机可读存储介质
CN106469446B (zh) 深度图像的分割方法和分割装置
CN114697530B (zh) 一种智能取景推荐的拍照方法及装置
CN113610884A (zh) 图像处理方法、装置、电子设备及计算机可读存储介质
WO2023230927A1 (zh) 图像处理方法、装置及可读存储介质
CN113256503B (zh) 图像优化方法及装置、移动终端及存储介质
CN115409916A (zh) 三分图制作方法、三分图制作装置及存储介质
JP5962491B2 (ja) 画角調整装置、方法、およびプログラム
US20200265596A1 (en) Method for captured image positioning

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20931726

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2022548804

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 07.02.2023)

122 Ep: pct application non-entry in european phase

Ref document number: 20931726

Country of ref document: EP

Kind code of ref document: A1