WO2021031819A1 - 一种图像处理方法和电子设备 - Google Patents

一种图像处理方法和电子设备 Download PDF

Info

Publication number
WO2021031819A1
WO2021031819A1 PCT/CN2020/105710 CN2020105710W WO2021031819A1 WO 2021031819 A1 WO2021031819 A1 WO 2021031819A1 CN 2020105710 W CN2020105710 W CN 2020105710W WO 2021031819 A1 WO2021031819 A1 WO 2021031819A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
background
background image
foreground
key frame
Prior art date
Application number
PCT/CN2020/105710
Other languages
English (en)
French (fr)
Inventor
王利强
卢恒惠
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Priority to US17/636,542 priority Critical patent/US20220301180A1/en
Priority to EP20855503.7A priority patent/EP4002272A4/en
Publication of WO2021031819A1 publication Critical patent/WO2021031819A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/194—Segmentation; Edge detection involving foreground-background segmentation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4038—Image mosaicing, e.g. composing plane images from plane sub-images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/11—Region-based segmentation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10016—Video; Image sequence
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20212—Image combination
    • G06T2207/20221—Image fusion; Image merging

Definitions

  • This application relates to the field of image processing technology, and in particular to an image processing method and electronic equipment.
  • a target area such as a face
  • a mobile terminal device such as a mobile phone, a tablet, or a notebook computer to achieve the effect of real-time processing of the target area.
  • the target area (such as a human face) can be the foreground area in the entire image.
  • the area of the foreground area occupies a large proportion in the image.
  • the foreground area will usually be deformed after processing; If the foreground area is spliced back to the original image, it will cause the problem that the foreground area and the background image area (that is, the area other than the target area in the original image) does not match. Therefore, when the foreground area is processed and deformed, it is urgent to process the background image area to solve the phenomenon that the foreground does not match the background.
  • the background image area is stretched and deformed through the foreground correspondence before and after the deformation, so that the background image area is stretched and distorted to exactly match the deformed foreground edge area.
  • image affine transformation based on image affine transformation
  • the background image area is stretched and deformed when the face is beautified and thinned to solve the problem of mismatch between the foreground and the background when stitching.
  • the background image area is stretched and deformed, there will be a problem of obvious processing traces, the true feeling of the entire image will be destroyed, and there will be a problem of poor image synthesis effect after the foreground image processing.
  • the embodiments of the present application provide an image processing method and electronic device, which are used to improve the image synthesis effect after foreground image processing.
  • an embodiment of the present application provides an image processing method applied to an electronic device with a camera, wherein the method includes: displaying a first key frame image obtained from a sequence of frame video streams, and The sequence frame video stream is obtained by shooting the target scene including the first target object by the camera; acquiring the first foreground image and the first background image obtained after foreground and background separation processing is performed on the first key frame image,
  • the first foreground image includes the first target object; acquiring a second foreground image obtained by subjecting the first target object on the first foreground image to foreground image processing, and the second foreground image includes: The first target object after foreground image processing; acquiring a third background image obtained by performing background repair processing on the first background image through a second background image, the second background image being included in the second key frame image A background image, where the second key frame image is obtained by shooting the target scene through the camera before acquiring the first key frame image; acquiring a foreground background synthesis of the second foreground image and the third background image
  • the second background image can be obtained before the first key frame image is obtained.
  • the second background image can be used to perform background restoration processing on the first background image. Therefore, the obtained third background image can contain as comprehensive as possible
  • the background image information is used to repair the blank and untextured area caused when the second foreground image is generated, and the image synthesis effect after the foreground image processing is improved.
  • the method before the displaying the first key frame image obtained from the sequence frame video stream, the method further includes: displaying the second key frame image obtained from the preview video stream, so The preview video stream is obtained by previewing and shooting the target scene before the camera generates the sequence frame video stream; acquiring the second background image separated from the second key frame image.
  • the target scene is previewed before the camera generates the sequence frame video stream to obtain the preview video stream, and the second key frame image is obtained from the preview video stream.
  • the second The background image is separated from the key frame image, and the separated background image is used as the second background image.
  • the method before the obtaining the background repair process on the first background image through the second background image, the method further includes: displaying the obtained video stream from the sequence of frames A second key frame image; acquiring a fourth background image obtained after foreground and background separation processing is performed on the second key frame image; acquiring a fourth background image obtained after background restoration processing is performed on the fourth background image through the fifth background image
  • the second background image, the fifth background image is separated from the third key frame image in the preview video stream, and the preview video stream performs processing on the target scene before the camera generates the sequence frame video stream Get the preview shot.
  • the target scene is previewed before the camera generates the sequence frame video stream to obtain the preview video stream, and the third key frame image is obtained from the preview video stream.
  • the third The background image is separated from the key frame image, and the separated background image is used as the fifth background image, and the fifth background image can be used for background image restoration processing on the fourth background image.
  • the method before the displaying the first key frame image obtained from the sequence frame video stream, the method further includes: before the camera generates the sequence frame video stream, passing the The camera continuously photographs the target scene to obtain a plurality of continuous background images; and obtains the second background image obtained by accumulating and superimposing the plurality of continuous background images.
  • the target scene in a scene where the camera is fixed and the shooting environment is unchanged, the target scene can be continuously shot through the camera, and multiple continuous background images can be accumulated and superimposed to obtain a more comprehensive background image.
  • the image can be used as the second background image, and the second background image can be used for background image restoration processing on the first background image.
  • the displaying the first key frame image obtained from the sequence frame video stream includes: displaying n sequence frame images obtained from the sequence frame video stream, where n is greater than Or a positive integer equal to 2; obtain 1 original key frame image and n-1 related frame images determined from the n sequence frame images; use the pixel information of the n-1 related frame images to compare all
  • the original key frame image is subjected to definition enhancement processing, and the original key frame image after the definition enhancement processing is determined as the first key frame image.
  • the definition of key frame images can be enhanced, and one original key frame image and n-1 related frame images can be determined from n sequence frame images.
  • the original key frame image is the main image, and the related frame
  • the image refers to the n-1 frames of images that are currently used as the original key frame images except for the n frames of images.
  • the pixel information of the related frame image can be the energy value of each pixel of the related frame image.
  • the original key frame image is performed based on the pixel information of the n-1 related frame images.
  • the definition enhancement processing uses the original key frame image after the definition enhancement processing as the first key frame image, and the first key frame image with higher definition is used to further improve the effect of image synthesis.
  • the using the pixel information of the n-1 related frame images to perform sharpness enhancement processing on the original key frame image includes: acquiring the original image point of the original key frame image Energy value; obtain k related frame images before and after the original key frame image from the n-1 related frame images, and the value of k is less than or equal to (n-1) ⁇ 2;
  • the image point energy values of the k related frame images before and after the original key frame image are optimized to obtain the image point energy value of the original key frame image. value.
  • the definition of the key frame image can be enhanced.
  • the energy value of the frame image is optimized to enhance the image clarity of the original key frame.
  • the method further includes: acquiring the first key frame image obtained by performing foreground edge fusion processing on the first key frame image after the foreground and background synthesis processing; outputting the first key frame image after the foreground edge fusion processing Frame image.
  • the edge area of the foreground image is processed to reduce the unnatural problem of the image transition caused by the sudden change of the edge.
  • the edge processing can be realized through the feathering process.
  • the video is inserted into the frame to realize the smooth output of the processed result image and video.
  • the performing background restoration processing on the first background image by using the second background image includes: according to the feature point correspondence between the first background image and the second background image Relationship, acquiring the inter-frame change position relationship between the first background image and the second background image; acquiring the transformed position relationship obtained by performing perspective transformation on the second background image according to the inter-frame change position relationship A second background image; splicing the transformed second background image onto the first background image to obtain the third background image.
  • the feature point correspondence between the first background image and the second background image refers to the correspondence between the same feature point in the two background images. According to the feature point correspondence, the first background image and the first background image can be obtained.
  • inter-frame change position relationship may be the inter-frame position relationship when changing from the second background image to the first background image, for example, the inter-frame change position relationship may be the second background
  • the posture change relationship between the image and the first background image Next, perspective transformation is performed on the second background image according to the changing position relationship between frames to obtain the transformed second background image. Finally, the transformed second background image is spliced to the first background image, so that the blank and untextured area existing in the background image of the current frame can be repaired. After the second background image after perspective transformation is spliced on the first background image, it can be output
  • the third background image the third background image can be used for the composition of the foreground background image. Since the third background image has been filled with blank and untextured areas, the effect of the image composition can be improved.
  • Changing the positional relationship between frames includes: acquiring the current frame attitude parameter corresponding to the first background image; acquiring the first background according to the last frame attitude parameter corresponding to the second background image and the current frame attitude parameter The initial inter-frame positional relationship between the image and the second background image; acquiring the feature point correspondences obtained after feature extraction and feature matching of the first background image and the second background image; using the feature The point correspondence relationship optimizes the initial positional relationship between the frames to obtain the inter-frame changing positional relationship between the first background image and the second background image.
  • the current frame pose parameters corresponding to the first background image can be completed by the IMU of the electronic device, and the last frame pose parameters corresponding to the second background image can be obtained from pre-stored pose parameters, and the last frame pose parameters are used Calculate the initial positional relationship between the first background image and the second background image with the current frame pose parameters, and then perform feature extraction and feature matching on the first background image and the second background image to obtain the feature point correspondence,
  • the extracted feature may be a DAISY feature
  • the initial position relationship between frames is optimized using the corresponding relationship of the feature points to obtain the inter-frame changing position relationship between the first background image and the second background image.
  • the performing perspective transformation on the second background image according to the inter-frame change position relationship includes: using the inter-frame change position relationship to obtain the first background image and the A perspective transformation matrix between the second background images; obtaining a transformed second background image obtained by using the perspective transformation matrix to perform perspective transformation on the second background image.
  • the perspective transformation matrix between the current frame image and the previous frame image is calculated through the optimized positional relationship between the changes between frames, the previous frame of background image is perspective transformed according to the perspective transformation matrix, and the transformed images are stitched together
  • repair the blank untextured area in the background image of the current frame finally, the repaired background image and the pose parameters of the current frame are output as the background image of the previous frame and the pose parameters of the previous frame. Go to the next frame loop and output the repaired background image.
  • the performing background restoration processing on the first background image by using the second background image includes: segmenting a background object image from the second background image; when the background object image When entering the missing foreground area on the first background image, use the background object image to perform background repair processing on the first background image.
  • the person target area in the background of each key frame is segmented by the inter-frame difference method and saved as the target to be matched; Secondly, when the target tracking algorithm judges that the target to be matched enters the missing foreground area of the current key frame, the target image with higher contour similarity is searched from the target to be matched to repair the missing foreground area.
  • the embodiments of the present application also provide an electronic device, which has the function of realizing the foregoing aspects and the behavior of the electronic device in the possible implementation manners of the foregoing aspects.
  • the function can be realized by hardware, or by hardware executing corresponding software.
  • the hardware or software includes one or more modules or units corresponding to the above-mentioned functions. For example, a camera, one or more processors, memory, multiple application programs; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computers
  • the program includes instructions.
  • the electronic device executes the following steps: displaying the first key frame image acquired from the sequence frame video stream, the sequence frame video stream being The camera captures the target scene including the first target object; acquires a first foreground image and a first background image obtained after foreground and background separation processing is performed on the first key frame image, the first foreground image includes The first target object; acquiring a second foreground image obtained after foreground image processing is performed on the first target object on the first foreground image, the second foreground image includes: a first foreground image processed Target object; acquiring a third background image obtained by performing background restoration processing on the first background image through a second background image, the second background image being a background image included in a second key frame image, the second The key frame image is obtained by photographing the target scene by the camera before the first key frame image is obtained; obtaining the foreground and background synthesis obtained by performing the foreground and background synthesis processing on the second foreground image and the third background image
  • the first key frame image is obtained by photographing the
  • the electronic device when the instruction is executed by the electronic device, the electronic device is caused to specifically perform the following steps: before displaying the first key frame image obtained from the sequence frame video stream, displaying the slave Preview the second key frame image acquired in the video stream, where the preview video stream is obtained by previewing and shooting the target scene before the camera generates the sequence frame video stream; acquiring the second key frame image The second background image separated in.
  • the electronic device when the instruction is executed by the electronic device, the electronic device is caused to specifically perform the following steps: before the background restoration process is performed on the first background image through the second background image, Display the second key frame image obtained from the sequence of frame video streams; obtain a fourth background image obtained after foreground and background separation processing is performed on the second key frame image; obtain a fifth background image
  • the fourth background image is the second background image obtained after background restoration processing, the fifth background image is separated from the third key frame image in the preview video stream, and the preview video stream is in the camera It is obtained by previewing and shooting the target scene before generating the sequence frame video stream.
  • the electronic device when the instruction is executed by the electronic device, the electronic device specifically executes the following steps: before displaying the first key frame image obtained from the sequence frame video stream, Before the camera generates the sequence of frame video streams, the target scene is continuously photographed by the camera to obtain a plurality of continuous background images; and all the continuous background images are obtained after accumulating and superimposing the plurality of continuous background images. Describe the second background image.
  • the electronic device when the instruction is executed by the electronic device, the electronic device is caused to specifically perform the following steps: display n sequence frame images obtained from the sequence frame video stream, and the n Is a positive integer greater than or equal to 2; acquiring 1 original key frame image and n-1 related frame images determined from the n sequence frame images; using the pixel information of the n-1 related frame images Performing definition enhancement processing on the original key frame image, and determining the original key frame image after the definition enhancement processing as the first key frame image.
  • the electronic device when the instruction is executed by the electronic device, the electronic device specifically executes the following steps: obtain the original energy value of the image point of the original key frame image; Acquire k related frame images before and after the original key frame image from 1 related frame image, and the value of k is less than or equal to (n-1) ⁇ 2; obtain the before and after passing the original key frame image
  • the image point energy value of each k related frame images is optimized for the image point original energy value of the original key frame image to obtain the optimized energy value of the image point of the original key frame image.
  • the electronic device when the instruction is executed by the electronic device, the electronic device is caused to specifically perform the following steps: obtaining foreground edge fusion on the first key frame image after the foreground background synthesis processing The processed first key frame image after foreground edge fusion processing is obtained; outputting the first key frame image after foreground edge fusion processing.
  • the electronic device when the instruction is executed by the electronic device, the electronic device is caused to specifically perform the following steps: according to the characteristic points between the first background image and the second background image Correspondence, acquiring the inter-frame change position relationship between the first background image and the second background image; acquiring the transformed position obtained by performing perspective transformation on the second background image according to the inter-frame change position relationship The second background image; splicing the transformed second background image to the first background image to obtain the third background image.
  • the electronic device when the instruction is executed by the electronic device, the electronic device is caused to specifically perform the following steps: acquiring the current frame attitude parameter corresponding to the first background image; and according to the second The previous frame posture parameter and the current frame posture parameter corresponding to the background image are acquired, the initial positional relationship between the frames between the first background image and the second background image is acquired;
  • the second background image is the feature point corresponding relationship obtained after feature extraction and feature matching; using the feature point corresponding relationship to optimize the initial position relationship between the frames to obtain the first background image and the second background
  • the inter-frame change position relationship between images when the instruction is executed by the electronic device, the electronic device is caused to specifically perform the following steps: acquiring the current frame attitude parameter corresponding to the first background image; and according to the second The previous frame posture parameter and the current frame posture parameter corresponding to the background image are acquired, the initial positional relationship between the frames between the first background image and the second background image is acquired;
  • the second background image is the feature point corresponding relationship obtained after feature extraction and feature matching; using the feature point corresponding
  • the electronic device when the instruction is executed by the electronic device, the electronic device is caused to specifically perform the following steps: acquiring the first background image and the first background image and the first background image using the inter-frame change position relationship A perspective transformation matrix between two background images; obtaining a transformed second background image obtained by using the perspective transformation matrix to perform perspective transformation on the second background image.
  • the electronic device when the instruction is executed by the electronic device, the electronic device specifically executes the following steps: segmenting a background object image from the second background image; when the background object When the image enters the missing foreground area on the first background image, the background object image is used to perform background repair processing on the first background image.
  • the component modules of the electronic device can also perform the steps described in the foregoing first aspect and various possible implementations. For details, see the foregoing description of the first aspect and various possible implementations. Description.
  • an embodiment of the present application also provides an electronic device, including: a camera; one or more processors; a memory; multiple application programs; and one or more computer programs.
  • one or more computer programs are stored in the memory, and the one or more computer programs include instructions.
  • the electronic device is caused to execute the image processing method in any possible implementation of any one of the foregoing aspects.
  • an embodiment of the present application also provides an electronic device, including one or more processors and one or more memories.
  • the one or more memories are coupled with one or more processors, and the one or more memories are used to store computer program codes.
  • the computer program codes include computer instructions.
  • the electronic device Execute the image processing method in any possible implementation of any one of the above aspects.
  • the embodiments of the present application provide a computer-readable storage medium having instructions stored in the computer-readable storage medium, which when run on a computer, cause the computer to execute the method described in the first aspect.
  • embodiments of the present application provide a computer program product containing instructions, which when run on a computer, cause the computer to execute the method described in the first aspect.
  • the present application provides a chip system including a processor for supporting electronic devices to implement the functions involved in the above aspects, for example, sending or processing data and/or information involved in the above methods .
  • the chip system further includes a memory, and the memory is used to store necessary program instructions and data of the electronic device.
  • the chip system may be composed of chips, or may include chips and other discrete devices.
  • FIG. 1 is a schematic diagram of the composition structure of an electronic device provided by an embodiment of the application.
  • Figure 2a is a schematic diagram of an electronic device displaying an image provided by an embodiment of the application
  • Figure 2b is a schematic diagram of a video scene provided by an embodiment of the application.
  • Figure 2c is a schematic diagram of another video scene provided by an embodiment of the application.
  • FIG. 3 is a flowchart of an image processing method provided by an embodiment of the application.
  • Fig. 4 is a flowchart of an image processing method provided by an embodiment of the application.
  • FIG. 5 is a schematic diagram of key frame generation provided by an embodiment of the application.
  • FIG. 6 is a schematic diagram of foreground and background separation processing provided by an embodiment of this application.
  • FIG. 7 is a schematic diagram of background image stitching processing provided by an embodiment of the application.
  • FIG. 8 is a schematic diagram of a background image stitching processing flow provided by an embodiment of the application.
  • FIG. 9 is a flowchart of camera calibration provided by an embodiment of the application.
  • FIG. 10a is a schematic diagram of a background image of the n-1th frame provided by an embodiment of the application.
  • 10b is a schematic diagram of a background image of the nth frame provided by an embodiment of the application.
  • FIG. 10c is a schematic diagram of separating a background object in a background image of the nth frame according to an embodiment of the application.
  • 10d is a schematic diagram of a background image of the n+1th frame provided by an embodiment of the application.
  • FIG. 10e is a schematic diagram of a background image of the n+1th frame after background restoration provided by an embodiment of the application.
  • FIG. 11 is a flowchart of an image processing method provided by an embodiment of this application.
  • FIG. 12 is a flowchart of an image processing method provided by an embodiment of this application.
  • FIG. 13 is a schematic diagram of the composition structure of an electronic device provided by an embodiment of the application.
  • FIG. 14 is a schematic diagram of the composition structure of an electronic device provided by an embodiment of the application.
  • first and second are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, the features defined with “first” and “second” may explicitly or implicitly include one or more of these features. In the description of this embodiment, unless otherwise specified, “plurality” means two or more.
  • the embodiment of the application provides an image processing method, which can be applied to an electronic device, and a second background image can be obtained before the first key frame image is obtained.
  • the second background image is a background image included in the second key frame image.
  • the second key frame image is obtained by shooting the target scene with the camera before the first key frame image is obtained.
  • the first foreground image and the first background image can be obtained through foreground and background separation processing, and the second background image is used
  • the first background image can be subjected to background restoration processing to obtain a third background image. Therefore, the obtained third background image can contain as comprehensive background image information as possible.
  • the third background image and the second foreground image are subjected to foreground background synthesis processing.
  • the second foreground image is obtained after foreground image processing is performed on the first foreground image, so as to repair the blank and untextured area caused when the second foreground image is generated, and improve the image synthesis effect after the foreground image processing.
  • a target object may include multiple objects of the target object.
  • a complete image can be divided into two parts: a foreground image and a background image.
  • the area where the target object is located refers to the foreground image where the object belonging to the target object is located.
  • the specific one or more objects refers to one or more objects specified by a user or one or more objects preset by an electronic device.
  • a specific one or more objects refers to objects included in one or more object types specified by the user, or objects included in one or more object types preset by an electronic device, and the position and size of the object are determined by Or objects included in multiple object types.
  • Image segmentation can also be called semantic segmentation, which refers to the technology and process of dividing an image into a number of specific areas with special properties and proposing objects of interest, such as segmenting a complete image into a foreground image and a background image.
  • semantic segmentation refers to the technology and process of dividing an image into a number of specific areas with special properties and proposing objects of interest, such as segmenting a complete image into a foreground image and a background image.
  • image segmentation methods For example, the face area is detected by face recognition and the body area is estimated, the foreground image area is segmented from the complete image by the graph cut method, and the background image area is separated, and then the frame is used
  • the inter-differential algorithm or the optical flow tracking algorithm tracks and segment the foreground image area, while acquiring the background image area.
  • the embodiments of this application provide an image processing method, which can be applied to mobile phones, tablet computers, wearable devices, vehicle-mounted devices, augmented reality (AR)/virtual reality (VR) devices, notebook computers, super Mobile personal computers (ultra-mobile personal computers, UMPC), netbooks, personal digital assistants (personal digital assistants, PDAs), smart terminals, video conferencing terminals, image capture terminals and other electronic devices, the embodiments of this application have specific details about electronic devices. There are no restrictions on the type.
  • the operating system may be Android, iOS, Windows Phone, BlackBerry OS and other systems, which are not specifically limited in the embodiment of the present application.
  • FIG. 1 shows a block diagram of a part of the structure of the mobile phone 100 related to an embodiment of the present application.
  • the mobile phone 100 includes an RF (Radio Frequency) circuit 110, a memory 120, other input devices 130, a display screen 140, a sensor 150, an audio circuit 160, an I/O subsystem 170, a processor 180, and a camera. 190 and other parts.
  • RF Radio Frequency
  • FIG. 1 does not constitute a limitation on the mobile phone, and may include more or less components than shown in the figure, or combine certain components, or split certain components, or Different component arrangements.
  • the display screen 140 belongs to a user interface (UI, User Interface), and the mobile phone 100 may include a user interface that is less than that shown or less.
  • UI User Interface
  • the RF circuit 110 can be used for receiving and sending signals during information transmission or communication, in particular, after receiving the downlink information of the base station, it is processed by the processor 180; in addition, the designed uplink data is sent to the base station.
  • the RF circuit includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a Low Noise Amplifier (LNA), a duplexer, and the like.
  • the RF circuit 110 can also communicate with the network and other devices through wireless communication.
  • the wireless communication can use any communication standard or protocol, including but not limited to Global System of Mobile Communication (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (Code Division Multiple Access). Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), Email, Short Messaging Service (SMS), etc.
  • GSM Global System of Mobile Communication
  • GPRS General Packet Radio Service
  • CDMA Code Division Multiple Access
  • WCDMA Wideband Code Division
  • the memory 120 may be used to store software programs and modules.
  • the processor 180 executes various functional applications and data processing of the mobile phone 100 by running the software programs and modules stored in the memory 120.
  • the memory 120 may mainly include a storage program area and a storage data area.
  • the storage program area may store an operating system, an application program required by at least one function (such as a sound playback function, an image playback function, etc.), etc.;
  • the storage data area may store Data (such as audio data, phone book, etc.) created according to the use of the mobile phone 100, etc.
  • the memory 120 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other volatile solid-state storage devices.
  • the other input device 130 may be used to receive inputted numeric or character information, and generate key signal input related to user settings and function control of the mobile phone 100.
  • other input devices 130 may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, switch buttons, etc.), trackballs, mice, joysticks, optical mice (optical mice are touch sensitive that do not display visual output). A surface, or an extension of a touch-sensitive surface formed by a touch screen).
  • the other input device 130 is connected to the other input device controller 171 of the I/O subsystem 170, and performs signal interaction with the processor 180 under the control of the other device input controller 171.
  • the display screen 140 can be used to display information input by the user or information provided to the user and various menus of the mobile phone 100, and can also accept user input.
  • the specific display screen 140 may include a display panel 141 and a touch panel 142.
  • the display panel 141 can be configured in the form of LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode, organic light-emitting diode), etc.
  • the touch panel 142 also called a touch screen, a touch-sensitive screen, etc., can collect user contact or non-contact operations on or near it (for example, the user uses any suitable objects or accessories such as fingers, stylus, etc., on the touch panel 142 Or operations near the touch panel 142 may also include somatosensory operations; the operations include single-point control operations, multi-point control operations and other types of operations.), and drive the corresponding connection device according to a preset program.
  • the touch panel 142 may include two parts: a touch detection device and a touch controller.
  • the touch detection device detects the user's touch position and posture, and detects the signal caused by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device and converts it into a processor capable of The processed information is then sent to the processor 180, and can receive and execute commands from the processor 180.
  • the touch panel 142 can be realized by various types such as resistive, capacitive, infrared, and surface acoustic wave, or any technology developed in the future can be adopted to realize the touch panel 142.
  • the touch panel 142 can cover the display panel 141, and the user can display content on the display panel 141 according to the content displayed on the display panel 141 (the display content includes, but is not limited to, soft keyboard, virtual mouse, virtual keys, icons, etc.)
  • the touch panel 142 detects a touch operation on or near it, and transmits it to the processor 180 through the I/O subsystem 170 to determine the type of touch application to determine the user Then the processor 180 provides corresponding visual output on the display panel 141 through the I/O subsystem 170 according to the user input on the display panel according to the type of touch application.
  • the touch panel 142 and the display panel 141 are used as two independent components to implement the input and input functions of the mobile phone 100, but in some embodiments, the touch panel 142 and the display panel 141 can be integrated And realize the input and output functions of the mobile phone 100.
  • the mobile phone 100 may also include at least one sensor 150, such as a light sensor, a motion sensor, and other sensors.
  • the light sensor may include an ambient light sensor and a proximity sensor.
  • the ambient light sensor can adjust the brightness of the display panel 141 according to the brightness of the ambient light.
  • the proximity sensor can close the display panel 141 and the display panel 141 when the mobile phone 100 is moved to the ear. / Or backlight.
  • the accelerometer sensor can detect the magnitude of acceleration in various directions (usually three-axis), and can detect the magnitude and direction of gravity when stationary, and can be used to identify mobile phone posture applications (such as horizontal and vertical screen switching, related Games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, percussion), etc.; as for the other sensors such as gyroscope, barometer, hygrometer, thermometer, infrared sensor, etc. that can be configured in the mobile phone 100, we will not Repeat it again.
  • mobile phone posture applications such as horizontal and vertical screen switching, related Games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, percussion), etc.
  • vibration recognition related functions such as pedometer, percussion
  • the other sensors such as gyroscope, barometer, hygrometer, thermometer, infrared sensor, etc. that can be configured in the mobile phone 100, we will not Repeat it again.
  • the audio circuit 160, the speaker 161, and the microphone 162 can provide an audio interface between the user and the mobile phone 100.
  • the audio circuit 160 can transmit the converted signal of the received audio data to the speaker 161, which is converted into a sound signal for output by the speaker 161; on the other hand, the microphone 162 converts the collected sound signal into a signal, which is received by the audio circuit 160
  • the audio data is converted into audio data, and then the audio data is output to the RF circuit 110 to be sent to, for example, another mobile phone, or the audio data is output to the memory 120 for further processing.
  • the I/O subsystem 170 is used to control input and output external devices, and may include other device input controller 171, sensor controller 172, and display controller 173.
  • one or more other input control device controllers 171 receive signals from other input devices 130 and/or send signals to other input devices 130, and other input devices 130 may include physical buttons (press buttons, rocker buttons, etc.) , Dial, slide switch, joystick, click wheel, optical mouse (optical mouse is a touch-sensitive surface that does not display visual output, or an extension of the touch-sensitive surface formed by a touch screen).
  • the other input control device controller 171 may be connected to any one or more of the above-mentioned devices.
  • the display controller 173 in the I/O subsystem 170 receives signals from the display screen 140 and/or sends signals to the display screen 140. After the display screen 140 detects the user input, the display controller 173 converts the detected user input into an interaction with the user interface object displayed on the display screen 140, that is, human-computer interaction is realized.
  • the sensor controller 172 may receive signals from one or more sensors 150 and/or send signals to one or more sensors 150.
  • the processor 180 is the control center of the mobile phone 100. It uses various interfaces and lines to connect various parts of the entire mobile phone. By running or executing software programs and/or modules stored in the memory 120, and calling data stored in the memory 120, Perform various functions of the mobile phone 100 and process data, thereby monitoring the mobile phone as a whole.
  • the processor 180 may include one or more processing units; preferably, the processor 180 may integrate an application processor and a modem processor, where the application processor mainly processes the operating system, user interface, application programs, etc. , The modem processor mainly deals with wireless communication. It can be understood that the foregoing modem processor may not be integrated into the processor 180.
  • the mobile phone 100 may also include a camera 190.
  • the mobile phone 100 may include one or more cameras.
  • the user can operate the camera to shoot the user's face to generate a sequence of frame video streams.
  • the electronic device can preview the shooting through the camera To generate a preview video stream as the image to be used for background processing.
  • the user when shooting with the camera of an electronic device, consider that the user opens the camera to preview first, and adjusts the camera or target's angle or posture according to the preview shooting effect.
  • the user holds the mobile phone to shoot the preview, it is constantly adjusted
  • the position and angle of the mobile phone can also change the posture and the angle of the face to achieve better shooting results.
  • the user will also adjust his position to achieve better shooting results, so you can use the user to advance in the preview stage Turn on the camera and collect background information in advance to obtain as comprehensive background information as possible.
  • the mobile phone 100 may also include a power source (such as a battery) for supplying power to various components.
  • a power source such as a battery
  • the power source may be logically connected to the processor 180 through a power management system, so that the power management system can manage charging, discharging, and Power consumption and other functions.
  • the mobile phone 100 may also include a Bluetooth module, etc., which will not be repeated here.
  • Figure 2a shows a graphical user interface (GUI) of the mobile phone, and the GUI is the desktop of the mobile phone.
  • GUI graphical user interface
  • the camera application can be started, and the complete image collected by the camera is displayed on the shooting interface.
  • the preview image can be displayed in the shooting interface in real time.
  • the size of the shooting interface can be different in the camera mode and the video mode (ie, the video shooting mode).
  • the mobile phone performs a photo or video recording operation.
  • the complete image shown in Figure 2a can be segmented into a foreground image and a background image.
  • an inter-frame difference algorithm or an optical flow tracking algorithm can be used to track and segment the foreground image while acquiring the background image.
  • the face and body parts shown in Figure 2a constitute the foreground image, and the other image areas in the entire image except the foreground image constitute the background image.
  • the foreground image and background image shown in Figure 2a it is only an example.
  • the embodiments of the present application may also use other image content, and the color of the image is not limited.
  • the background image of the current frame is combined with the pre-collected background image to repair or stitch, so that after the foreground image is stretched, the background image processed by the embodiment of the present application can better match the foreground Image combination, thereby eliminating the problem of poor image synthesis effect after foreground image processing.
  • FIG. 2b is a schematic diagram of a video scene provided by an embodiment of the application.
  • the electronic device may be a notebook, and the keyboard of the notebook is provided with a camera, and the camera and the display screen are not on the same plane.
  • FIG. 2c is a schematic diagram of another video scene provided by an embodiment of the application.
  • the electronic device may be a desktop computer.
  • the desktop computer is provided with a camera under the display screen, and the camera and the display screen are not on the same plane. on.
  • the electronic device can collect key frame images through its own camera.
  • the camera and the display screen of the electronic device there is a distance between the camera and the display screen it includes, and when the orientation of the camera and the display screen it includes are different, when the user uses the electronic device and other displays
  • the user’s video chat or video conference if the user’s face is not facing the camera, but facing other places, such as facing the display screen, there will be faces in the image sequence collected by the electronic device through the camera.
  • An image whose orientation is not parallel to the vertical line of the display screen of the electronic device. In this case, it is necessary to perform face calibration on the foreground image through the embodiment of the present application.
  • FIG. 3 a flowchart of an image processing method provided by an embodiment of this application.
  • the arrows show the multiple processing procedures of the video stream from top to bottom, namely: turn on the camera, generate key frame images, separate foreground and background, process the foreground and background separately, and combine the foreground and background.
  • the user can collect background information in advance during the preview stage to compensate
  • Existing image areas without texture, for example, performing a face-lifting beautification operation on the face of the target object on the foreground image will result in a blank untextured area when the thinned face part and the original background image are synthesized.
  • the sequence frame images in the preview stage can be captured (for example, a total of n ⁇ m frames are captured), and 1 key frame image can be generated for every n frames of the sequence image, and then it is detected and generated from the n ⁇ m frame images m key frame images; perform foreground and background separation on the key frame images to obtain m foreground images and corresponding m background images.
  • m background images are processed in the following way for background restoration, and the images are spliced according to the order of the images.
  • the first frame of background image and the second frame of background image are spliced, and the spliced image is used as the second frame of background image, and then the The second frame of background image and the third frame of background image are spliced, the spliced image is used as the third frame of image, and so on, each time the two frames of image are spliced, so that the background image can be repaired.
  • the deformation of the foreground image results in the blank and untextured area when the foreground and background are synthesized.
  • the background image of the current frame and the pre-collected background image can be spliced or repaired.
  • the spliced or repaired background image includes more comprehensive background information, so based on the embodiments of the present application
  • the spliced and repaired background image is combined with the foreground image processed by the foreground image, it can fill in the existing blank non-textured area.
  • the background will not be distorted, and the image effect of the combined foreground image and background image is real natural.
  • the application scenario of the embodiment of this application is to use a mobile phone, tablet, laptop or other electronic device with shooting function to photograph and process a target object, and the target area is obviously deformed during processing, which makes it difficult to seamlessly match the original background image after the deformation. Stitched scenes.
  • the specific application scenarios of the embodiments of the present application include, but are not limited to, scenarios in which the area of the foreground area such as beautification and camera calibration occupies a relatively large area in the entire image and the foreground area is significantly deformed during processing.
  • each frame of image data from the camera as a sequence frame video stream
  • the sequence frame video stream refers to a video data stream composed of multiple sequence frame images
  • 1 key frame image is output every n frames of images to form a key frame image video stream.
  • the key frame image is obtained from the sequence frame video stream, and the subsequent processing of the key frame image according to the embodiment of the present application can reduce the amount of data calculation, and can also eliminate the repetition of several consecutive frames of images and avoid repeated processing.
  • the key frame image generation stage in order to be able to generate a clearer key frame image, first determine an original key frame image from n sequence frame images, and then use the original key frame image as the center , Obtain n-1 related frame images from n sequence frame images, where related frame refers to the n sequence frame images excluding other n-1 frame images that are currently used as key frame images, through the energy value of the relevant frame image Image fusion is performed on the original key frame images to generate clearer key frame images, which can solve the problem of generating clearer key frame images.
  • the foreground and background are separated into two video streams.
  • Receive the key frame video stream perform foreground segmentation on each image of the key frame, and separate the segmented foreground image from the background image to generate two video streams, namely: the foreground image video stream and the background image video stream.
  • the foreground and background separation processing refers to the foreground segmentation of the foreground part and the background part in the key frame image, so that the segmented foreground and background images are separated into two separate images, and the foreground and background separation process Two images can be generated: foreground image and background image.
  • the background restoration processing can be implemented in multiple ways.
  • the background image of the current frame and the background image of the previous frame are spliced.
  • the background image of the previous frame has two implementation modes. One is that the background image of the previous frame is a background image corresponding to the key frame image of the previous frame of the current frame in the sequence frame video stream. The other is that the background image of the previous frame is the background image corresponding to the preview image in the preview video stream collected by the camera in the preview stage.
  • perspective transformation can be performed on the background image of the previous frame, so that the background image of the previous frame and the background image of the current frame are texture aligned. Fill in all the background image information as much as possible when there is no foreground image. For example, when taking a portrait of the user in an ideal situation, even if the portrait is cut out from the key frame image, the image is still complete through background stitching and patching. Background image.
  • the perspective transformation matrix needs to be calculated first.
  • the posture parameters of each frame of background image can be This is done through an inertial measurement unit (IMU).
  • IMU inertial measurement unit
  • the other is not to use posture parameters, but to use the alignment of the feature points of the background images of the two frames before and after to estimate the positional relationship of changes between frames, and calculate the perspective transformation matrix through the positional relationship of changes between frames.
  • the feature of the background image may be DAISY feature. It is not limited that other feature point collection methods may also be used in the embodiments of the present application, which are not limited here.
  • the multiple key frame images are processed in a cyclic relationship, and the background image of the current frame The data should be saved to be used as the "background image of the previous frame" in the next frame.
  • the background restoration process can be implemented in other ways, for example, background image restoration is performed on the background image of the current frame, and background restoration is performed for different situations.
  • the embodiments of this application may include at least the following three types of background image restoration Way: One is a scene where the camera is fixed and the shooting environment is unchanged. In this case, the background image area can be accumulated and superimposed to obtain a more comprehensive background image. The other is a scene where the camera is moving and the shooting environment remains unchanged. In this case, the background image stitching processing method described above can be used.
  • the position change relationship between the previous frame and the current frame can be directly estimated through feature matching.
  • the third type is a scene where the camera is fixed and the shooting environment changes locally. For example, if there are people walking in the shooting background, the target area of the person in the background of each key frame is divided by the inter-frame difference method and saved as the target to be matched, and then the target is passed When the tracking algorithm judges that the target to be matched enters the missing foreground area of the current key frame, it searches for the target image with higher contour similarity from the target to be matched to repair the missing foreground area.
  • the background image of the previous frame is spliced with the background image of the current frame to expand the background image area of the current frame, or repair the blank untextured area left by the foreground segmentation, so as to ensure that the processed foreground image can be restored to the background
  • the background image is synthesized.
  • the foreground image and background image belonging to the same key frame are synthesized, the effective area texture of the foreground image is preserved, and the edge area where the foreground image area and the background image area are merged are processed, and the synthesized result is taken as the video Stream output.
  • the background image information has been filled as much as possible, that is, the background image information obtained from other angles can be filled in by the processing in step 4 A blank, untextured area formed by changes in the foreground image.
  • FIG. 4 it is a flowchart of an image processing method provided by an embodiment of this application.
  • the processing flow in the beauty application mainly includes the following processes:
  • Step S01 Turn on the camera to obtain the sequence frame images in the preview stage, and output the preview video stream. Then a key frame image is generated, for example, the generated multiple key frame images are formed into a key frame video stream.
  • Key frame image detection methods include, but are not limited to: lens-based methods, frame averaging methods, histogram averaging methods, motion-based analysis methods, clustering-based methods, etc.
  • the energy value of the original key frame image is optimized to enhance the image clarity of the original key frame.
  • the energy formula is as follows:
  • pi refers to the i-th image point p in the image
  • E(pi) refers to the energy value at the image point pi
  • I refers to the image, which generally refers to any image in n frames of images.
  • E(p) refers to the energy value of any image point in general
  • Ik(qi) refers to the pixel value of the image point qi of the k-th frame image in the n frame image
  • I(qi) refers to the generated key frame image at the image point
  • B(p) refers to the establishment of a 5 ⁇ 5 image block with any image point p in the image I as the center.
  • FIG. 5 a schematic diagram of key frame generation provided by an embodiment of this application.
  • n there are a total of n ⁇ m frames in a video stream of sequence frames, so that m key frame images can be generated.
  • 2k+1 in Figure 5 is equal to n, that is, 1 key frame image is generated for every 2k+1 frame image, so the video stream There are m 2k+1 frame images.
  • the generated m key frame images are output in the form of a video stream.
  • Step S02 Perform foreground and background separation.
  • FIG. 6 a schematic diagram of the foreground and background separation processing provided by this embodiment of the application.
  • the foreground image and the background image shown in FIG. 6 are only an example case. In actual applications, the embodiment of the present application may also use other image content, and the color of the image is not limited.
  • Step S03 foreground beautification processing.
  • Read each frame of the foreground video stream use the beauty algorithm to process the foreground image, such as 2D beauty algorithm, 3D geometric beautification algorithm, and output the processed foreground video stream.
  • the beauty algorithm such as 2D beauty algorithm, 3D geometric beautification algorithm
  • Step S04 background image stitching.
  • the specific splicing method is as follows: perform perspective transformation on the previous key frame background image to align it with the current key frame background image, and then The current key frame background image is spliced to expand the area of the current background image or repair the untextured area left by the foreground image segmentation. According to the different angles between the images, the result of image splicing and repair can be to expand the background image area or repair the foreground image. The blank area left after the extension, or the blank area left after subtracting the background image area and repairing the foreground image.
  • FIG. 7 it is a schematic diagram of background image stitching processing provided by an embodiment of this application.
  • the background image of the previous frame is to the left
  • the background image of the current frame is to the right
  • the background image of the previous frame is spliced into the current background image.
  • splicing The subsequent background information of the background image of the current frame will expand to the left due to the supplement of the background image of the previous frame.
  • the background image of the previous frame After performing perspective transformation on the background image of the previous frame, it merges with the background image of the current frame. A blank area is left after the current image portrait is subtracted. The background image of the previous frame is taken from another angle, and the background image information of the blank area can be captured. The background image of the previous frame is perspective transformed and repaired to the current blank area. That is, the background repair of the area behind the foreground is formed.
  • the processed background results are output in two ways: one is combined with the current pose parameters and looped as the pose parameters of the previous frame of background image and the previous frame of image, for the stitching of the next frame of background image; the other is for stitching
  • the background output is used in the next step.
  • the perspective transformation matrix can be initialized according to the current pose parameters obtained by the IMU, and then obtained through DAISY feature matching optimization of the two frames of images.
  • a schematic diagram of the background image stitching processing flow provided by this embodiment of the application mainly includes the following flow:
  • S048 Perform background stitching and repair using the positional relationship between frames.
  • the posture parameters of the current shooting are obtained through the IMU, and the posture parameters of the previous frame are calculated to obtain the initial positional relationship between the current frame and the previous frame; secondly, the background of the current frame output in step S03
  • the image and the background image of the previous frame are DAISY feature extraction and matching, and the initial position relationship between the current frame and the previous frame is optimized through the corresponding relationship between the feature points, and the position relationship between the frames is obtained; then, through the optimized frame Change the position relationship to calculate the perspective transformation matrix between the current frame image and the previous frame image, perform perspective transformation on the background image of the previous frame according to the perspective transformation matrix, stitch the transformed image to the background image of the current frame, and repair the current frame The blank and untextured area in the background image; finally, the repaired background image and the pose parameters of the current frame are output as the background image of the previous frame and the pose parameters of the previous frame to participate in the next frame cycle, and at the same time, the repair The subsequent background image is output as the input data of step S05.
  • the pose parameters of the current frame correspond to the current frame.
  • the pose parameters of the current frame become the pose parameters of the previous frame, and the next frame becomes the current frame.
  • the pose parameters of the next frame are changed to the pose parameters of the current frame.
  • Step S05 Foreground and background synthesis.
  • the processed foreground image and the spliced background image are synthesized, the effective area of the foreground image is reserved, and the invalid area is supplemented by the background, and output as the synthesized key frame video stream.
  • background image mosaic repair is provided. If the angle is large enough and the background image information is relatively complete, a full background image will be formed, and all the blank parts of the current image deducted from the foreground area will be filled, so the current scene is deformed and Moving or even disappearing, the background image is complete, so you can fill the invalid area (ie, the blank untextured area) left by the foreground change with the image of the complete background.
  • Step S06 Edge fusion processing.
  • the edge processing can be realized by feathering.
  • Step S07 Output display.
  • the video is inserted into the frame to realize the smooth output of the processed result image and video.
  • Sequence frames are generated between key frames through frame interpolation, which makes the video smoother and smoother, and the frame interpolation algorithm is not limited.
  • the user obtains more background image information in the preview stage before the beauty selfie, and solves the problem of repairing the texture-free gap area caused by processing such as face thinning, which can be conveniently applied to mobile terminal devices such as mobile phones and tablets. Protection of background information while beautifying.
  • the technology of the embodiments of the present application better obtains more comprehensive background information through the method of multi-frame learning, and effectively supplements the gaps in the splicing of the front background without background. Distortion, the effect is real and natural.
  • FIG. 9 it is a flow chart of camera calibration provided by this embodiment of the application.
  • the system flow of the camera calibration application in the embodiment of this application is described step by step as follows:
  • Step S11 reference may be made to step S01 in the foregoing embodiment, which will not be repeated here.
  • Step S12 reference may be made to step S02 in the foregoing embodiment, which will not be repeated here.
  • Step S13 Foreground face calibration.
  • the main processing method of foreground face calibration three-dimensional reconstruction of the foreground area, and angle adjustment of the three-dimensional face, and then map to generate a 2D image.
  • Step S14 background image restoration, performing background restoration separately for different situations, for example:
  • One is a scene where the camera is fixed and the shooting environment is unchanged.
  • the background image area can be accumulated and superimposed to obtain a more comprehensive background image.
  • the background image stitching processing method in step S04 can be used, in which the posture change between the background image of the previous frame and the background image of the current frame is estimated.
  • the position change relationship between the previous frame and the current frame is directly estimated through DAISY feature matching.
  • the third is a scene where the camera is fixed and the shooting environment changes locally. For example, if there are people walking in the shooting background, the target area of the person in the background of each key frame is divided by the inter-frame difference method and saved as the target to be matched; secondly, the target is passed When the tracking algorithm judges that the target to be matched enters the missing foreground area of the current key frame, it searches for the target image with higher contour similarity from the target to be matched to repair the missing foreground area.
  • FIG. 10a it is a schematic diagram of the background image of the n-1th frame provided by this embodiment of the application
  • FIG. 10b is a schematic diagram of the background image of the nth frame provided by this embodiment of the application, and it passes through the nth frame.
  • the comparison between the background image of and the background image of the n-1th frame shows that the background object shown in FIG. 10c can be separated from the background image of the nth frame.
  • Fig. 10d shows a schematic diagram of the background image of the n+1th frame provided by this embodiment of the application.
  • the background object shown in Fig. 10c and the background image of the n+1th frame are used for background restoration, and the foreground image is in the nth frame.
  • FIG. 10e is a schematic diagram of the background image of the n+1th frame after background restoration provided in an embodiment of the application, as shown in FIG. 10c
  • the background object can be synthesized with the background image of the n+1th frame shown in FIG. 10d, so as to realize the background repair when the background image is a dynamic image.
  • Figure 10b there is a person walking, and the person does not enter the background in the n-1th frame shown in Figure 10a, and the person enters the background in the nth frame. You can subtract the n-1th frame image from the nth frame image.
  • the background person image shown in Figure 10c is obtained.
  • the person walks to the foreground image area at the n+1th frame.
  • the part of the walking background person will be deducted from the foreground person (ie the white in Figure 10d).
  • Area) occlusion In order to supplement the occluded background character area, the background character area occluded by the foreground in the background image is complemented by the previously acquired background character image, so as to achieve the effect of repairing the background character when there are people walking in the background.
  • the execution device of step S14 may be a laptop computer.
  • the background processing process of step S14 on the basis of the background processing technology of the aforementioned step S04, multiple implementation scenarios of background restoration are added. For example, it can be processed for a locally changed background, which is different from the background in the aforementioned step S04. Splicing method.
  • Step S15 reference may be made to step S05 in the foregoing embodiment, which will not be repeated here.
  • Step S16 reference may be made to step S06 in the foregoing embodiment, which will not be repeated here.
  • Step S17 reference may be made to step S07 in the foregoing embodiment, which will not be repeated here.
  • the embodiments of the present application can be extended to other scenes that need to be deformed or moved after the foreground segmentation and need to protect the background information.
  • the background image can be obtained in advance
  • the scenes include but are not limited to : Use the camera to capture the human body to achieve game interaction. While processing and deforming the segmented human body image, it also repairs the texture of the blank area left by the human body change to make the game more realistic; augmented reality technology is used to capture the target object in the scene (such as sofas, tables, balloons, etc.) perform virtual movement or deformation to repair the untextured area left after the object changes, making the interaction effect more realistic.
  • an image processing method which can be implemented in an electronic device (such as a mobile phone, a tablet computer, etc.) with a camera as shown in FIG. 1.
  • an image processing method is applied to an electronic device with a camera. The method may include the following steps:
  • Display the first key frame image acquired from the sequence frame video stream, and the sequence frame video stream is obtained by shooting a target scene including the first target object by a camera.
  • the camera captures the target scene including the first target object, thereby generating a sequence frame video stream, where the sequence frame video stream refers to a video data stream composed of multiple sequence frame images captured by the camera ,
  • the first target object may be the avatar of the user who controls the electronic device, and the target scene may refer to the shooting environment scene including the user’s avatar.
  • the target scene may be a shooting background. In the target scene except for the user’s avatar, In addition, there are background images.
  • the target scene may be a meeting scene shot by the user during a video conference.
  • the first key frame image may be a certain key frame image in the sequence frame video stream.
  • the first key frame image may be step S01 shown in FIG. 4 or step S11 shown in FIG. The current key frame image of.
  • step 1101 displays the first key frame image obtained from the sequence frame video stream
  • the method provided in the embodiment of the present application further includes the following steps:
  • the preview video stream is obtained by previewing the target scene before the camera generates the sequence frame video stream;
  • the first key frame image may be the first image extracted from the sequence frame video stream.
  • the target scene is previewed and shot to obtain the preview video stream, and the second image is obtained from the preview video stream.
  • the key frame image is separated from the second key frame image, and the separated background image is used as the second background image.
  • the second background image can be used to compare the first background For the background image restoration processing of the image, see the description in the subsequent embodiment for details.
  • the user when shooting with the camera of an electronic device, consider that the user opens the camera to preview first, and adjusts the camera or target's angle or posture according to the preview shooting effect. Therefore, the user can be used to preview the background image in advance. Collect to make up for the blank and untextured areas left when the foreground image is processed and the shape changes.
  • step 1101 displays the first key frame image obtained from the sequence frame video stream
  • the method provided in the embodiments of the present application further includes the following steps:
  • the target scene is continuously captured by the camera to obtain multiple continuous background images
  • the second background image can be used for background image restoration processing on the first background image, see the description in the subsequent embodiments for details.
  • step 1101 displays the first key frame image obtained from the sequence frame video stream, including:
  • n is a positive integer greater than or equal to 2;
  • the embodiment of the present application can enhance the definition of the key frame image, and determine 1 original key frame image and n-1 related frame images from n sequence frame images.
  • the original key frame image is the main image, and the related The frame image refers to the n-1 frames of images that are currently used as the original key frame images except for the n frames of images.
  • the pixel information of the related frame image can be the energy value of each pixel of the related frame image.
  • the original key frame image is performed based on the pixel information of the n-1 related frame images.
  • the definition enhancement processing uses the original key frame image after the definition enhancement processing as the first key frame image, and the first key frame image with higher definition is used to further improve the effect of image synthesis.
  • using pixel information of n-1 related frame images to perform sharpness enhancement processing on the original key frame image includes:
  • the energy value of the key frame image is optimized to enhance the image clarity of the original key frame.
  • a complete key frame image can be It is divided into: a foreground image and a background image.
  • the segmented first foreground image may include the first target object.
  • the first target object may include the user's face and body part.
  • the foreground and background separation processing refers to the foreground segmentation of the foreground part and the background part in the key frame image, so that the segmented foreground and background images are separated into two separate images, and the foreground and background separation process Two images can be generated: foreground image and background image.
  • step 1103 can be executed first, and then step 1104; or step 1104 can be executed first, and then step 1103; or step 1103 and step 1103 can be executed simultaneously 1104, there is no limit here.
  • the foreground and background separation processing in step 1102 can also be implemented by a depth camera.
  • the depth camera can use a time of flight (TOF) algorithm to detect the first foreground image and the first key frame image.
  • TOF time of flight
  • the foreground image processing may be performed on the first target object on the first foreground image.
  • the foreground beauty processing in FIG. 4 may be performed, or the foreground beauty processing in FIG. 9 may be performed.
  • the foreground calibration processing is not limited here.
  • the processed first foreground image is called a second foreground image, and the second foreground image is input to step 1105.
  • the second background image is the background image included in the second key frame image
  • the second key frame image The key frame image is obtained by shooting the target scene with the camera before.
  • the camera captures the target scene to obtain the second key frame image.
  • the second key frame image can be obtained at this time.
  • the second key frame image is extracted from the generated sequence frame video stream, the second key frame image not only includes the target object, but also includes the target scene.
  • the second key frame image may be a certain key frame image in the sequence frame video stream or a key frame image in the preview video stream.
  • the second key frame image may be the step S04 shown in FIG. Or the key frame image of the previous frame (that is, the previous frame) in step S14 shown in FIG. 9.
  • the first background image is subjected to background restoration processing through the second background image
  • the background restoration processing method may include the background image stitching in step S04 shown in FIG. 4 or the step S14 shown in FIG. Background image restoration.
  • the second background image and the first background image are spliced and repaired, thereby expanding the area of the first background image, or repairing the blank untextured area left by the segmentation of the first foreground image, so as to ensure that the processed second foreground image is compatible
  • the third background image after the background restoration is synthesized to ensure the effect of image synthesis.
  • step 1104 before step 1104 is obtained to perform background restoration processing on the first background image through the second background image, the method provided in the embodiments of the present application further includes:
  • the fifth background image is separated from the third key frame image in the preview video stream, and the preview video stream generates a sequence on the camera Preview and shoot the target scene before the frame video stream.
  • the target scene is previewed before the camera generates the sequence frame video stream to obtain the preview video stream, and the third key frame image is obtained from the preview video stream.
  • the third key frame image is obtained from the preview video stream.
  • the third key frame image since the camera only shoots the target scene, the third key frame image
  • the background image is separated in the, and the separated background image is used as the fifth background image, and the fifth background image can be used for the background image restoration processing of the fourth background image.
  • the sequence frame images in the preview stage can be captured (for example, a total of n ⁇ m frames are captured), and 1 key frame image can be generated for every n frames of the sequence image, and then it is detected and generated from the n ⁇ m frame images m key frame images; separate the foreground and background of the key frame images to obtain m foreground images and background images; perform background restoration processing on the adjacent backgrounds of m background images to obtain as comprehensive background image information as possible.
  • m background images are processed in the following manner for background restoration, and they are spliced in the order of the images.
  • two images are spliced each time, so that the restoration of the background image can be completed, so as to repair the blank untextured area that exists when the foreground image is deformed and the foreground and background are synthesized.
  • step 1104 performs background restoration processing on the first background image through the second background image, including:
  • the transformed second background image is spliced onto the first background image to obtain a third background image.
  • the feature point correspondence between the first background image and the second background image refers to the correspondence between the same feature point in the two background images
  • the first background image and the second background image can be obtained according to the feature point correspondence
  • the inter-frame change position relationship between the frames wherein the inter-frame change position relationship may be the position relationship between the frames when changing from the second background image to the first background image.
  • the inter-frame change position relationship may be the second background image and the first background image.
  • perspective transformation is performed on the second background image according to the changing position relationship between frames to obtain the transformed second background image.
  • the transformed second background image is spliced to the first background image, so that the blank and untextured area existing in the background image of the current frame can be repaired.
  • the third background image After the second background image after perspective transformation is spliced on the first background image, it can be output
  • the third background image the third background image can be used for the composition of the foreground background image. Since the third background image has been filled with blank and untextured areas, the effect of the image composition can be improved.
  • acquiring the inter-frame change position relationship between the first background image and the second background image according to the corresponding relationship of the feature points between the first background image and the second background image includes:
  • the initial positional relationship between frames is optimized by using the feature point correspondence relationship to obtain the inter-frame changing positional relationship between the first background image and the second background image.
  • the current frame pose parameters corresponding to the first background image can be completed by the IMU of the electronic device, and the last frame pose parameters corresponding to the second background image can be obtained from the pre-stored pose parameters, using the previous frame pose parameters and the current frame
  • the posture parameters calculate the initial positional relationship between the first background image and the second background image, and then perform feature extraction and feature matching on the first background image and the second background image to obtain the corresponding relationship between the feature points, and the extracted features It can be a DAISY feature.
  • the initial position relationship between frames is optimized using the corresponding relationship of the feature points to obtain the inter-frame changing position relationship between the first background image and the second background image.
  • An example is as follows, as shown in Figure 8.
  • the current posture parameters during shooting are obtained through the IMU, and the posture parameters of the previous frame are calculated to obtain the initial position relationship between the current frame and the previous frame;
  • the output Perform DAISY feature extraction and matching on the background image of the current frame and the background image of the previous frame, and optimize the initial position relationship between the current frame and the previous frame through the corresponding relationship of feature points, and obtain the change position relationship between frames.
  • the pose parameters of the current frame correspond to the current frame.
  • the pose parameters of the current frame become the pose parameters of the previous frame, and the next frame becomes the current frame.
  • the pose parameters of the next frame are changed to the pose parameters of the current frame.
  • performing perspective transformation on the second background image according to the changing position relationship between frames includes:
  • the perspective transformation matrix between the current frame image and the previous frame image is calculated through the optimized inter-frame change position relationship, the previous frame background image is perspective transformed according to the perspective transformation matrix, and the transformed image is stitched to the current frame
  • the background image of the current frame is repaired in the blank untextured area that exists in the background image of the current frame; finally, the repaired background image and the pose parameters of the current frame are output as the background image of the previous frame and the pose parameters of the previous frame to participate in the next In the frame loop, the repaired background image is output at the same time.
  • performing background restoration processing on the first background image through the second background image includes:
  • the background object image When the background object image enters the foreground missing area on the first background image, the background object image is used to perform background repair processing on the first background image.
  • the tracking algorithm judges that the target to be matched enters the missing foreground area of the current key frame, it searches for the target image with higher contour similarity from the target to be matched to repair the missing foreground area.
  • the background can be obtained by subtracting the n-1th frame from the nth frame image.
  • the part of the walking background character area will be blocked by the deducted foreground character (white area).
  • the background character area occluded by the foreground in the background image is complemented by the previously acquired background character image, so as to achieve the effect of repairing the background character when there are people walking in the background.
  • performing foreground and background synthesis processing on the second foreground image and the third background image may include the foreground and background synthesis in step S05 shown in FIG. 4 or step S15 shown in FIG. 9.
  • the implementation of this application also include:
  • the edge area of the foreground image is processed to reduce the unnatural problem of the image transition caused by the sudden change of the edge.
  • the edge processing can be realized through the feathering process.
  • the key frame video stream the video is inserted into the frame to realize the smooth output of the processed result image and video.
  • Sequence frames are generated between key frames through frame interpolation, which makes the video smoother and smoother, and the frame interpolation algorithm is not limited.
  • an electronic device in order to implement the above-mentioned functions, includes hardware and/or software modules corresponding to each function.
  • this application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed by hardware or computer software-driven hardware depends on the specific application and design constraint conditions of the technical solution. Those skilled in the art can use different methods for each specific application in combination with the embodiments to implement the described functions, but such implementation should not be considered as going beyond the scope of the present application.
  • the electronic device can be divided into functional modules according to the foregoing method examples.
  • each functional module can be divided corresponding to each function, or two or more functions can be integrated into one processing module.
  • the above integrated modules can be implemented in the form of hardware. It should be noted that the division of modules in this embodiment is illustrative, and is only a logical function division, and there may be other division methods in actual implementation.
  • FIG. 13 shows a schematic diagram of a possible composition of the electronic device 1300 involved in the foregoing embodiment.
  • the electronic device 1300 may include: a processing unit 1301, a photographing unit 1302, a storage unit 1303, and an output unit 1304.
  • the electronic device 1300 may further include: an IMU1305.
  • the processing unit 1301 may be used to support the electronic device 1300 to execute the above steps 1101 to 1105, step 1106, etc., and/or other processes used in the technology described herein.
  • the fourth processing unit is a physical unit used to calculate the key frame image through the sequence of frames, to segment the foreground area and the background area of the key frame image, and to process them separately.
  • the photographing unit 1302 may be used to support the electronic device 1300 to photograph the target object, preview the target scene, etc., and/or be used in other processes of the technology described herein.
  • the photographing unit is a physical unit used to photograph a target image, such as a lens, an image sensor, an image signal processor (image signal processor, ISP), and so on.
  • the storage unit 1303 may be used to support the electronic device 1300 to store the data generated in steps 1101 to 1107, etc., and/or used in other processes of the technology described herein.
  • the storage unit refers to a physical unit used to store information such as the sequence of frame images output by the camera, the foreground image and the background image divided by the processing unit.
  • the output unit 1304 refers to a physical unit that outputs information such as the synthesis result of the foreground and background area and presents it to the user.
  • IMU1305 is used to collect the posture parameters corresponding to the camera when shooting images, and send the posture parameters to the processing unit 1301.
  • the electronic device provided in this embodiment is used to execute the above-mentioned image processing method, and therefore can achieve the same effect as the above-mentioned implementation method.
  • the electronic device may include a processing module, a storage module, and a communication module.
  • the processing module can be used to control and manage the actions of the electronic device, for example, can be used to support the electronic device to perform the steps performed by the processing unit 1301 and the photographing unit 1302.
  • the storage module may be used to support the electronic device to execute the steps performed by the above-mentioned storage unit 1303, and to store program codes and data.
  • the communication module can be used to support communication between electronic devices and other devices.
  • the processing module may be a processor or a controller. It can implement or execute various exemplary logical blocks, modules and circuits described in conjunction with the disclosure of this application.
  • the processor may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of digital signal processing (DSP) and a microprocessor, and so on.
  • the storage module may be a memory.
  • the communication module may specifically be a radio frequency circuit, a Bluetooth chip, a Wi-Fi chip, and other devices that interact with other electronic devices.
  • the electronic device involved in this embodiment may be a device having the structure shown in FIG. 1.
  • This embodiment also provides a computer storage medium in which computer instructions are stored.
  • the computer instructions run on an electronic device, the electronic device executes the above-mentioned related method steps to implement the image processing method in the above-mentioned embodiment.
  • This embodiment also provides a computer program product, which when the computer program product runs on a computer, causes the computer to execute the above-mentioned related steps to implement the image processing method in the above-mentioned embodiment.
  • the embodiments of the present application also provide a device.
  • the device may specifically be a chip, component or module.
  • the device may include a processor and a memory connected to each other.
  • the memory is used to store computer execution instructions.
  • the processor can execute the computer-executable instructions stored in the memory, so that the chip executes the image processing methods in the foregoing method embodiments.
  • the electronic equipment, computer storage medium, computer program product, or chip provided in this embodiment are all used to execute the corresponding method provided above. Therefore, the beneficial effects that can be achieved can refer to the corresponding method provided above. The beneficial effects of the method are not described here.
  • the disclosed device and method may be implemented in other ways.
  • the device embodiments described above are only illustrative, for example, the division of modules or units is only a logical function division, and there may be other divisions in actual implementation, for example, multiple units or components may be combined or It can be integrated into another device, or some features can be ignored or not implemented.
  • the displayed or discussed mutual coupling or direct coupling or communication connection may be indirect coupling or communication connection of devices or units through some interfaces, and may be electrical, mechanical or other forms.
  • the units described as separate components may or may not be physically separate, and the components displayed as units may be one physical unit or multiple physical units, that is, they may be located in one place, or they may be distributed to multiple different places. Some or all of the units may be selected according to actual needs to achieve the objectives of the solutions of the embodiments.
  • each unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units may be integrated into one unit.
  • the above-mentioned integrated unit can be implemented in the form of hardware or software functional unit.
  • the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium.
  • the technical solutions of the embodiments of the present application are essentially or the part that contributes to the prior art, or all or part of the technical solutions can be embodied in the form of software products, which are stored in a storage medium It includes a number of instructions to make a device (may be a single-chip microcomputer, a chip, etc.) or a processor (processor) execute all or part of the steps of the methods in the embodiments of the present application.
  • the aforementioned storage media include: U disk, mobile hard disk, read only memory (read only memory, ROM), random access memory (random access memory, RAM), magnetic disk or optical disk and other media that can store program codes.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Studio Devices (AREA)
  • Image Processing (AREA)

Abstract

本申请实施例公开了一种图像处理方法和电子设备，用于提高前景图像处理后的图像合成效果。本申请实施例提供一种图像处理方法包括：显示从序列帧视频流中获取到的第一关键帧图像；获取对第一关键帧图像进行前景背景分离处理后得到的第一前景图像和第一背景图像；获取对第一前景图像上的第一目标对象进行前景图像处理后得到的第二前景图像；获取通过第二背景图像对第一背景图像进行背景修复处理后得到的第三背景图像，第二背景图像是第二关键帧图像上包括的背景图像，第二关键帧图像在获取第一关键帧图像之前通过摄像头拍摄目标场景得到；获取对第二前景图像和第三背景图像进行前景背景合成处理后得到的前景背景合成处理后的第一关键帧图像。

Description

一种图像处理方法和电子设备
本申请要求于2019年08月22日提交中国专利局、申请号为201910779341.6、发明名称为“一种图像处理方法和电子设备”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及图像处理技术领域，尤其涉及一种图像处理方法和电子设备。
背景技术
人们可以通过手机、平板或笔记本电脑等移动终端设备的摄像头对目标区域(例如人脸)进行实时拍摄和处理，达到目标区域实时处理的效果。这些功能有效帮助用户生成较满意效果的图像。
但是在该功能应用中，目标区域(例如人脸)可以是整个图像中的前景区域，该前景区域的面积在图像中所占比例较大，处理后前景区域通常会发生形变；直接将变形后的前景区域拼接回原图像的话，会造成前景区域与背景图像区域(即原图中除目标区域外的其他区域)拼接不吻合问题。因此，在对前景区域进行处理并发生变形时，亟需对背景图像区域进行处理，以解决前景与背景不吻合的现象。
目前，存在一种基于图像仿射变换的背景融合技术，即通过变形前后的前景对应关系将背景图像区域进行拉伸变形，使得背景图像区域拉伸扭曲至恰好与变形后前景边缘区域吻合，以消除前景和背景拼接时的不吻合问题。例如，在美颜瘦脸时对背景图像区域进行拉伸变形，以解决前景和背景拼接时存在的不吻合问题。但是这种对背景图像区域进行拉伸变形后，会存在处理痕迹明显的问题，整个图像的真实感受破坏，存在前景图像处理后导致的图像合成效果差的问题。
发明内容
本申请实施例提供了一种图像处理方法和电子设备，用于提高前景图像处理后的图像合成效果。
为解决上述技术问题，本申请实施例提供以下技术方案：
第一方面，本申请实施例提供一种图像处理方法，应用于具有摄像头的电子设备，其特征在于，所述方法包括：显示从序列帧视频流中获取到的第一关键帧图像，所述序列帧视频流由所述摄像头对包括第一目标对象在内的目标场景进行拍摄得到；获取对所述第一关键帧图像进行前景背景分离处理后得到的第一前景图像和第一背景图像，所述第一前景图像包括所述第一目标对象；获取对所述第一前景图像上的所述第一目标对象进行前景图像处理后得到的第二前景图像，所述第二前景图像包括：前景图像处理后的第一目标对象；获取通过第二背景图像对所述第一背景图像进行背景修复处理后得到的第三背景图像，所述第二背景图像是第二关键帧图像上包括的背景图像，所述第二关键帧图像在获取所述第一关键帧图像之前通过所述摄像头拍摄所述目标场景得到；获取对所述第二前景图像和所 述第三背景图像进行前景背景合成处理后得到的前景背景合成处理后的第一关键帧图像。在该方案中，在获取第一关键帧图像之前可以获取到第二背景图像，使用该第二背景图像可以对第一背景图像进行背景修复处理，因此得到的第三背景图像可以包含尽可能全面的背景图像信息，以此修补生成第二前景图像时造成的空白无纹理区域，提高前景图像处理后的图像合成效果。
在一种可能的实现方式中，所述显示从序列帧视频流中获取到的第一关键帧图像之前，所述方法还包括：显示从预览视频流中获取到的第二关键帧图像，所述预览视频流是在所述摄像头生成所述序列帧视频流之前对所述目标场景进行预览拍摄得到；获取从所述第二关键帧图像中分离出的所述第二背景图像。在该方案中，在摄像头生成序列帧视频流之前对目标场景进行预览拍摄得到预览视频流，从预览视频流中获取第二关键帧图像，此时由于摄像头只拍摄了目标场景，因此从第二关键帧图像中分离出背景图像，将分离出的背景图像作为第二背景图像。
在一种可能的实现方式中，所述获取通过第二背景图像对所述第一背景图像进行背景修复处理之前，所述方法还包括：显示从所述序列帧视频流中获取到的所述第二关键帧图像；获取对所述第二关键帧图像进行前景背景分离处理后得到的第四背景图像；获取通过第五背景图像对所述第四背景图像进行者背景修复处理后得到的所述第二背景图像，所述第五背景图像从预览视频流中的第三关键帧图像中分离出，所述预览视频流在所述摄像头生成所述序列帧视频流之前对所述目标场景进行预览拍摄得到。在该方案中，在摄像头生成序列帧视频流之前对目标场景进行预览拍摄得到预览视频流，从预览视频流中获取第三关键帧图像，此时由于摄像头只拍摄了目标场景，因此从第三关键帧图像中分离出背景图像，将分离出的背景图像作为第五背景图像，第五背景图像可以用于对第四背景图像的背景图像修复处理。
在一种可能的实现方式中，所述显示从序列帧视频流中获取到的第一关键帧图像之前，所述方法还包括：在所述摄像头生成所述序列帧视频流之前，通过所述摄像头对所述目标场景进行连续拍摄，得到多个连续的背景图像；获取对所述多个连续的背景图像进行累积叠加处理后得到的所述第二背景图像。在该方案中，在摄像头固定、拍摄环境不变的场景下，通过摄像头对目标场景进行连续拍摄，可以对多个连续的背景图像进行累积叠加，获取更全面的背景图像，该更全面的背景图像可以作为第二背景图像，第二背景图像可以用于对第一背景图像的背景图像修复处理。
在一种可能的实现方式中，所述显示从序列帧视频流中获取到的第一关键帧图像，包括：显示从序列帧视频流中获取到的n个序列帧图像，所述n为大于或等于2的正整数；获取从所述n个序列帧图像中确定出的1个原始关键帧图像和n-1个相关帧图像；使用所述n-1个相关帧图像的像素信息对所述原始关键帧图像进行清晰度增强处理，确定清晰度增强处理后的原始关键帧图像作为所述第一关键帧图像。在该方案中，可以对关键帧图像进行清晰度增强，从n个序列帧图像中确定出1个原始关键帧图像和n-1个相关帧图像，以原始关键帧图像为主图，相关帧图像是指n帧图像中除去当前作为原始关键帧图像的其他n-1帧图像。获取n-1个相关帧图像的像素信息，相关帧图像的像素信息可以是该相关帧图像的每个像素点的能量值，基于n-1个相关帧图像的像素信息对原始关键帧图像进行 清晰度增强处理，将清晰度增强处理后的原始关键帧图像作为第一关键帧图像，使用清晰度更高的第一关键帧图像，可以进一步的提高图像合成的效果。
在一种可能的实现方式中，所述使用所述n-1个相关帧图像的像素信息对所述原始关键帧图像进行清晰度增强处理，包括：获取所述原始关键帧图像的图像点原始能量值；从所述n-1个相关帧图像中获取位于所述原始关键帧图像的前后各k个相关帧图像，所述k的取值小于或等于(n-1)÷2；获取通过所述原始关键帧图像的前后各k个相关帧图像的图像点能量值对所述原始关键帧图像的图像点原始能量值进行优化处理后得到的所述原始关键帧图像的图像点优化后能量值。在该方案中，可以对关键帧图像进行清晰度增强，以原始关键帧图像为主图，确定前后各k(例如：k＝1,2,3)张相关帧图像，基于能量方式对原始关键帧图像的能量值进行最优化处理，以增强原始关键帧的图像清晰度。
在一种可能的实现方式中，所述获取对所述第二前景图像和所述第三背景图像进行前景背景合成处理后得到的前景背景合成处理后的第一关键帧图像之后，所述方法还包括：获取对所述前景背景合成处理后的第一关键帧图像进行前景边缘融合处理后得到的前景边缘融合处理后的第一关键帧图像；输出所述前景边缘融合处理后的第一关键帧图像。在该方案中，处理前景图像边缘区域，减少边缘因为突变造成图像过渡不自然问题，例如边缘的处理可有通过羽化处理来实现。根据关键帧视频流，对视频进行插帧处理，以实现处理结果图像视频流畅输出。
在一种可能的实现方式中，所述通过第二背景图像对所述第一背景图像进行背景修复处理，包括：根据所述第一背景图像和所述第二背景图像之间的特征点对应关系，获取所述第一背景图像和所述第二背景图像之间的帧间变化位置关系；获取根据所述帧间变化位置关系对所述第二背景图像进行透视变换后得到的变换后的第二背景图像；将所述变换后的第二背景图像拼接到所述第一背景图像上，得到所述第三背景图像。在该方案中，第一背景图像和第二背景图像之间的特征点对应关系是指相同的特征点在两个背景图像中的对应关系，根据特征点对应关系可以获取第一背景图像和第二背景图像之间的帧间变化位置关系，其中，帧间变化位置关系可以是从第二背景图像变化至第一背景图像时的帧间位置关系，例如帧间变化位置关系可以是第二背景图像与第一背景图像之间的姿态变化关系。接下来根据帧间变化位置关系对第二背景图像进行透视变换，得到变换后的第二背景图像。最后将变换后的第二背景图像拼接到第一背景图像上，从而可以修补当前帧背景图像存在的空白无纹理区域，第一背景图像上拼接有透视变换后的第二背景图像后，可以输出第三背景图像，该第三背景图像可用于前景背景图像的合成，由于第三背景图像上已经补了空白无纹理区域，因此可以提高图像合成的效果。
在一种可能的实现方式中，所述根据所述第一背景图像和所述第二背景图像之间的特征点对应关系，获取所述第一背景图像和所述第二背景图像之间的帧间变化位置关系，包括：获取所述第一背景图像对应的当前帧姿态参数；根据所述第二背景图像对应的上一帧姿态参数和所述当前帧姿态参数，获取所述第一背景图像和所述第二背景图像之间的帧间初始位置关系；获取对所述第一背景图像和所述第二背景图像进行特征提取和特征匹配后得到的特征点对应关系；使用所述特征点对应关系对所述帧间初始位置关系进行优化，得到所述第一背景图像和所述第二背景图像之间的帧间变化位置关系。在该方案中，第一背 景图像对应的当前帧姿态参数可以通过电子设备的IMU来完成，第二背景图像对应的上一帧姿态参数可以通过预先存储的姿态参数得到，使用上一帧姿态参数和当前帧姿态参数计算出第一背景图像和第二背景图像之间的帧间初始位置关系，接下来对第一背景图像和第二背景图像进行特征提取和特征匹配，得到特征点对应关系，提取的特征可以是DAISY特征，最后使用特征点对应关系对帧间初始位置关系进行优化，得到第一背景图像和第二背景图像之间的帧间变化位置关系。
在一种可能的实现方式中，所述根据所述帧间变化位置关系对所述第二背景图像进行透视变换，包括：使用所述帧间变化位置关系获取所述第一背景图像和所述第二背景图像之间的透视变换矩阵；获取使用所述透视变换矩阵对所述第二背景图像进行透视变换后得到的变换后的第二背景图像。在该方案中，通过优化后的帧间变化位置关系计算当前帧图像与上一帧图像之间的透视变换矩阵，根据透视变换矩阵对上一帧背景图像进行透视变换，将变换后的图像拼接至当前帧的背景图像，修补当前帧的背景图像存在的空白无纹理区域；最后，将修补后的背景图像和当前帧的姿态参数作为上一帧的背景图像和上一帧的姿态参数输出参与到下一帧循环中去，同时将修补后的背景图像进行输出。
在一种可能的实现方式中，所述通过第二背景图像对所述第一背景图像进行背景修复处理，包括：从所述第二背景图像中分割出背景对象图像；当所述背景对象图像进入所述第一背景图像上的前景缺失区域时，使用所述背景对象图像对所述第一背景图像进行背景修补处理。在该方案中，在摄像头固定、拍摄环境局部改变的场景下，例如拍摄背景中有人物走动的情况，首先通过帧间差分法分割各关键帧背景中的人物目标区域，作为待匹配目标保存；其次通过目标跟踪算法判断待匹配目标进入当前关键帧的前景缺失区域时，从待匹配目标中搜索轮廓相似度较高的目标图像对前景缺失区域进行修补。
第二方面，本申请实施例还提供了一种电子设备，该电子设备具有实现上述方面及上述方面的可能实现方式中电子设备行为的功能。功能可以通过硬件实现，也可以通过硬件执行相应的软件实现。硬件或软件包括一个或多个与上述功能相对应的模块或单元。例如，摄像头、一个或多个处理器、存储器、多个应用程序；以及一个或多个计算机程序，其中所述一个或多个计算机程序被存储在所述存储器中，所述一个或多个计算机程序包括指令，当所述指令被所述电子设备执行时，使得所述电子设备执行以下步骤:显示从序列帧视频流中获取到的第一关键帧图像，所述序列帧视频流由所述摄像头对包括第一目标对象在内的目标场景进行拍摄得到；获取对所述第一关键帧图像进行前景背景分离处理后得到的第一前景图像和第一背景图像，所述第一前景图像包括所述第一目标对象；获取对所述第一前景图像上的所述第一目标对象进行前景图像处理后得到的第二前景图像，所述第二前景图像包括：前景图像处理后的第一目标对象；获取通过第二背景图像对所述第一背景图像进行背景修复处理后得到的第三背景图像，所述第二背景图像是第二关键帧图像上包括的背景图像，所述第二关键帧图像在获取所述第一关键帧图像之前通过所述摄像头拍摄所述目标场景得到；获取对所述第二前景图像和所述第三背景图像进行前景背景合成处理后得到的前景背景合成处理后的第一关键帧图像。
在一种可能的实现方式中，当所述指令被所述电子设备执行时，使得所述电子设备具体执行以下步骤：显示从序列帧视频流中获取到的第一关键帧图像之前，显示从预览视频 流中获取到的第二关键帧图像，所述预览视频流是在所述摄像头生成所述序列帧视频流之前对所述目标场景进行预览拍摄得到；获取从所述第二关键帧图像中分离出的所述第二背景图像。
在一种可能的实现方式中，当所述指令被所述电子设备执行时，使得所述电子设备具体执行以下步骤：获取通过第二背景图像对所述第一背景图像进行背景修复处理之前，显示从所述序列帧视频流中获取到的所述第二关键帧图像；获取对所述第二关键帧图像进行前景背景分离处理后得到的第四背景图像；获取通过第五背景图像对所述第四背景图像进行者背景修复处理后得到的所述第二背景图像，所述第五背景图像从预览视频流中的第三关键帧图像中分离出，所述预览视频流在所述摄像头生成所述序列帧视频流之前对所述目标场景进行预览拍摄得到。
在一种可能的实现方式中，当所述指令被所述电子设备执行时，使得所述电子设备具体执行以下步骤：显示从序列帧视频流中获取到的第一关键帧图像之前，在所述摄像头生成所述序列帧视频流之前，通过所述摄像头对所述目标场景进行连续拍摄，得到多个连续的背景图像；获取对所述多个连续的背景图像进行累积叠加处理后得到的所述第二背景图像。
在一种可能的实现方式中，当所述指令被所述电子设备执行时，使得所述电子设备具体执行以下步骤：显示从序列帧视频流中获取到的n个序列帧图像，所述n为大于或等于2的正整数；获取从所述n个序列帧图像中确定出的1个原始关键帧图像和n-1个相关帧图像；使用所述n-1个相关帧图像的像素信息对所述原始关键帧图像进行清晰度增强处理，确定清晰度增强处理后的原始关键帧图像作为所述第一关键帧图像。
在一种可能的实现方式中，当所述指令被所述电子设备执行时，使得所述电子设备具体执行以下步骤：获取所述原始关键帧图像的图像点原始能量值；从所述n-1个相关帧图像中获取位于所述原始关键帧图像的前后各k个相关帧图像，所述k的取值小于或等于(n-1)÷2；获取通过所述原始关键帧图像的前后各k个相关帧图像的图像点能量值对所述原始关键帧图像的图像点原始能量值进行优化处理后得到的所述原始关键帧图像的图像点优化后能量值。
在一种可能的实现方式中，当所述指令被所述电子设备执行时，使得所述电子设备具体执行以下步骤：获取对所述前景背景合成处理后的第一关键帧图像进行前景边缘融合处理后得到的前景边缘融合处理后的第一关键帧图像；输出所述前景边缘融合处理后的第一关键帧图像。
在一种可能的实现方式中，当所述指令被所述电子设备执行时，使得所述电子设备具体执行以下步骤：根据所述第一背景图像和所述第二背景图像之间的特征点对应关系，获取所述第一背景图像和所述第二背景图像之间的帧间变化位置关系；获取根据所述帧间变化位置关系对所述第二背景图像进行透视变换后得到的变换后的第二背景图像；将所述变换后的第二背景图像拼接到所述第一背景图像上，得到所述第三背景图像。
在一种可能的实现方式中，当所述指令被所述电子设备执行时，使得所述电子设备具体执行以下步骤：获取所述第一背景图像对应的当前帧姿态参数；根据所述第二背景图像对应的上一帧姿态参数和所述当前帧姿态参数，获取所述第一背景图像和所述第二背景图 像之间的帧间初始位置关系；获取对所述第一背景图像和所述第二背景图像进行特征提取和特征匹配后得到的特征点对应关系；使用所述特征点对应关系对所述帧间初始位置关系进行优化，得到所述第一背景图像和所述第二背景图像之间的帧间变化位置关系。
在一种可能的实现方式中，当所述指令被所述电子设备执行时，使得所述电子设备具体执行以下步骤：使用所述帧间变化位置关系获取所述第一背景图像和所述第二背景图像之间的透视变换矩阵；获取使用所述透视变换矩阵对所述第二背景图像进行透视变换后得到的变换后的第二背景图像。
在一种可能的实现方式中，当所述指令被所述电子设备执行时，使得所述电子设备具体执行以下步骤：从所述第二背景图像中分割出背景对象图像；当所述背景对象图像进入所述第一背景图像上的前景缺失区域时，使用所述背景对象图像对所述第一背景图像进行背景修补处理。
在本申请的第二方面中，电子设备的组成模块还可以执行前述第一方面以及各种可能的实现方式中所描述的步骤，详见前述对第一方面以及各种可能的实现方式中的说明。
第三方面，本申请实施例还提供了一种电子设备，包括:摄像头；一个或多个处理器；存储器；多个应用程序；以及一个或多个计算机程序。其中，一个或多个计算机程序被存储在存储器中，一个或多个计算机程序包括指令。当指令被电子设备执行时，使得电子设备执行上述任一方面任一项可能的实现中的图像处理方法。
第四方面，本申请实施例还提供了一种电子设备，包括一个或多个处理器和一个或多个存储器。该一个或多个存储器与一个或多个处理器藕合，一个或多个存储器用于存储计算机程序代码，计算机程序代码包括计算机指令，当一个或多个处理器执行计算机指令时，使得电子设备执行上述任一方面任一项可能的实现中的图像处理方法。
第五方面，本申请实施例提供了一种计算机可读存储介质，所述计算机可读存储介质中存储有指令，当其在计算机上运行时，使得计算机执行上述第一方面所述的方法。
第六方面，本申请实施例提供了一种包含指令的计算机程序产品，当其在计算机上运行时，使得计算机执行上述第一方面所述的方法。
第七方面，本申请提供了一种芯片系统，该芯片系统包括处理器，用于支持电子设备实现上述方面中所涉及的功能，例如，发送或处理上述方法中所涉及的数据和/或信息。在一种可能的设计中，所述芯片系统还包括存储器，所述存储器，用于保存电子设备必要的程序指令和数据。该芯片系统，可以由芯片构成，也可以包括芯片和其他分立器件。
附图说明
图1为本申请实施例提供的一种电子设备的组成结构示意图；
图2a为本申请实施例提供的电子设备显示图像的示意图；
图2b为本申请实施例提供的一种视频场景示意图；
图2c为本申请实施例提供的另一种视频场景示意图；
图3为本申请实施例提供的一种图像处理方法流程图；
图4为本申请实施例提供的一种图像处理方法流程图；
图5为本申请实施例提供的关键帧生成示意图；
图6为本申请实施例提供的前景背景分离处理示意图；
图7为本申请实施例提供的背景图像拼接处理示意图；
图8为本申请实施例提供的背景图像拼接处理流程示意图；
图9为本申请实施例提供的摄像头校准流程图；
图10a为本申请实施例提供的第n-1帧的背景图像的示意图；
图10b为本申请实施例提供的第n帧的背景图像的示意图；
图10c为本申请实施例提供的在第n帧的背景图像中分离出背景对象的示意图；
图10d为本申请实施例提供的第n+1帧的背景图像的示意图；
图10e为本申请实施例提供的背景修复后的第n+1帧的背景图像的示意图；
图11为本申请实施例提供的一种图像处理方法流程图；
图12为本申请实施例提供的一种图像处理方法流程图；
图13为本申请实施例提供的一种电子设备的组成结构示意图；
图14为本申请实施例提供的一种电子设备的组成结构示意图。
具体实施方式
下面将结合本申请实施例中的附图，对本申请实施例中的技术方案进行描述。其中，在本申请实施例的描述中，除非另有说明，“/”表示或的意思，例如，A/B可以表示A或B；本文中的“和/或”仅仅是一种描述关联对象的关联关系，表示可以存在三种关系，例如，A和/或B，可以表示:单独存在A，同时存在A和B，单独存在B这三种情况。另外，在本申请实施例的描述中，‘多个’，是指两个或多于两个。
以下，术语“第一”、“第二”仅用于描述目的，而不能理解为指示或暗示相对重要性或者隐含指明所指示的技术特征的数量。由此，限定有“第一”、“第二”的特征可以明示或者隐含地包括一个或者更多个该特征。在本实施例的描述中，除非另有说明，“多个”的含义是两个或两个以上。
本申请实施例提供了一种图像处理方法，可以应用于电子设备，在获取第一关键帧图像之前可以获取到第二背景图像，第二背景图像是第二关键帧图像上包括的背景图像，第二关键帧图像在获取第一关键帧图像之前通过摄像头拍摄目标场景得到，从第一关键帧图像中可以通过前景背景分离处理得到第一前景图像和第一背景图像，使用该第二背景图像可以对第一背景图像进行背景修复处理以得到第三背景图像，因此得到的第三背景图像可以包含尽可能全面的背景图像信息，第三背景图像和第二前景图像进行前景背景合成处理，该第二前景图像是对第一前景图像进行前景图像处理后得到，以此修补生成第二前景图像时造成的空白无纹理区域，提高前景图像处理后的图像合成效果。
其中，目标对象可以有多种，例如可以有人物、车辆、花、动物、建筑、地面、天空等等。一种目标对象可以包括该种目标对象的多个物体。本申请实施例中一个完整的图像可以分割为两个部分：前景图像和背景图像，目标对象所在的区域是指属于该目标对象的物体所在的前景图像。特定的一个或多个物体是指，用户指定的一个或多个物体，或电子设备预设的一个或多个物体。或者，特定的一个或多个物体是指，用户指定的一个或多个物体类型包括的物体，或电子设备预设的一个或多个物体类型包括的物体，物体的位置和 尺寸等确定的一个或多个物体类型包括的物体。
图像分割也可以称为语义分割，是指把图像分成若干个特定的、具有特殊性质的区域并提出感兴趣目标的技术和过程，例如将一个完整的图像分割为前景图像和背景图像。图像分割方法可以有多种，例如，通过人脸识别方式检测人脸区域并估计身体区域，利用图割法从完整图像上对前景图像区域进行分割，并与背景图像区域进行分离，然后使用帧间差分算法或光流跟踪算法对前景图像区域进行跟踪分割，同时获取背景图像区域。
本申请实施例提供了一种图像处理方法，可以应用于手机、平板电脑、可穿戴设备、车载设备、增强现实(augmented reality，AR)/虚拟现实(virtual reality,VR)设备、笔记本电脑、超级移动个人计算机(ultra-mobile personal computer,UMPC)、上网本、个人数字助理(personal digital assistant,PDA)、智能终端、视频会议终端、图像拍摄终端等电子设备上，本申请实施例对电子设备的具体类型不作任何限制。其操作系统可以是Android、iOS、Windows Phone、BlackBerry OS等系统，具体本申请实施例不作限定。
以终端100为手机为例，图1示出的是与本申请实施例相关的手机100的部分结构的框图。参考图1，手机100包括、RF(Radio Frequency，射频)电路110、存储器120、其他输入设备130、显示屏140、传感器150、音频电路160、I/O子系统170、处理器180、以及摄像头190等部件。本领域技术人员可以理解，图1中示出的手机结构并不构成对手机的限定，可以包括比图示更多或更少的部件，或者组合某些部件，或者拆分某些部件，或者不同的部件布置。本领领域技术人员可以理解显示屏140属于用户界面(UI，User Interface)，且手机100可以包括比图示或者更少的用户界面。
下面结合图1对手机100的各个构成部件进行具体的介绍：
RF电路110可用于收发信息或通话过程中，信号的接收和发送，特别地，将基站的下行信息接收后，给处理器180处理；另外，将设计上行的数据发送给基站。通常，RF电路包括但不限于天线、至少一个放大器、收发信机、耦合器、低噪声放大器(Low Noise Amplifier，LNA)、双工器等。此外，RF电路110还可以通过无线通信与网络和其他设备通信。所述无线通信可以使用任一通信标准或协议，包括但不限于全球移动通讯系统(Global System of Mobile communication，GSM)、通用分组无线服务(General Packet Radio Service，GPRS)、码分多址(Code Division Multiple Access，CDMA)、宽带码分多址(Wideband Code Division Multiple Access，WCDMA)、长期演进(Long Term Evolution，LTE)、电子邮件、短消息服务(Short Messaging Service，SMS)等。
存储器120可用于存储软件程序以及模块，处理器180通过运行存储在存储器120的软件程序以及模块，从而执行手机100的各种功能应用以及数据处理。存储器120可主要包括存储程序区和存储数据区，其中，存储程序区可存储操作系统、至少一个功能所需的应用程序(比如声音播放功能、图象播放功能等)等；存储数据区可存储根据手机100的使用所创建的数据(比如音频数据、电话本等)等。此外，存储器120可以包括高速随机存取存储器，还可以包括非易失性存储器，例如至少一个磁盘存储器件、闪存器件、或其他易失性固态存储器件。
其他输入设备130可用于接收输入的数字或字符信息，以及产生与手机100的用户设置以及功能控制有关的键信号输入。具体地，其他输入设备130可包括但不限于物理键盘、 功能键(比如音量控制按键、开关按键等)、轨迹球、鼠标、操作杆、光鼠(光鼠是不显示可视输出的触摸敏感表面，或者是由触摸屏形成的触摸敏感表面的延伸)等中的一种或多种。其他输入设备130与I/O子系统170的其他输入设备控制器171相连接，在其他设备输入控制器171的控制下与处理器180进行信号交互。
显示屏140可用于显示由用户输入的信息或提供给用户的信息以及手机100的各种菜单，还可以接受用户输入。具体的显示屏140可包括显示面板141，以及触控面板142。其中显示面板141可以采用LCD(Liquid Crystal Display，液晶显示器)、OLED(Organic Light-Emitting Diode,有机发光二极管)等形式来配置显示面板141。触控面板142，也称为触摸屏、触敏屏等，可收集用户在其上或附近的接触或者非接触操作(比如用户使用手指、触笔等任何适合的物体或附件在触控面板142上或在触控面板142附近的操作，也可以包括体感操作；该操作包括单点控制操作、多点控制操作等操作类型。)，并根据预先设定的程式驱动相应的连接装置。可选的，触控面板142可包括触摸检测装置和触摸控制器两个部分。其中，触摸检测装置检测用户的触摸方位、姿势，并检测触摸操作带来的信号，将信号传送给触摸控制器；触摸控制器从触摸检测装置上接收触摸信息，并将它转换成处理器能够处理的信息，再送给处理器180，并能接收处理器180发来的命令并加以执行。此外，可以采用电阻式、电容式、红外线以及表面声波等多种类型实现触控面板142，也可以采用未来发展的任何技术实现触控面板142。进一步的，触控面板142可覆盖显示面板141，用户可以根据显示面板141显示的内容(该显示内容包括但不限于，软键盘、虚拟鼠标、虚拟按键、图标等等)，在显示面板141上覆盖的当触控面板142上或者附近进行操作，触控面板142检测到在其上或附近的触摸操作后，通过I/O子系统170传送给处理器180以确定触摸应用的类型以确定用户输入，随后处理器180根据触摸应用的类型在显示面板根据用户输入通过I/O子系统170在显示面板141上提供相应的视觉输出。虽然在图1中，触控面板142与显示面板141是作为两个独立的部件来实现手机100的输入和输入功能，但是在某些实施例中，可以将触控面板142与显示面板141集成而实现手机100的输入和输出功能。
手机100还可包括至少一种传感器150，比如光传感器、运动传感器以及其他传感器。具体地，光传感器可包括环境光传感器及接近传感器，其中，环境光传感器可根据环境光线的明暗来调节显示面板141的亮度，接近传感器可在手机100移动到耳边时，关闭显示面板141和/或背光。作为运动传感器的一种，加速计传感器可检测各个方向上(一般为三轴)加速度的大小，静止时可检测出重力的大小及方向，可用于识别手机姿态的应用(比如横竖屏切换、相关游戏、磁力计姿态校准)、振动识别相关功能(比如计步器、敲击)等；至于手机100还可配置的陀螺仪、气压计、湿度计、温度计、红外线传感器等其他传感器，在此不再赘述。
音频电路160、扬声器161，麦克风162可提供用户与手机100之间的音频接口。音频电路160可将接收到的音频数据转换后的信号，传输到扬声器161，由扬声器161转换为声音信号输出；另一方面，麦克风162将收集的声音信号转换为信号，由音频电路160接收后转换为音频数据，再将音频数据输出至RF电路110以发送给比如另一手机，或者将音频数据输出至存储器120以便进一步处理。
I/O子系统170用来控制输入输出的外部设备，可以包括其他设备输入控制器171、传感器控制器172、显示控制器173。可选的，一个或多个其他输入控制设备控制器171从其他输入设备130接收信号和/或者向其他输入设备130发送信号，其他输入设备130可以包括物理按钮(按压按钮、摇臂按钮等)、拨号盘、滑动开关、操纵杆、点击滚轮、光鼠(光鼠是不显示可视输出的触摸敏感表面，或者是由触摸屏形成的触摸敏感表面的延伸)。值得说明的是，其他输入控制设备控制器171可以与任一个或者多个上述设备连接。所述I/O子系统170中的显示控制器173从显示屏140接收信号和/或者向显示屏140发送信号。显示屏140检测到用户输入后，显示控制器173将检测到的用户输入转换为与显示在显示屏140上的用户界面对象的交互，即实现人机交互。传感器控制器172可以从一个或者多个传感器150接收信号和/或者向一个或者多个传感器150发送信号。
处理器180是手机100的控制中心，利用各种接口和线路连接整个手机的各个部分，通过运行或执行存储在存储器120内的软件程序和/或模块，以及调用存储在存储器120内的数据，执行手机100的各种功能和处理数据，从而对手机进行整体监控。可选的，处理器180可包括一个或多个处理单元；优选的，处理器180可集成应用处理器和调制解调处理器，其中，应用处理器主要处理操作系统、用户界面和应用程序等，调制解调处理器主要处理无线通信。可以理解的是，上述调制解调处理器也可以不集成到处理器180中。
手机100还可以包括摄像头190。例如手机100中可以包括一个或者多个摄像头，本申请实施例中用户可以操作摄像头对用户的人脸进行拍摄以生成序列帧视频流，在用户预览调整摄像头的同时，电子设备可以通过摄像头预览拍摄，以生成预览视频流，作为背景处理的待使用图像。举例说明如下，在使用电子设备的摄像头进行拍摄时，考虑到用户打开摄像头首先进行预览，并根据预览拍摄效果对摄像机或目标进行角度或姿态调整，例如，用户拿着手机拍摄预览时，不断调整手机位置和角度，自己也变换姿势和脸的角度，达到拍摄较好效果，又如，用户在使用笔记本的摄像头之前，也会调整自身位置达到拍摄较好效果，故而可利用用户在预览阶段提前打开摄像头，提前对背景信息进行收集，以获取到尽可能全面的背景信息。
尽管未示出，手机100还可以包括给各个部件供电的电源(比如电池)，可选的，电源可以通过电源管理系统与处理器180逻辑相连，从而通过电源管理系统实现管理充电、放电、以及功耗等功能。又如，手机100还可以包括蓝牙模块等，在此不再赘述。
为了便于理解，本申请以下实施例将以具有图1所示结构的手机为例，结合附图对本申请实施例提供的拍图像处理方法进行具体阐述。
图2a示出了手机的一种图形用户界面(graphical user interface,GUI),该GUI为手机的桌面。当手机检测到用户点击桌面上的相机应用(application，APP)的图标的操作后，可以启动相机应用，在拍摄界面上显示摄像头采集到的完整图像。在预览状态下，拍摄界面内可以实时显示预览图像。可以理解的是，在拍照模式和录像模式(即视频拍摄模式)下，拍摄界面的大小可以不同。在拍照模式下，当手机检测到用户执行拍照或者录制视频的操作后，手机执行拍照操作或者视频录制操作。例如图2a所示的完整图像可以分割为前景图像和背景图像，例如可以使用帧间差分算法或光流跟踪算法对前景图像进行跟踪分割，同时获取背景图像。图2a所示的人脸以及身体部分构成前景图像，在整个图像中除了前景图 像之外的其它图像区域构成背景图像，对于图2a所示的前景图像和背景图像只是一种示例情况，在实际应用中本申请实施例还可以采用其它的图像内容，并且图像的色彩也不做限定。本申请实施例中，通过对当前帧的背景图像结合预先采集的背景图像进行修补或者拼接，使得在前景图像被拉伸处理后，通过本申请实施例处理后的背景图像能够更好的与前景图像结合，从而消除前景图像处理后导致的图像合成效果差的问题。
图2b为本申请实施例提供的一种视频场景示意图，在图2b中电子设备可以是笔记本，该笔记本的键盘上设置有摄像头，该摄像头和显示屏不在同一个平面上。同样的，图2c为本申请实施例提供的另一种视频场景示意图，在图2c中电子设备可以是台式电脑，该台式电脑的显示屏下方设置有摄像头，该摄像头和显示屏不在同一个平面上。电子设备通过自身的摄像头可以采集到关键帧图像。在该电子设备的摄像头和显示屏之间存在以下至少一种关系：摄像头与其包括的显示屏之间存在距离，摄像头的朝向与其包括的显示屏的朝向不同时，在用户使用该电子设备与其他用户进行视频聊天或视频会议的过程中，如果用户的面部并未朝向摄像头，而是朝向其他地方，如朝向显示屏时，该电子设备通过摄像头采集到的图像序列中，便会存在人脸面部的朝向与电子设备的显示屏的垂线不平行的图像，在这种情况下，需要通过本申请实施例对前景图像进行人脸校准。
如图3所示，为本申请实施例提供的一种图像处理方法流程图。在图3左侧部分中，箭头从上至下表示了对视频流的多个处理过程，分别为：打开摄像头，生成关键帧图像，前景背景分离，针对前景背景分别处理，前景背景合成。在使用电子设备的摄像头进行拍摄时，考虑到用户打开摄像头首先进行预览，并根据预览拍摄效果对摄像机或目标进行角度或姿态调整，故而可利用用户在预览阶段提前对背景信息进行收集，以弥补后期对前景图像区域进行处理发生形状变化时留下的空白无纹理区域，其中空白无纹理区域指的是由于前景图像的处理发生形状变化，基于处理后的前景图像和原始的背景图像进行合成时存在的没有纹理的图像区域，例如对于前景图像上的目标对象的人脸进行瘦脸美化操作，将导致瘦掉的人脸部分和原始的背景图像进行合成时存在空白无纹理区域。
本申请实施例中，可以捕获预览阶段的序列帧图像(例如共捕获到n×m帧的图像)，每n帧序列图像可以生成1帧关键帧图像，则从n×m帧图像中检测生成m个关键帧图像；对关键帧图像进行前景背景分离以得到m个前景图像和相应的m个背景图像。对m个背景图像以前后相邻背景进行背景修复处理，获取尽可能全面的背景图像信息。例如，m个背景图像采用如下的方式进行背景修复处理，按照图像先后顺序进行拼接，比如第一帧背景图像和第二帧背景图像进行拼接，拼接后图像作为第二帧背景图像，再将第二帧背景图像和第三帧背景图像进行拼接，拼接后图像作为第三帧图像，依次类推拼接下去，每次都是两帧图像进行拼接，从而可以完成对背景图像的修复处理，以此修复前景图像变形后造成前景背景合成时存在的空白无纹理区域。
本申请实施例中，可以对当前帧的背景图像与预采集的背景图像进行拼接或者修补，本申请实施例中拼接或者修补后的背景图像包括有更全面的背景信息，因此基于本申请实施例拼接修补后的背景图像与前景图像处理后的前景图像进行合成时，能够填补存在的空白无纹理区域，本申请实施例中不会导致背景失真，且前景图像和背景图像合成后的图像效果真实自然。
本申请实施例的应用场景为使用手机、平板、笔记本电脑或其他具备拍摄功能的电子设备对目标对象进行拍摄与处理，且处理时目标区域发生明显形变，导致变形后与原背景图像难以无缝拼接的场景。本申请实施例的具体应用场景包括但不局限于美颜、摄像头校准等前景区域面积在整幅图像中占比较大且处理时前景区域发生明显形变的场景。
例如，在美颜应用场景下，可以在2D、3D美颜中，处理2D、3D瘦脸等形变下前景和背景的自然无缝拼接。在笔记本的摄像头校准场景下，可以处理人脸角度校正后前景和背景的自然无缝拼接。
如图3所示，接下来对本申请实施例提供的图像处理方法进行举例说明：
1、获取预览图像。
打开摄像头，进行预览，并根据预览拍摄效果，采集目标背景图像作为预览图像。
2、生成关键帧图像。
从摄像头获取每一帧图像数据作为序列帧视频流，其中，序列帧视频流是指包括多个序列帧图像组成的视频数据流，根据获取的每n个序列帧图像生成1个关键帧图像，依次循环在序列帧视频流中每间隔n帧图像输出1个关键帧图像，形成关键帧图像视频流。其中，从序列帧视频流中获取关键帧图像，针对关键帧图像按照本申请实施例进行后续处理，可以减少数据计算量，还可以去除连续几帧图像基本一样的重复情况，避免重复处理。
在另一实施例中，在关键帧图像生成阶段，为了能够生成更为清晰的关键帧图像，首先从n个序列帧图像中确定出一个原始关键帧图像，再以该原始关键帧图像作为中心，从n个序列帧图像中获取n-1个相关帧图像，其中，相关帧是指n个序列帧图像中除去当前作为关键帧图像的其他n-1帧图像，通过相关帧图像的能量值对原始关键帧图像进行图像融合，生成更为清晰的关键帧图像，从而可以解决生成更清晰关键帧图像的问题。
3、前景背景分离为两路视频流。
接收关键帧视频流，对关键帧的每张图像进行前景分割，并将分割后的前景图像和背景图像分离，生成两路视频流，分别为：前景图像视频流和背景图像视频流。
本申请实施例中，前景背景分离处理是指对关键帧图像中的前景部分和背景部分进行前景分割，从而将分割后的前景和背景的图像分离为两张单独的图像，通过前景背景分离处理可以生成两张图像：前景图像和背景图像。
4、前景背景分别处理。
读取步骤3输出的前景图像视频流和背景图像视频流，对分割后的前景图像视频流和背景图像视频流分别按照各自不同处理方式进行处理。具体的，可以对前景图像进行变形处理，对背景图像进行拼接修补处理，并将处理后的前景图像视频流和背景图像视频流作为两路视频流输出。
接下来说明对当前帧的背景图像的背景修复处理，背景修复处理可以有多种实现方式。例如，对当前帧的背景图像、前一帧(也可以称为上一帧)的背景图像进行拼接。前一帧的背景图像具有两种实现方式，一种是前一帧的背景图像为序列帧视频流中的当前帧的前一帧的关键帧图像对应的背景图像。另一种是前一帧的背景图像为预览阶段通过摄像头采集到的预览视频流中的预览图像对应的背景图像。
在对当前帧的背景图像、前一帧的背景图像进行拼接时，可以对前一帧的背景图像进 行透视变换，使得前一帧的背景图像与当前帧的背景图像进行纹理对齐。在没有前景图像的情况下尽可能填满所有的背景图像信息，比如在理想情况下拍用户的人像的时候，即使把人像从关键帧图像中抠除，通过背景拼接修补，图像仍然是完整的背景图像。
在另一实施例中，对前一帧的背景图像进行透视变换，需要首先计算透视变换矩阵。透视变换矩阵的计算有两种方式：一种方式是通过前后两帧背景图像的姿态参数生成帧间变化位置关系，通过帧间变化位置关系计算出透视变换矩阵，每帧背景图像的姿态参数可以通过惯性测量单元(inertial measurement unit，IMU)来完成。另一种是不使用姿态参数，而是使用前后两帧背景图像的特征点对齐来估计出帧间变化位置关系，通过帧间变化位置关系计算出透视变换矩阵，例如本申请实施例中可以计算背景图像的特征可以是DAISY特征。不限定的是，本申请实施例中还可以采用其它的特征点采集方式，此处不做限定。
需要说明的是，本申请实施例中需要对序列帧视频流中的多个关键帧图像进行背景修复处理，因此本申请实施例中对多个关键帧图像是循环处理关系，当前帧的背景图像的数据要保存下来，以用作下一帧中的“前一帧的背景图像”。
在另一些实施例中，背景修复处理可以有其它实现方式，例如，对当前帧的背景图像进行背景图像修复，针对不同情况进行背景修复，本申请实施例中可以至少包括如下三种背景图像修复的方式：一种是摄像头固定、拍摄环境不变的场景，在这种情况下，可以对背景图像区域进行累积叠加，获取更全面的背景图像。另一种是摄像头移动、拍摄环境不变的场景，在这种情况下可以基于前述的背景图像拼接处理方式，其中，在估算上一帧背景图像与当前帧背景图像之间的姿态变化关系时，若无法采集姿态参数，则可以直接通过特征匹配估算上一帧与当前帧的帧间位置变化关系。第三种是摄像头固定、拍摄环境局部改变的场景，例如拍摄背景中有人物走动的情况，首先通过帧间差分法分割各关键帧背景中的人物目标区域，作为待匹配目标保存，其次通过目标跟踪算法判断待匹配目标进入当前关键帧的前景缺失区域时，从待匹配目标中搜索轮廓相似度较高的目标图像对前景缺失区域进行修补。
通过上一帧的背景图像与当前帧的背景图像进行拼接，从而扩展当前帧的背景图像区域面积，或修补前景分割留下的空白无纹理区域，从而保证处理后的前景图像能够和背景修复后的背景图像进行合成。
5、前景背景合成。
对每一次输出处理后属于同一关键帧的前景图像和背景图像进行合成，保留前景图像的有效区域纹理，并对前景图像区域与背景图像区域交接的边缘区域进行融合处理，并将合成结果作为视频流输出。
在前景背景合成后，因为在上一步对当前帧的背景图像进行拼接修补，已经尽可能的把背景图像信息填满，即通过其他角度获取的背景图像信息通过步骤4的处理，可以填补原来由于前景图像变化形成的空白无纹理区域。
6、输出显示。
读取步骤5输出的每一帧图像作为结果输出显示。
接下来以下以美颜应用为例说明本申请实施例的具体实施过程。在美颜瘦脸时，由于人脸面部面积发生变化，导致美颜后的人脸与背景拼接不吻合的问题。本实施例利用预览 时搜集更多的背景信息，将美颜后人脸与背景图像区域之间存在的空白无纹理区域进行修补，使得在美颜的同时，使用真实的背景图像进行前景背景的合成处理。
如图4所示，为本申请实施例提供的一种图像处理方法流程图。以美颜应用为例，在美颜应用中的处理流程主要包括如下过程：
步骤S01：打开摄像头获取预览阶段的序列帧图像，输出预览视频流。然后生成关键帧图像，例如将生成的多个关键帧图像形成关键帧视频流。
首先进行关键帧图像的检测，对每n(如：n＝3,4,5,6,7)个序列帧图像进行关键帧图像检测。关键帧图像检测方法包括但不局限于：基于镜头的方法、帧平均法、直方图平均法、基于运动的分析法方法以及基于聚类的方法等。
接下来，本申请实施例可以对关键帧图像进行清晰度增强，以原始关键帧图像为主图，确定前后各k(如：k＝1,2,3)张相关帧图像，其中，每n个序列帧图像生成一帧关键帧图像，相关帧图像是指n帧图像中除去当前作为原始关键帧图像的其他n-1帧图像。基于能量方式对原始关键帧图像的能量值进行最优化处理，以增强原始关键帧的图像清晰度。其中，能量公式如下：
E total＝∑E(p i)，
Figure PCTCN2020105710-appb-000001
Figure PCTCN2020105710-appb-000002
其中，pi是指图像中的第i个图像点p，E(pi)是指在图像点pi处的能量值，I是指图像，泛指n帧图像中的任意一张图像。E(p)是泛指任意一个图像点的能量值，Ik(qi)指n帧图像中第k帧图像的图像点qi的像素值，I(qi)是指生成的关键帧图像在图像点qi修改后的像素值，B(p)是指以图像I中的任意一个图像点p为中心，建立一个5×5个图像块。
如图5所示，为本申请实施例提供的关键帧生成示意图。例如，序列帧视频流中共有n×m帧图像，从而可以生成m帧关键帧图像，图5中的2k+1等于n，即每2k+1帧图像生成1帧关键帧图像，因此视频流里有m个2k+1帧图像组成，通过上述的图像融合处理，可以生成清晰度高的m帧关键帧图像。
在关键帧图像生成之后，将生成的m个关键帧图像以视频流形式输出。
步骤S02：进行前景背景分离。
如图6所示，为本申请实施例提供的前景背景分离处理示意图。对读取的每一张关键帧图像进行前景分割，以实现前景背景分离，其中前景分割的方法有多种，例如可以是：a、基于人脸识别算法检测人脸区域并估计身体区域，利用图割(GrabCut)算法分割人体区域；b、当存在先验信息(如已有前一关键帧分割结果)时，可使用帧间差分算法或光流跟踪算法对前景区域进行跟踪分割，同时获取背景图像区域，并输出前景图像和背景图像两个视频流。对于图6所示的前景图像和背景图像只是一种示例情况，在实际应用中本申请实施例还可以采用其它的图像内容，并且图像的色彩也不做限定。
步骤S03：前景美颜处理。
读取前景视频流的每一帧图像，采用美颜算法处理前景图像，如2D美颜算法、3D几何美化算法，并输出处理后的前景视频流。
步骤S04：背景图像拼接。
读取背景视频流的每一帧图像，对分割后的背景分别进行背景拼接处理。
例如，对当前关键帧背景图像及前一关键帧背景图像进行背景拼接修补，具体拼接方法如下：将上一关键帧背景图像进行透视变换，使其与当前关键帧背景图像进行纹理对齐，然后与当前关键帧背景图像进行拼接，扩展当前背景图像区域面积或修补前景图像分割留下的无纹理区域，根据图像之间角度不同，图像拼接修补的结果可以是扩展背景图像区域，或修补前景图扣除后留下的空白区域，或扩展背景图像区域和修补前景图像扣除后留下的空白区域。
如图7所示，为本申请实施例提供的背景图像拼接处理示意图。例如，上一帧的背景图像靠左，当前帧的背景图像靠右，把上一帧的背景图像拼接到当前的背景图像中，在图7右边所示的拼接修补后的背景图像中，拼接后的当前帧的背景图像背景信息就会由于上一帧的背景图像的补充而向左边扩展。
又如，对上一帧的背景图像进行透视变换后，与当前帧的背景图像进行合并。当前图像人像扣除后留下空白区域，上一帧的背景图像从另外一个角度拍摄，可以拍到空白区域的背景图像信息，将该上一帧的背景图像进行透视变换并修补到当前空白区域，即形成前景背后区域的背景修补。
最后将处理后的背景结果分两路输出：一路结合当前姿态参数循环作为上一帧背景图像和上一帧图像的姿态参数输出，以用于下一帧背景图像的拼接；另一路作为待拼接背景输出到下一步中使用。
其中，透视变换矩阵可根据IMU获取当前姿态参数进行初始化，然后通过两帧图像的DAISY特征匹配优化得到。
如图8所示，为本申请实施例提供的背景图像拼接处理流程示意图，主要包括如下流程：
S041、获取关键帧视频流。
S042、对关键帧图像进行前景背景分割，得到分割后的背景图像。
S043、通过IMU获取当前帧姿态参数。
S044、获取已存储的上一帧姿态参数。
S045、根据当前帧姿态参数和上一帧姿态参数，获取帧间初始位置关系。
S046、获取上一帧背景图像。
S047、根据分割后的背景图像和上一帧背景图像，对帧间初始位置关系进行优化，得到帧间变化位置关系。
S048、使用帧间变化位置关系进行背景拼接修补。
S049、输出拼接修补后的当前帧的背景图像。
在图8中，通过IMU获取当前拍摄时的姿态参数，与上一帧的姿态参数进行计算得到当前帧与上一帧之间的帧间初始位置关系；其次，对步骤S03输出的当前帧背景图像和上一帧背景图像进行DAISY特征提取和匹配，通过特征点对应关系优化当前帧与上一帧之间的帧间初始位置关系，得到帧间变化位置关系；然后，通过优化后的帧间变化位置关系计算当前帧图像与上一帧图像之间的透视变换矩阵，根据透视变换矩阵对上一帧背景图像进 行透视变换，将变换后的图像拼接至当前帧的背景图像，修补当前帧的背景图像存在的空白无纹理区域；最后，将修补后的背景图像和当前帧的姿态参数作为上一帧的背景图像和上一帧的姿态参数输出参与到下一帧循环中去，同时将修补后的背景图像作为步骤S05的输入数据进行输出。
需要说明的是，当前帧的姿态参数和当前帧对应，当前帧变为上一帧的同时，当前帧的姿态参数变为上一帧图像的姿态参数，而下一帧图像变为当前帧图像的同时，拍摄下一帧时的姿态参数变为当前帧的姿态参数。
步骤S05：前景背景合成。
将处理后的前景图像和拼接后的背景图像进行合成，保留前景图像有效区域，无效区域通过背景进行补充，作为合成后的关键帧视频流输出。基于步骤S04可知，提供背景图像拼接修补，如果角度够大，背景图像信息比较全，会形成一张全背景图像，将当前图像中扣除前景区域的空白部分全部进行填补，因此当前景发生变形、移动甚至消失，背景图像都是完整的，因此可以用拼接完整背景的图像填充前景变化后留下的无效区域(即空白无纹理区域)。
步骤S06：边缘融合处理。
处理前景图像边缘区域，减少边缘因为突变造成图像过渡不自然问题，例如边缘的处理可有通过羽化处理来实现。
步骤S07：输出显示。
根据关键帧视频流，对视频进行插帧处理，以实现处理结果图像视频流畅输出。关键帧之间通过插帧技术生成序列帧，使得视频平滑流畅性较好，插帧算法不做限定。
本申请实施例中，通过用户在美颜自拍前预览阶段获取更多背景图像信息，解决由于瘦脸等处理造成的无纹理缝隙区域修补，可便捷地应用于手机和平板等移动终端设备上，实现在美颜的同时对背景信息的保护。
相较于当前美颜瘦脸时对背景进行拉伸调整的技术，本申请实施例技术通过多帧学习的方法较好的获得更全面的背景信息，有效补充了前背景拼接时的空隙，无背景失真，效果真实自然。
接下来，以下通过摄像头校准应用实施例，说明基于本申请实施例的另一个应用解决方案的实现流程。用户在使用连接笔记本电脑的内置摄像头或外置摄像头时，由于摄像头位置与屏幕存在一定距离和角度，使得需要对人脸区域进行调整，调整后包含人脸的前景区域与背景同样存在拼接不吻合的问题，为修补前景背景合成时存在的无纹理空白区域，同样需要本专利技术进行处理。
如图9所示，为本申请实施例提供的摄像头校准流程图。本申请实施例在摄像头校准应用的系统流程，分步说明如下：
步骤S11：可以参照前述实施例中的步骤S01，此处不再赘述。
步骤S12：可以参照前述实施例中的步骤S02，此处不再赘述。
步骤S13：前景人脸校准。
前景人脸校准主要处理方式：对前景区域进行三维重建，并对三维人脸进行角度调整，后映射生成2D图像。
步骤S14：背景图像修复，针对不同情况分别进行背景修复，例如：
一种是摄像头固定、拍摄环境不变的场景，在这种情况下，可以对背景图像区域进行累积叠加，获取更全面的背景图像。
另一种是摄像头移动、拍摄环境不变的场景，在这种情况下可以基于前述步骤S04的背景图像拼接处理方式，其中，在估算上一帧背景图像与当前帧背景图像之间的姿态变化关系时，如无法采集姿态参数，直接通过DAISY特征匹配估算上一帧与当前帧的帧间位置变化关系。
第三种是摄像头固定、拍摄环境局部改变的场景，例如拍摄背景中有人物走动的情况，首先通过帧间差分法分割各关键帧背景中的人物目标区域，作为待匹配目标保存；其次通过目标跟踪算法判断待匹配目标进入当前关键帧的前景缺失区域时，从待匹配目标中搜索轮廓相似度较高的目标图像对前景缺失区域进行修补。
如图10a所示，为本申请实施例提供的第n-1帧的背景图像的示意图，图10b所示，为本申请实施例提供的第n帧的背景图像的示意图，则通过第n帧的背景图像和第n-1帧的背景图像的比较可知，在第n帧的背景图像中可以分离出图10c所示的背景对象。图10d所示，为本申请实施例提供的第n+1帧的背景图像的示意图，使用图10c所示的背景对象和第n+1帧的背景图像中进行背景修复，前景图像在第n帧移动到第n+1帧时出现移动，会导致前景图像出现前景缺失区域，图10e为本申请实施例提供的背景修复后的第n+1帧的背景图像的示意图，图10c所示的背景对象可以和图10d所示的第n+1帧的背景图像进行合成，从而实现背景图是动态图时的背景修补。具体的，图10b中有人物走动，在图10a所示的第n-1帧时人物未进入背景，在第n帧时人物进入背景，用第n帧图像减去第n-1帧图像可以得到图10c所示的背景人物图像，图10d中在第n+1帧时人物走到前景图像区域，这时作为走动的背景人物部分区域会被扣除掉的前景人物(即图10d中的白色区域)遮挡。为补充被遮挡的背景人物区域，通过前面获取的背景人物图像将背景图像中被前景遮挡的背景人物区域补全，达到背景中有人物走动情况下对背景人物的修补效果。
步骤S14的执行设备可以是笔记本电脑。在步骤S14的背景处理过程中，在前述步骤S04的背景处理技术基础上，增加了背景修复的多种实现场景，例如可以针对局部变化的背景进行处理，这点区别于前述步骤S04中的背景拼接方式。
步骤S15：可以参照前述实施例中的步骤S05，此处不再赘述。
步骤S16：可以参照前述实施例中的步骤S06，此处不再赘述。
步骤S17：可以参照前述实施例中的步骤S07，此处不再赘述。
通过前述的举例说明可知，用户在使用笔记本电脑的内置摄像头或外置摄像头时，通过在预览阶段搜集更多背景图像信息，解决由于面部校正或姿态调整造成的前景背景合成不吻合问题，实现在人脸校准的同时对背景信息的保护。
需要说明的是，本申请实施例可扩展至其他涉及需要对前景分割后进行变形或移动处理并需要保护背景信息的场景，在存在可以提前获取背景图像的情况下，其场景包括但不局限于：通过摄像头捕捉人体实现游戏交互，在对分割后的人体图像进行处理变形的同时，修补由于人体变化留下的空白区域纹理，使得游戏更真实；增强现实技术，对拍摄场景中的目标物体(如沙发、桌子、气球等)进行虚拟移动或变形，修补物体变化后留下的无纹 理区域，使得交互效果更真实。
结合上述实施例及相关附图，本申请实施例提供了一种图像处理方法，该方法可以在如图1所示的具有摄像头的电子设备(例如手机、平板电脑等)中实现。如图11所示，一种图像处理方法，应用于具有摄像头的电子设备，该方法可以包括以下步骤:
1101、显示从序列帧视频流中获取到的第一关键帧图像，序列帧视频流由摄像头对包括第一目标对象在内的目标场景进行拍摄得到。
示例性的，摄像头对包括第一目标对象在内的目标场景进行拍摄得到，从而可以生成序列帧视频流，其中，序列帧视频流是指摄像头拍摄到的多个序列帧图像组成的视频数据流，第一目标对象可以是控制电子设备的用户的头像，目标场景可以是指包括用户的头像在内的拍摄环境场景，例如目标场景可以是一个拍摄背景，在该目标场景内除了用户的头像之外，还有背景图像。例如目标场景可以是用户在进行视频会议时拍摄的会议场景。
在一个示例中，第一关键帧图像可以是序列帧视频流中的某一个关键帧图像，例如第一关键帧图像可以是前述图4所示的步骤S01或图9中所示的步骤S11中的当前关键帧图像。
在本申请的一些实施例中，步骤1101显示从序列帧视频流中获取到的第一关键帧图像之前，本申请实施例提供的方法还包括如下步骤：
显示从预览视频流中获取到的第二关键帧图像，预览视频流是在摄像头生成序列帧视频流之前对目标场景进行预览拍摄得到；
获取从第二关键帧图像中分离出的第二背景图像。
其中，第一关键帧图像可以是从序列帧视频流中提取出的第一个图像，在摄像头生成序列帧视频流之前对目标场景进行预览拍摄得到预览视频流，从预览视频流中获取第二关键帧图像，此时由于摄像头只拍摄了目标场景，因此从第二关键帧图像中分离出背景图像，将分离出的背景图像作为第二背景图像，第二背景图像可以用于对第一背景图像的背景图像修复处理，详见后续实施例中的描述。
举例说明如下，在使用电子设备的摄像头进行拍摄时，考虑到用户打开摄像头首先进行预览，并根据预览拍摄效果对摄像机或目标进行角度或姿态调整，故而可利用用户在预览阶段提前对背景图像进行收集，以弥补后期对前景图像进行处理发生形状变化时留下的空白无纹理区域。
在本申请的另一些实施例中，步骤1101显示从序列帧视频流中获取到的第一关键帧图像之前，本申请实施例提供的方法还包括如下步骤：
在摄像头生成序列帧视频流之前，通过摄像头对目标场景进行连续拍摄，得到多个连续的背景图像；
获取对多个连续的背景图像进行累积叠加处理后得到的第二背景图像。
例如，在摄像头固定、拍摄环境不变的场景下，通过摄像头对目标场景进行连续拍摄，可以对多个连续的背景图像进行累积叠加，获取更全面的背景图像，该更全面的背景图像可以作为第二背景图像，第二背景图像可以用于对第一背景图像的背景图像修复处理，详见后续实施例中的描述。
在本申请的一些实施例中，步骤1101显示从序列帧视频流中获取到的第一关键帧图 像，包括：
显示从序列帧视频流中获取到的n个序列帧图像，n为大于或等于2的正整数；
获取从n个序列帧图像中确定出的1个原始关键帧图像和n-1个相关帧图像；
使用n-1个相关帧图像的像素信息对原始关键帧图像进行清晰度增强处理，确定清晰度增强处理后的原始关键帧图像作为第一关键帧图像。
其中，本申请实施例可以对关键帧图像进行清晰度增强，从n个序列帧图像中确定出1个原始关键帧图像和n-1个相关帧图像，以原始关键帧图像为主图，相关帧图像是指n帧图像中除去当前作为原始关键帧图像的其他n-1帧图像。获取n-1个相关帧图像的像素信息，相关帧图像的像素信息可以是该相关帧图像的每个像素点的能量值，基于n-1个相关帧图像的像素信息对原始关键帧图像进行清晰度增强处理，将清晰度增强处理后的原始关键帧图像作为第一关键帧图像，使用清晰度更高的第一关键帧图像，可以进一步的提高图像合成的效果。
进一步的，在本申请的一些实施例中，使用n-1个相关帧图像的像素信息对原始关键帧图像进行清晰度增强处理，包括：
获取原始关键帧图像的图像点原始能量值；
从n-1个相关帧图像中获取位于原始关键帧图像的前后各k个相关帧图像，k的取值小于或等于(n-1)÷2；
获取通过原始关键帧图像的前后各k个相关帧图像的图像点能量值对原始关键帧图像的图像点原始能量值进行优化处理后得到的原始关键帧图像的图像点优化后能量值。
其中，本申请实施例可以对关键帧图像进行清晰度增强，以原始关键帧图像为主图，确定前后各k(例如：k＝1,2,3)张相关帧图像，基于能量方式对原始关键帧图像的能量值进行最优化处理，以增强原始关键帧的图像清晰度。详见前述实施例中步骤S01中的解释说明。
1102、获取对第一关键帧图像进行前景背景分离处理后得到的第一前景图像和第一背景图像，第一前景图像包括第一目标对象。
示例性的，在获取到第一关键帧图像之后，针对第一关键帧图像进行前景背景分离处理，得到第一前景图像和第一背景图像，结合图2a所示，一个完整的关键帧图像可以被分割为：一个前景图像和一个背景图像。其中，分割出的第一前景图像可以包括第一目标对象，结合图2a所示，第一目标对象可以包括是用户的人脸以及身体部分。
本申请实施例中，前景背景分离处理是指对关键帧图像中的前景部分和背景部分进行前景分割，从而将分割后的前景和背景的图像分离为两张单独的图像，通过前景背景分离处理可以生成两张图像：前景图像和背景图像。
在步骤1102执行完成之后，本申请实施例可以触发执行步骤1103和步骤1104，例如可以先执行步骤1103，再执行步骤1104；或者先执行步骤1104，再执行步骤1103；或者同时执行步骤1103和步骤1104，此处不做限定。
需要说明的是，步骤1102中前景背景分离处理还可以通过深度摄像头实现，例如深度摄像头可以使用飞行时间(time of flight，TOF)算法，从第一关键帧图像中检测出第一前景图像和第一背景图像。
1103、获取对第一前景图像上的第一目标对象进行前景图像处理后得到的第二前景图像，第二前景图像包括：前景图像处理后的第一目标对象。
示例性的，对第一前景图像进行处理时，可以对第一前景图像上的第一目标对象进行前景图像处理，例如可以执行前述图4中的前景美颜处理，也可以执行前述图9中的前景校准处理，此处不做限定。为了便于表示，可以处理后的第一前景图像称为第二前景图像，将该第二前景图像输入至步骤1105。
1104、获取通过第二背景图像对第一背景图像进行背景修复处理后得到的第三背景图像，第二背景图像是第二关键帧图像上包括的背景图像，第二关键帧图像在获取第一关键帧图像之前通过摄像头拍摄目标场景得到。
示例性的，摄像头对目标场景进行拍摄得到第二关键帧图像，例如在预览拍摄阶段只对目标场景进行拍摄，此时可以得到第二关键帧图像。又如，在生成的序列帧视频流中提取出第二关键帧图像，则第二关键帧图像中不仅包括有目标对象，还包括有目标场景。
在一个示例中，第二关键帧图像可以是序列帧视频流中的某一个关键帧图像或者是预览视频流中的关键帧图像，例如第二关键帧图像可以是前述图4所示的步骤S04或图9中所示的步骤S14中的上一帧(即前一帧)的关键帧图像。
在一个示例中，通过第二背景图像对第一背景图像进行背景修复处理，背景修复处理的方式可以包括前述图4所示的步骤S04的背景图像拼接或图9中所示的步骤S14中的背景图像修复。
通过第二背景图像与第一背景图像进行拼接修补，从而扩展第一背景图像的区域面积，或修补第一前景图像分割留下的空白无纹理区域，从而保证处理后的第二前景图像能够和背景修复后的第三背景图像进行合成，以保证图像合成的效果。
在本申请的一些实施例中，步骤1104获取通过第二背景图像对第一背景图像进行背景修复处理之前，本申请实施例提供的方法还包括：
显示从序列帧视频流中获取到的第二关键帧图像；
获取对第二关键帧图像进行前景背景分离处理后得到的第四背景图像；
获取通过第五背景图像对第四背景图像进行者背景修复处理后得到的第二背景图像，第五背景图像从预览视频流中的第三关键帧图像中分离出，预览视频流在摄像头生成序列帧视频流之前对目标场景进行预览拍摄得到。
其中，在摄像头生成序列帧视频流之前对目标场景进行预览拍摄得到预览视频流，从预览视频流中获取第三关键帧图像，此时由于摄像头只拍摄了目标场景，因此从第三关键帧图像中分离出背景图像，将分离出的背景图像作为第五背景图像，第五背景图像可以用于对第四背景图像的背景图像修复处理。
本申请实施例中，可以捕获预览阶段的序列帧图像(例如共捕获到n×m帧的图像)，每n帧序列图像可以生成1帧关键帧图像，则从n×m帧图像中检测生成m个关键帧图像；对关键帧图像进行前景背景分离以得到m个前景图像和背景图像；对m个背景图像以前后相邻背景进行背景修复处理，获取尽可能全面的背景图像信息。例如，m个背景图像采用如下的方式进行背景修复处理，按照图像先后顺序进行拼接，比如先使用预览帧视频流中的关键帧图像对第一帧背景图像进行背景图像修复处理，输出第一帧背景图像，然后将第 一帧背景图像和第二帧背景图像进行拼接，拼接后图像作为第二帧背景图像，再和第三帧背景图像进行拼接，拼接后图像作为第三帧图像，依次类推拼接下去，每次都是两帧图像进行拼接，从而可以完成对背景图像的修复处理，以此修复前景图像变形后造成前景背景合成时存在的空白无纹理区域。
在本申请的一些实施例中，步骤1104通过第二背景图像对第一背景图像进行背景修复处理，包括：
根据第一背景图像和第二背景图像之间的特征点对应关系，获取第一背景图像和第二背景图像之间的帧间变化位置关系；
获取根据帧间变化位置关系对第二背景图像进行透视变换后得到的变换后的第二背景图像；
将变换后的第二背景图像拼接到第一背景图像上，得到第三背景图像。
其中，第一背景图像和第二背景图像之间的特征点对应关系是指相同的特征点在两个背景图像中的对应关系，根据特征点对应关系可以获取第一背景图像和第二背景图像之间的帧间变化位置关系，其中，帧间变化位置关系可以是从第二背景图像变化至第一背景图像时的帧间位置关系，例如帧间变化位置关系可以是第二背景图像与第一背景图像之间的姿态变化关系。接下来根据帧间变化位置关系对第二背景图像进行透视变换，得到变换后的第二背景图像。最后将变换后的第二背景图像拼接到第一背景图像上，从而可以修补当前帧背景图像存在的空白无纹理区域，第一背景图像上拼接有透视变换后的第二背景图像后，可以输出第三背景图像，该第三背景图像可用于前景背景图像的合成，由于第三背景图像上已经补了空白无纹理区域，因此可以提高图像合成的效果。
在另一个示例中，根据第一背景图像和第二背景图像之间的特征点对应关系，获取第一背景图像和第二背景图像之间的帧间变化位置关系，包括：
获取第一背景图像对应的当前帧姿态参数；
根据第二背景图像对应的上一帧姿态参数和当前帧姿态参数，获取第一背景图像和第二背景图像之间的帧间初始位置关系；
获取对第一背景图像和第二背景图像进行特征提取和特征匹配后得到的特征点对应关系；
使用特征点对应关系对帧间初始位置关系进行优化，得到第一背景图像和第二背景图像之间的帧间变化位置关系。
其中，第一背景图像对应的当前帧姿态参数可以通过电子设备的IMU来完成，第二背景图像对应的上一帧姿态参数可以通过预先存储的姿态参数得到，使用上一帧姿态参数和当前帧姿态参数计算出第一背景图像和第二背景图像之间的帧间初始位置关系，接下来对第一背景图像和第二背景图像进行特征提取和特征匹配，得到特征点对应关系，提取的特征可以是DAISY特征，最后使用特征点对应关系对帧间初始位置关系进行优化，得到第一背景图像和第二背景图像之间的帧间变化位置关系。举例说明如下，如图8所示，首先通过IMU获取当前拍摄时的姿态参数，与上一帧的姿态参数进行计算得到当前帧与上一帧之间的帧间初始位置关系；其次，对输出的当前帧背景图像和上一帧背景图像进行DAISY特征提取和匹配，通过特征点对应关系优化当前帧与上一帧之间的帧间初始位置关系，得到 帧间变化位置关系。
需要说明的是，当前帧的姿态参数和当前帧对应，当前帧变为上一帧的同时，当前帧的姿态参数变为上一帧图像的姿态参数，而下一帧图像变为当前帧图像的同时，拍摄下一帧时的姿态参数变为当前帧的姿态参数。
在另一个示例中，根据帧间变化位置关系对第二背景图像进行透视变换，包括：
使用帧间变化位置关系获取第一背景图像和第二背景图像之间的透视变换矩阵；
获取使用透视变换矩阵对第二背景图像进行透视变换后得到的变换后的第二背景图像。
其中，通过优化后的帧间变化位置关系计算当前帧图像与上一帧图像之间的透视变换矩阵，根据透视变换矩阵对上一帧背景图像进行透视变换，将变换后的图像拼接至当前帧的背景图像，修补当前帧的背景图像存在的空白无纹理区域；最后，将修补后的背景图像和当前帧的姿态参数作为上一帧的背景图像和上一帧的姿态参数输出参与到下一帧循环中去，同时将修补后的背景图像进行输出。
在另一个示例中，通过第二背景图像对第一背景图像进行背景修复处理，包括：
从第二背景图像中分割出背景对象图像；
当背景对象图像进入第一背景图像上的前景缺失区域时，使用背景对象图像对第一背景图像进行背景修补处理。
其中，在摄像头固定、拍摄环境局部改变的场景下，例如拍摄背景中有人物走动的情况，首先通过帧间差分法分割各关键帧背景中的人物目标区域，作为待匹配目标保存；其次通过目标跟踪算法判断待匹配目标进入当前关键帧的前景缺失区域时，从待匹配目标中搜索轮廓相似度较高的目标图像对前景缺失区域进行修补。
如图10a至图10e所示，有人物走动，在第n-1帧时人物未进入背景，在第n帧时人物进入背景，用第n帧图像减去第n-1帧图像可以得到背景人物图像，在第n+1帧时人物走到前景图像区域，这时作为走动的背景人物部分区域会被扣除掉的前景人物(白色区域)遮挡。为补充被遮挡的背景人物区域，通过前面获取的背景人物图像将背景图像中被前景遮挡的背景人物区域补全，达到背景中有人物走动情况下对背景人物的修补效果。
1105、获取对第二前景图像和第三背景图像进行前景背景合成处理后得到的前景背景合成处理后的第一关键帧图像。
示例性的，对第二前景图像和第三背景图像进行前景背景合成处理，可以包括前述图4所示的步骤S05或图9中所示的步骤S15中的前景背景合成。
在另一实施例中，参见图12，在上述步骤1105获取对第二前景图像和第三背景图像进行前景背景合成处理后得到的前景背景合成处理后的第一关键帧图像之后，本申请实施例提供的图像处理方法还包括：
1106、获取对前景背景合成处理后的第一关键帧图像进行前景边缘融合处理后得到的前景边缘融合处理后的第一关键帧图像；
1107、输出前景边缘融合处理后的第一关键帧图像。
其中，处理前景图像边缘区域，减少边缘因为突变造成图像过渡不自然问题，例如边缘的处理可有通过羽化处理来实现。根据关键帧视频流，对视频进行插帧处理，以实现处 理结果图像视频流畅输出。关键帧之间通过插帧技术生成序列帧，使得视频平滑流畅性较好，插帧算法不做限定。
可以理解的是，电子设备为了实现上述功能，其包含了执行各个功能相应的硬件和/或软件模块。结合本文中所公开的实施例描述的各示例的算法步骤，本申请能够以硬件或硬件和计算机软件的结合形式来实现。某个功能究竟以硬件还是计算机软件驱动硬件的方式来执行，取决于技术方案的特定应用和设计约束条件。本领域技术人员可以结合实施例对每个特定的应用来使用不同方法来实现所描述的功能，但是这种实现不应认为超出本申请的范围。
本实施例可以根据上述方法示例对电子设备进行功能模块的划分，例如，可以对应各个功能划分各个功能模块，也可以将两个或两个以上的功能集成在一个处理模块中。上述集成的模块可以采用硬件的形式实现。需要说明的是，本实施例中对模块的划分是示意性的，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式。
在采用对应各个功能划分各个功能模块的情况下，图13示出了上述实施例中涉及的电子设备1300的一种可能的组成示意图，如图13所示，该电子设备1300可以包括:处理单元1301、拍摄单元1302、存储单元1303和输出单元1304。如图14所示，该电子设备1300还可以包括：IMU1305。
其中，处理单元1301可以用于支持电子设备1300执行上述步骤1101至步骤1105、步骤1106等，和/或用于本文所描述的技术的其他过程。处理单元四用于通过序列帧计算关键帧图像、对关键帧图像的前景区域和背景区域进行分割并分别进行处理的物理单元。
拍摄单元1302可以用于支持电子设备1300对目标对象进行拍摄、对目标场景进行预览拍摄等，和/或用于本文所描述的技术的其他过程。拍摄单元用于拍摄目标图像的物理单元，如镜头、图像传感器、图像信号处理器(image signal processor，ISP)等。
存储单元1303可以用于支持电子设备1300存储上述步骤1101至步骤1107中的产生的数据等，和/或用于本文所描述的技术的其他过程。存储单元指用于存储摄像头输出的序列帧图像、处理单元分割的前景图像和背景图像等信息的物理单元。
输出单元1304指的是输出前景背景区域合成结果等信息呈现给用户的物理单元。
IMU1305，用于采集摄像头在拍摄图像时对应的姿态参数，将该姿态参数发送给处理单元1301。
需要说明的是，上述方法实施例涉及的各步骤的所有相关内容均可以援引到对应功能模块的功能描述，在此不再赞述。
本实施例提供的电子设备，用于执行上述图像处理方法，因此可以达到与上述实现方法相同的效果。
在采用集成的单元的情况下，电子设备可以包括处理模块、存储模块和通信模块。
其中，处理模块可以用于对电子设备的动作进行控制管理，例如，可以用于支持电子设备执行上述处理单元1301、拍摄单元1302执行的步骤。存储模块可以用于支持电子设备执行上述存储单元1303执行的步骤，以及存储程序代码和数据等。通信模块，可以用于支持电子设备与其他设备的通信。
其中，处理模块可以是处理器或控制器。其可以实现或执行结合本申请公开内容所描 述的各种示例性的逻辑方框，模块和电路。处理器也可以是实现计算功能的组合，例如包含一个或多个微处理器组合，数字信号处理(digital signal processing,DSP)和微处理器的组合等等。存储模块可以是存储器。通信模块具体可以为射频电路、蓝牙芯片、Wi-Fi芯片等与其他电子设备交互的设备。
在一个实施例中，当处理模块为处理器，存储模块为存储器时，本实施例所涉及的电子设备可以为具有图1所示结构的设备。
本实施例还提供一种计算机存储介质，该计算机存储介质中存储有计算机指令，当该计算机指令在电子设备上运行时，使得电子设备执行上述相关方法步骤实现上述实施例中的图像处理方法。
本实施例还提供了一种计算机程序产品，当该计算机程序产品在计算机上运行时，使得计算机执行上述相关步骤，以实现上述实施例中的图像处理方法。
另外，本申请的实施例还提供一种装置，这个装置具体可以是芯片，组件或模块，该装置可包括相连的处理器和存储器；其中，存储器用于存储计算机执行指令，当装置运行时，处理器可执行存储器存储的计算机执行指令，以使芯片执行上述各方法实施例中的图像处理方法。
其中，本实施例提供的电子设备、计算机存储介质、计算机程序产品或芯片均用于执行上文所提供的对应的方法，因此，其所能达到的有益效果可参考上文所提供的对应的方法中的有益效果，此处不再赞述。
通过以上实施方式的描述，所属领域的技术人员可以了解到，为描述的方便和简洁，仅以上述各功能模块的划分进行举例说明，实际应用中，可以根据需要而将上述功能分配由不同的功能模块完成，即将装置的内部结构划分成不同的功能模块，以完成以上描述的全部或者部分功能。
在本申请所提供的几个实施例中，应该理解到，所揭露的装置和方法，可以通过其它的方式实现。例如，以上所描述的装置实施例仅仅是示意性的，例如，模块或单元的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，例如多个单元或组件可以结合或者可以集成到另一个装置，或一些特征可以忽略，或不执行。
另一点，所显示或讨论的相互之间的藕合或直接藕合或通信连接可以是通过一些接口，装置或单元的间接藕合或通信连接，可以是电性，机械或其它的形式。作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是一个物理单元或多个物理单元，即可以位于一个地方，或者也可以分布到多个不同地方。可以根据实际的需要选择其中的部分或者全部单元来实现本实施例方案的目的。
另外，在本申请各个实施例中的各功能单元可以集成在一个处理单元中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个单元中。上述集成的单元既可以采用硬件的形式实现，也可以采用软件功能单元的形式实现。
集成的单元如果以软件功能单元的形式实现并作为独立的产品销售或使用时，可以存储在一个可读取存储介质中。基于这样的理解，本申请实施例的技术方案本质上或者说对现有技术做出贡献的部分或者该技术方案的全部或部分可以以软件产品的形式体现出来，该软件产品存储在一个存储介质中，包括若干指令用以使得一个设备(可以是单片机，芯片 等)或处理器(processor)执行本申请各个实施例方法的全部或部分步骤。而前述的存储介质包括:U盘、移动硬盘、只读存储器(read only memory,ROM)、随机存取存储器(random access memory,RAM)、磁碟或者光盘等各种可以存储程序代码的介质。
以上内容，仅为本申请的具体实施方式，但本申请的保护范围并不局限于此，任何熟悉本技术领域的技术人员在本申请揭露的技术范围内，可轻易想到变化或替换，都应涵盖在本申请的保护范围之内。因此，本申请的保护范围应以权利要求的保护范围为准。

Claims (25)

  1. 一种图像处理方法，应用于具有摄像头的电子设备，其特征在于，所述方法包括：
    显示从序列帧视频流中获取到的第一关键帧图像，所述序列帧视频流由所述摄像头对包括第一目标对象在内的目标场景进行拍摄得到；
    获取对所述第一关键帧图像进行前景背景分离处理后得到的第一前景图像和第一背景图像，所述第一前景图像包括所述第一目标对象；
    获取对所述第一前景图像上的所述第一目标对象进行前景图像处理后得到的第二前景图像，所述第二前景图像包括：前景图像处理后的第一目标对象；
    获取通过第二背景图像对所述第一背景图像进行背景修复处理后得到的第三背景图像，所述第二背景图像是第二关键帧图像上包括的背景图像，所述第二关键帧图像在获取所述第一关键帧图像之前通过所述摄像头拍摄所述目标场景得到；
    获取对所述第二前景图像和所述第三背景图像进行前景背景合成处理后得到的前景背景合成处理后的第一关键帧图像。
  2. 根据权利要求1所述的方法，其特征在于，所述显示从序列帧视频流中获取到的第一关键帧图像之前，所述方法还包括：
    显示从预览视频流中获取到的第二关键帧图像，所述预览视频流是在所述摄像头生成所述序列帧视频流之前对所述目标场景进行预览拍摄得到；
    获取从所述第二关键帧图像中分离出的所述第二背景图像。
  3. 根据权利要求1所述的方法，其特征在于，所述获取通过第二背景图像对所述第一背景图像进行背景修复处理之前，所述方法还包括：
    显示从所述序列帧视频流中获取到的所述第二关键帧图像；
    获取对所述第二关键帧图像进行前景背景分离处理后得到的第四背景图像；
    获取通过第五背景图像对所述第四背景图像进行者背景修复处理后得到的所述第二背景图像，所述第五背景图像从预览视频流中的第三关键帧图像中分离出，所述预览视频流在所述摄像头生成所述序列帧视频流之前对所述目标场景进行预览拍摄得到。
  4. 根据权利要求1所述的方法，其特征在于，所述显示从序列帧视频流中获取到的第一关键帧图像之前，所述方法还包括：
    在所述摄像头生成所述序列帧视频流之前，通过所述摄像头对所述目标场景进行连续拍摄，得到多个连续的背景图像；
    获取对所述多个连续的背景图像进行累积叠加处理后得到的所述第二背景图像。
  5. 根据权利要求1至4中任一项所述的方法，其特征在于，所述显示从序列帧视频流中获取到的第一关键帧图像，包括：
    显示从序列帧视频流中获取到的n个序列帧图像，所述n为大于或等于2的正整数；
    获取从所述n个序列帧图像中确定出的1个原始关键帧图像和n-1个相关帧图像；
    使用所述n-1个相关帧图像的像素信息对所述原始关键帧图像进行清晰度增强处理，确定清晰度增强处理后的原始关键帧图像作为所述第一关键帧图像。
  6. 根据权利要求5所述的方法，其特征在于，所述使用所述n-1个相关帧图像的像素信息对所述原始关键帧图像进行清晰度增强处理，包括：
    获取所述原始关键帧图像的图像点原始能量值；
    从所述n-1个相关帧图像中获取位于所述原始关键帧图像的前后各k个相关帧图像，所述k的取值小于或等于(n-1)÷2；
    获取通过所述原始关键帧图像的前后各k个相关帧图像的图像点能量值对所述原始关键帧图像的图像点原始能量值进行优化处理后得到的所述原始关键帧图像的图像点优化后能量值。
  7. 根据权利要求1至6中任一项所述的方法，其特征在于，所述获取对所述第二前景图像和所述第三背景图像进行前景背景合成处理后得到的前景背景合成处理后的第一关键帧图像之后，所述方法还包括：
    获取对所述前景背景合成处理后的第一关键帧图像进行前景边缘融合处理后得到的前景边缘融合处理后的第一关键帧图像；
    输出所述前景边缘融合处理后的第一关键帧图像。
  8. 根据权利要求1至7中任一项所述的方法，其特征在于，所述通过第二背景图像对所述第一背景图像进行背景修复处理，包括：
    根据所述第一背景图像和所述第二背景图像之间的特征点对应关系，获取所述第一背景图像和所述第二背景图像之间的帧间变化位置关系；
    获取根据所述帧间变化位置关系对所述第二背景图像进行透视变换后得到的变换后的第二背景图像；
    将所述变换后的第二背景图像拼接到所述第一背景图像上，得到所述第三背景图像。
  9. 根据权利要求8所述的方法，其特征在于，所述根据所述第一背景图像和所述第二背景图像之间的特征点对应关系，获取所述第一背景图像和所述第二背景图像之间的帧间变化位置关系，包括：
    获取所述第一背景图像对应的当前帧姿态参数；
    根据所述第二背景图像对应的上一帧姿态参数和所述当前帧姿态参数，获取所述第一背景图像和所述第二背景图像之间的帧间初始位置关系；
    获取对所述第一背景图像和所述第二背景图像进行特征提取和特征匹配后得到的特征点对应关系；
    使用所述特征点对应关系对所述帧间初始位置关系进行优化，得到所述第一背景图像和所述第二背景图像之间的帧间变化位置关系。
  10. 根据权利要求8所述的方法，其特征在于，所述根据所述帧间变化位置关系对所述第二背景图像进行透视变换，包括：
    使用所述帧间变化位置关系获取所述第一背景图像和所述第二背景图像之间的透视变换矩阵；
    获取使用所述透视变换矩阵对所述第二背景图像进行透视变换后得到的变换后的第二背景图像。
  11. 根据权利要求1至7中任一项所述的方法，其特征在于，所述通过第二背景图像对所述第一背景图像进行背景修复处理，包括：
    从所述第二背景图像中分割出背景对象图像；
    当所述背景对象图像进入所述第一背景图像上的前景缺失区域时，使用所述背景对象图像对所述第一背景图像进行背景修补处理。
  12. 一种电子设备，其特征在于，包括:
    摄像头、一个或多个处理器；存储器；多个应用程序；以及一个或多个计算机程序，其中所述一个或多个计算机程序被存储在所述存储器中，所述一个或多个计算机程序包括指令，当所述指令被所述电子设备执行时，使得所述电子设备执行以下步骤:
    显示从序列帧视频流中获取到的第一关键帧图像，所述序列帧视频流由所述摄像头对包括第一目标对象在内的目标场景进行拍摄得到；
    获取对所述第一关键帧图像进行前景背景分离处理后得到的第一前景图像和第一背景图像，所述第一前景图像包括所述第一目标对象；
    获取对所述第一前景图像上的所述第一目标对象进行前景图像处理后得到的第二前景图像，所述第二前景图像包括：前景图像处理后的第一目标对象；
    获取通过第二背景图像对所述第一背景图像进行背景修复处理后得到的第三背景图像，所述第二背景图像是第二关键帧图像上包括的背景图像，所述第二关键帧图像在获取所述第一关键帧图像之前通过所述摄像头拍摄所述目标场景得到；
    获取对所述第二前景图像和所述第三背景图像进行前景背景合成处理后得到的前景背景合成处理后的第一关键帧图像。
  13. 根据权利要求12所述的电子设备，其特征在于，当所述指令被所述电子设备执行时，使得所述电子设备具体执行以下步骤：
    显示从序列帧视频流中获取到的第一关键帧图像之前，显示从预览视频流中获取到的第二关键帧图像，所述预览视频流是在所述摄像头生成所述序列帧视频流之前对所述目标场景进行预览拍摄得到；
    获取从所述第二关键帧图像中分离出的所述第二背景图像。
  14. 根据权利要求12所述的电子设备，其特征在于，当所述指令被所述电子设备执行时，使得所述电子设备具体执行以下步骤：
    获取通过第二背景图像对所述第一背景图像进行背景修复处理之前，显示从所述序列帧视频流中获取到的所述第二关键帧图像；
    获取对所述第二关键帧图像进行前景背景分离处理后得到的第四背景图像；
    获取通过第五背景图像对所述第四背景图像进行者背景修复处理后得到的所述第二背景图像，所述第五背景图像从预览视频流中的第三关键帧图像中分离出，所述预览视频流在所述摄像头生成所述序列帧视频流之前对所述目标场景进行预览拍摄得到。
  15. 根据权利要求12所述的电子设备，其特征在于，当所述指令被所述电子设备执行时，使得所述电子设备具体执行以下步骤：
    显示从序列帧视频流中获取到的第一关键帧图像之前，在所述摄像头生成所述序列帧视频流之前，通过所述摄像头对所述目标场景进行连续拍摄，得到多个连续的背景图像；
    获取对所述多个连续的背景图像进行累积叠加处理后得到的所述第二背景图像。
  16. 根据权利要求12至15中任一项所述的电子设备，其特征在于，当所述指令被所述电子设备执行时，使得所述电子设备具体执行以下步骤：
    显示从序列帧视频流中获取到的n个序列帧图像，所述n为大于或等于2的正整数；
    获取从所述n个序列帧图像中确定出的1个原始关键帧图像和n-1个相关帧图像；
    使用所述n-1个相关帧图像的像素信息对所述原始关键帧图像进行清晰度增强处理，确定清晰度增强处理后的原始关键帧图像作为所述第一关键帧图像。
  17. 根据权利要求16所述的电子设备，其特征在于，当所述指令被所述电子设备执行时，使得所述电子设备具体执行以下步骤：
    获取所述原始关键帧图像的图像点原始能量值；
    从所述n-1个相关帧图像中获取位于所述原始关键帧图像的前后各k个相关帧图像，所述k的取值小于或等于(n-1)÷2；
    获取通过所述原始关键帧图像的前后各k个相关帧图像的图像点能量值对所述原始关键帧图像的图像点原始能量值进行优化处理后得到的所述原始关键帧图像的图像点优化后能量值。
  18. 根据权利要求12至17中任一项所述的电子设备，其特征在于，当所述指令被所述电子设备执行时，使得所述电子设备具体执行以下步骤：
    获取对所述前景背景合成处理后的第一关键帧图像进行前景边缘融合处理后得到的前景边缘融合处理后的第一关键帧图像；
    输出所述前景边缘融合处理后的第一关键帧图像。
  19. 根据权利要求12至18中任一项所述的电子设备，其特征在于，当所述指令被所述电子设备执行时，使得所述电子设备具体执行以下步骤：
    根据所述第一背景图像和所述第二背景图像之间的特征点对应关系，获取所述第一背景图像和所述第二背景图像之间的帧间变化位置关系；
    获取根据所述帧间变化位置关系对所述第二背景图像进行透视变换后得到的变换后的第二背景图像；
    将所述变换后的第二背景图像拼接到所述第一背景图像上，得到所述第三背景图像。
  20. 根据权利要求19所述的电子设备，其特征在于，当所述指令被所述电子设备执行时，使得所述电子设备具体执行以下步骤：
    获取所述第一背景图像对应的当前帧姿态参数；
    根据所述第二背景图像对应的上一帧姿态参数和所述当前帧姿态参数，获取所述第一背景图像和所述第二背景图像之间的帧间初始位置关系；
    获取对所述第一背景图像和所述第二背景图像进行特征提取和特征匹配后得到的特征点对应关系；
    使用所述特征点对应关系对所述帧间初始位置关系进行优化，得到所述第一背景图像和所述第二背景图像之间的帧间变化位置关系。
  21. 根据权利要求19所述的电子设备，其特征在于，当所述指令被所述电子设备执行时，使得所述电子设备具体执行以下步骤：
    使用所述帧间变化位置关系获取所述第一背景图像和所述第二背景图像之间的透视变换矩阵；
    获取使用所述透视变换矩阵对所述第二背景图像进行透视变换后得到的变换后的第二 背景图像。
  22. 根据权利要求12至18中任一项所述的电子设备，其特征在于，当所述指令被所述电子设备执行时，使得所述电子设备具体执行以下步骤：
    从所述第二背景图像中分割出背景对象图像；
    当所述背景对象图像进入所述第一背景图像上的前景缺失区域时，使用所述背景对象图像对所述第一背景图像进行背景修补处理。
  23. 一种电子设备，包括存储器，一个或多个处理器，多个应用程序，以及一个或多个程序；其中所述一个或多个程序被存储在所述存储器中；其特征在于，所述一个或多个处理器在执行所述一个或多个程序时，使得所述电子设备实现如权利要求1至11中任一项所述的图像处理方法。
  24. 一种计算机存储介质，其特征在于，包括计算机指令，当所述计算机指令在电子设备上运行时，使得所述电子设备执行如权利要求1-11中任一项所述的图像处理方法。
  25. 一种计算机程序产品，其特征在于，当所述计算机程序产品在计算机上运行时，使得所述计算机执行如权利要求1-11中任一项所述的图像处理方法。
PCT/CN2020/105710 2019-08-22 2020-07-30 一种图像处理方法和电子设备 WO2021031819A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
US17/636,542 US20220301180A1 (en) 2019-08-22 2020-07-30 Image Processing Method and Electronic Device
EP20855503.7A EP4002272A4 (en) 2019-08-22 2020-07-30 IMAGE PROCESSING METHOD AND ELECTRONIC DEVICE

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201910779341.6 2019-08-22
CN201910779341.6A CN110675420B (zh) 2019-08-22 2019-08-22 一种图像处理方法和电子设备

Publications (1)

Publication Number Publication Date
WO2021031819A1 true WO2021031819A1 (zh) 2021-02-25

Family

ID=69075507

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/105710 WO2021031819A1 (zh) 2019-08-22 2020-07-30 一种图像处理方法和电子设备

Country Status (4)

Country Link
US (1) US20220301180A1 (zh)
EP (1) EP4002272A4 (zh)
CN (1) CN110675420B (zh)
WO (1) WO2021031819A1 (zh)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114037633A (zh) * 2021-11-18 2022-02-11 南京智谱科技有限公司 一种红外图像处理的方法及装置
WO2022262344A1 (zh) * 2021-06-15 2022-12-22 荣耀终端有限公司 一种拍摄方法及电子设备
WO2023010913A1 (zh) * 2021-07-31 2023-02-09 荣耀终端有限公司 一种图像处理方法及电子设备

Families Citing this family (20)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110675420B (zh) * 2019-08-22 2023-03-24 华为技术有限公司 一种图像处理方法和电子设备
CN111275696B (zh) * 2020-02-10 2023-09-15 腾讯医疗健康(深圳)有限公司 一种医学图像处理的方法、图像处理的方法及装置
CN111405199B (zh) * 2020-03-27 2022-11-01 维沃移动通信（杭州）有限公司 一种图像拍摄方法和电子设备
CN111476747A (zh) * 2020-03-31 2020-07-31 深圳奥比中光科技有限公司 一种消除屏下成像中鬼像的系统及方法
CN111494947B (zh) * 2020-04-20 2023-05-23 上海米哈游天命科技有限公司 摄像机移动轨迹确定方法、装置、电子设备及存储介质
CN111601033A (zh) * 2020-04-27 2020-08-28 北京小米松果电子有限公司 视频处理方法、装置及存储介质
CN111783781B (zh) * 2020-05-22 2024-04-05 深圳赛安特技术服务有限公司 基于产品协议字符识别的恶意条款识别方法、装置、设备
CN111680671A (zh) * 2020-08-13 2020-09-18 北京理工大学 一种基于光流的摄像方案自动生成方法
CN112184722B (zh) * 2020-09-15 2024-05-03 上海传英信息技术有限公司 图像处理方法、终端及计算机存储介质
CN113012031A (zh) * 2020-10-30 2021-06-22 北京达佳互联信息技术有限公司 图像处理方法和图像处理装置
CN112637674B (zh) * 2020-12-14 2022-10-25 深圳市纪元数码技术开发有限公司 视频转场效果的处理方法、装置、电子设备及存储介质
CN112598580B (zh) * 2020-12-29 2023-07-25 广州光锥元信息科技有限公司 提升人像照片清晰度的方法及装置
CN112651956B (zh) * 2020-12-30 2024-05-03 深圳云天励飞技术股份有限公司 图像处理方法、装置、电子设备及存储介质
CN115379105B (zh) * 2021-05-20 2024-02-27 北京字跳网络技术有限公司 视频拍摄方法、装置、电子设备和存储介质
CN113469041A (zh) * 2021-06-30 2021-10-01 北京市商汤科技开发有限公司 一种图像处理方法、装置、计算机设备和存储介质
CN113596556B (zh) * 2021-07-02 2023-07-21 咪咕互动娱乐有限公司 视频传输方法、服务器及存储介质
CN113435445A (zh) * 2021-07-05 2021-09-24 深圳市鹰硕技术有限公司 图像过优化自动纠正方法以及装置
CN113973190A (zh) * 2021-10-28 2022-01-25 联想(北京)有限公司 视频虚拟背景图像处理方法、装置及计算机设备
CN115209232B (zh) * 2022-09-14 2023-01-20 北京达佳互联信息技术有限公司 一种视频处理方法、装置、电子设备及存储介质
CN117177017B (zh) * 2023-10-27 2024-01-23 成方金融科技有限公司 一种视频处理方法、装置、设备及介质

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101324963A (zh) * 2008-07-24 2008-12-17 上海交通大学 基于静止图像的流体视频合成方法
CN105513030A (zh) * 2014-09-24 2016-04-20 联想(北京)有限公司 一种信息处理方法、装置及电子设备
CN108848325A (zh) * 2018-06-26 2018-11-20 蒋大武 一种基于自然图像抠像的视频合成方法
CN109361850A (zh) * 2018-09-28 2019-02-19 Oppo广东移动通信有限公司 图像处理方法、装置、终端设备及存储介质
CN110675420A (zh) * 2019-08-22 2020-01-10 华为技术有限公司 一种图像处理方法和电子设备

Family Cites Families (14)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7676081B2 (en) * 2005-06-17 2010-03-09 Microsoft Corporation Image segmentation of foreground from background layers
US8300890B1 (en) * 2007-01-29 2012-10-30 Intellivision Technologies Corporation Person/object image and screening
US8433101B2 (en) * 2008-07-31 2013-04-30 Samsung Electronics Co., Ltd. System and method for waving detection based on object trajectory
US20130182184A1 (en) * 2012-01-13 2013-07-18 Turgay Senlet Video background inpainting
US9247129B1 (en) * 2013-08-30 2016-01-26 A9.Com, Inc. Self-portrait enhancement techniques
CN103544685B (zh) * 2013-10-22 2017-01-04 华南理工大学 一种基于主体调整的图像构图美化方法及系统
US9979952B2 (en) * 2013-12-13 2018-05-22 Htc Corporation Method of creating a parallax video from a still image
US9454819B1 (en) * 2015-06-03 2016-09-27 The United States Of America As Represented By The Secretary Of The Air Force System and method for static and moving object detection
CN107025648A (zh) * 2017-03-20 2017-08-08 中国人民解放军空军工程大学 一种电路板故障红外图像自动检测方法
CN107038683B (zh) * 2017-03-27 2020-09-15 中国科学院自动化研究所 运动目标的全景成像方法
JP2019083364A (ja) * 2017-10-27 2019-05-30 キヤノン株式会社 画像処理装置、撮像装置および制御方法
CN110012209B (zh) * 2018-01-05 2020-08-14 Oppo广东移动通信有限公司 全景图像生成方法、装置、存储介质及电子设备
CN108665510B (zh) * 2018-05-14 2022-02-08 Oppo广东移动通信有限公司 连拍图像的渲染方法、装置、存储介质及终端
CN110047053A (zh) * 2019-04-26 2019-07-23 腾讯科技（深圳）有限公司 人像图片生成方法、装置和计算机设备

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101324963A (zh) * 2008-07-24 2008-12-17 上海交通大学 基于静止图像的流体视频合成方法
CN105513030A (zh) * 2014-09-24 2016-04-20 联想(北京)有限公司 一种信息处理方法、装置及电子设备
CN108848325A (zh) * 2018-06-26 2018-11-20 蒋大武 一种基于自然图像抠像的视频合成方法
CN109361850A (zh) * 2018-09-28 2019-02-19 Oppo广东移动通信有限公司 图像处理方法、装置、终端设备及存储介质
CN110675420A (zh) * 2019-08-22 2020-01-10 华为技术有限公司 一种图像处理方法和电子设备

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP4002272A4

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2022262344A1 (zh) * 2021-06-15 2022-12-22 荣耀终端有限公司 一种拍摄方法及电子设备
WO2023010913A1 (zh) * 2021-07-31 2023-02-09 荣耀终端有限公司 一种图像处理方法及电子设备
CN114037633A (zh) * 2021-11-18 2022-02-11 南京智谱科技有限公司 一种红外图像处理的方法及装置

Also Published As

Publication number Publication date
EP4002272A4 (en) 2022-10-19
CN110675420A (zh) 2020-01-10
CN110675420B (zh) 2023-03-24
EP4002272A1 (en) 2022-05-25
US20220301180A1 (en) 2022-09-22

Similar Documents

Publication Publication Date Title
WO2021031819A1 (zh) 一种图像处理方法和电子设备
WO2021238325A1 (zh) 一种图像处理方法及装置
CN108537845B (zh) 位姿确定方法、装置及存储介质
KR102114377B1 (ko) 전자 장치에 의해 촬영된 이미지들을 프리뷰하는 방법 및 이를 위한 전자 장치
WO2019223468A1 (zh) 相机姿态追踪方法、装置、设备及系统
US20200302670A1 (en) Image processing method, electronic device, and storage medium
WO2020192465A1 (zh) 一种三维对象重建方法和装置
CN111541907B (zh) 物品显示方法、装置、设备及存储介质
CN111726536A (zh) 视频生成方法、装置、存储介质及计算机设备
CN108712603B (zh) 一种图像处理方法及移动终端
JP2021524957A (ja) 画像処理方法およびその、装置、端末並びにコンピュータプログラム
CN108776822B (zh) 目标区域检测方法、装置、终端及存储介质
CN112287852B (zh) 人脸图像的处理方法、显示方法、装置及设备
CN110599593B (zh) 数据合成的方法、装置、设备及存储介质
CN112614057A (zh) 一种图像虚化处理方法及电子设备
US9137461B2 (en) Real-time camera view through drawn region for image capture
CN113426117B (zh) 虚拟相机拍摄参数获取方法、装置、电子设备和存储介质
CN105427369A (zh) 移动终端及其三维形象的生成方法
US20230334789A1 (en) Image Processing Method, Mobile Terminal, and Storage Medium
CN112581358A (zh) 图像处理模型的训练方法、图像处理方法及装置
WO2020238454A1 (zh) 拍摄方法及终端
CN113822798B (zh) 生成对抗网络训练方法及装置、电子设备和存储介质
CN110956571A (zh) 基于slam进行虚实融合的方法及电子设备
US20230347240A1 (en) Display method and apparatus of scene picture, terminal, and storage medium
CN110135329B (zh) 从视频中提取姿势的方法、装置、设备及存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20855503

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2020855503

Country of ref document: EP

Effective date: 20220218