CN113706421B

CN113706421B - Image processing method and device, electronic equipment and storage medium

Info

Publication number: CN113706421B
Application number: CN202111251393.XA
Authority: CN
Inventors: 王东
Original assignee: Shenzhen TetrasAI Technology Co Ltd
Current assignee: Shenzhen TetrasAI Technology Co Ltd
Priority date: 2021-10-27
Filing date: 2021-10-27
Publication date: 2022-02-22
Anticipated expiration: 2041-10-27
Also published as: CN113706421A

Abstract

The present disclosure relates to an image processing method and apparatus, an electronic device, and a storage medium, the method including: acquiring an ith frame in an image frame sequence, wherein i is an integer greater than 1, and the image frame sequence comprises a plurality of frame images; fusing the image frames of the previous i frames to obtain fused frames corresponding to the previous i frames; determining a target frame with highest definition in the previous i frames; and determining pixel difference values of the fusion frame corresponding to the previous i frame and the target frame, and performing image processing on the fusion frame and/or the target frame corresponding to the previous i frame based on the pixel difference values to obtain a processed image. The embodiment of the disclosure can improve the image definition of the obtained processed image.

Description

Image processing method and device, electronic equipment and storage medium

Technical Field

The present disclosure relates to the field of computer technologies, and in particular, to an image processing method and apparatus, an electronic device, and a storage medium.

Background

Image multiframe noise reduction technology and ghost removal technology are common technologies in image processing, and are generally applied to improving the quality of original images and improving the signal-to-noise ratio under a dark field, for example, in a night scene mode of a mobile phone end, multiframe images are continuously shot, and noise reduction is performed by utilizing the multiframe images to obtain clear night scene images. Because the acquisition moments of multi-frame images or multi-shot images are different, and the camera is in the relative motion process, all input images are difficult to ensure to have the same definition, when the quality of the images is improved by using the multi-frame images, if the original blurred images are mixed, the definition of a fusion result is reduced, and the phenomenon that the quality of the multi-frame image fusion result is not as good as that of a single-frame image is caused.

Means for avoiding the inconsistent image quality of multiple frames comprise the step of fixing a camera through a tripod to shoot or the step of using a better optical anti-shake device to stabilize the camera, but the methods greatly depend on hardware, the shooting scene is limited, and the method is difficult to enter daily application equipment of common users.

In addition, there are also many schemes for reducing noise or removing ghosts by algorithms, but the image sharpness of the noise-reduced image or the image with the ghosts removed by the conventional image processing algorithms needs to be further improved.

Disclosure of Invention

The present disclosure proposes an image processing technical solution.

In one aspect of the present disclosure, an image processing method is provided, including: acquiring an ith frame in an image frame sequence, wherein i is an integer greater than 1, and the image frame sequence comprises a plurality of frame images; fusing the image frames of the previous i frames to obtain fused frames corresponding to the previous i frames; determining a target frame with highest definition in the previous i frames; and determining pixel difference values of the fusion frame corresponding to the previous i frame and the target frame, and performing image processing on the fusion frame and/or the target frame corresponding to the previous i frame based on the pixel difference values to obtain a processed image.

In the embodiment of the disclosure, after an ith frame in an image frame sequence is acquired, a fusion frame corresponding to the ith frame is obtained by fusing image frames of the ith frame, then a target frame with the highest definition in the ith frame is determined, a pixel difference value between the fusion frame corresponding to the ith frame and the target frame is determined, and image processing is performed on the fusion frame and/or the target frame corresponding to the ith frame based on the pixel difference value, so as to obtain a processed image. Because the pixel difference value can embody fine difference information in the fusion frame and the target frame, image processing such as denoising and ghost removing is carried out based on the pixel difference value, the processed image quality is better, the definition is higher, and the user experience is higher.

In a possible implementation manner, the fusing the image frames of the previous i frames to obtain a fused frame corresponding to the previous i frame includes: acquiring a pre-stored fusion frame of a previous (i-1) frame; aligning the fusion frame corresponding to the ith frame and the previous (i-1) frame; and fusing the pixel points which have the alignment relation in the fusion frame corresponding to the ith frame and the previous (i-1) frame to obtain a fusion frame corresponding to the previous i frame, and storing the fusion frame corresponding to the previous i frame.

In the embodiment of the disclosure, the image frames are fused frame by frame, the fusion frame of the i-th frame and the pre-stored fusion frame of the previous (i-1) frame are fused during the fusion, the fusion result is the fusion frame of the previous i frame, and the fusion frame of the previous i frame is stored at the same time so as to be used during the fusion of the previous (i +1) frame, so that the image frames are fused frame by frame, multi-frame fusion can be performed without buffering all the frames, and memory resources occupied during the fusion of the image frames are saved.

In a possible implementation manner, the determining a target frame with the highest definition in the previous i frames includes: acquiring a definition value of an image frame with the highest definition in a pre-stored previous (i-1) frame; calculating a sharpness value of the ith frame; and comparing the definition value of the ith frame with the definition value of the image frame with the highest definition in the previous (i-1) frame to determine the image frame with the highest definition in the previous i frame.

In the embodiment of the disclosure, the target frame with the highest definition in the previous i frames is selected by performing frame selection on the image frames frame by frame, so that the frame selection can be performed without buffering all the frames, and the memory resource occupied when the target frame with the highest definition is selected is saved.

In one possible implementation, after determining a target frame with the highest definition in the previous i frames, the method further includes: under the condition that the target frame is the ith frame, recording the definition value of the ith frame as the highest definition value in the previous ith frame; and storing the ith frame as a target frame.

In the embodiment of the disclosure, when the clearest image frame in the previous i frames is the ith frame, the frame with the highest definition in the previous (i-1) frames stored in advance is updated, that is, the ith frame is stored as the target frame, so that when the previous (i +1) frame is selected, the image frame with the highest definition in the previous (i +1) frames can be determined according to the definition value of the (i +1) frame and the definition value of the target frame in the previous i frames, thereby implementing frame-by-frame selection of the image frame, and saving the memory resource occupied when the target frame with the highest definition is selected.

In a possible implementation manner, the determining pixel difference values of the fusion frame corresponding to the previous i frame and the target frame includes: aligning the fusion frame corresponding to the previous i frame with the target frame to obtain pixel points with an alignment relationship in the fusion frame corresponding to the previous i frame and the target frame; and determining the difference value of the pixel values of the pixel points with the alignment relation in the fusion frame corresponding to the previous i frames and the target frame after alignment as the pixel difference value.

In the embodiment of the present disclosure, a difference value of pixel values of pixel points having an alignment relationship in a fusion frame and a target frame corresponding to an i frame before alignment is determined, and the difference value is used as the pixel difference value, so as to accurately represent a difference degree between the fusion frame and the target frame.

In a possible implementation manner, performing image processing on the fusion frame and/or the target frame corresponding to the previous i frame based on the pixel difference value to obtain a processed image includes: determining a noise part and/or a dynamic fuzzy part in the pixel difference map based on the pixel difference map formed by the pixel difference values; subtracting the dynamic fuzzy part from the fusion frame corresponding to the previous i frames to obtain a de-ghosting image; and/or subtracting the noise part from the target frame to obtain a noise-reduced image.

In the embodiment of the present disclosure, a noise portion and/or a dynamic blur portion in a pixel difference map is determined based on the pixel difference value, and since a fused frame of a previous i frame is obtained by superimposing a plurality of frames of images, there may be a dynamic blur, and therefore, a de-ghosting image may be obtained by subtracting the dynamic blur portion from the fused frame corresponding to the previous i frame, and in addition, a target frame is a frame with the highest definition in the previous i frame, and therefore, a noise portion is subtracted from the target frame, and a noise-reduced image may be obtained. Because the pixel difference image can embody the fine difference information in the fusion frame and the target frame, the dynamic fuzzy part of the fusion frame relative to the target frame and the noise part of the target frame relative to the fusion frame can be accurately obtained, and the image quality of the de-ghost image obtained by subtracting the dynamic fuzzy part based on the pixel difference image from the fusion frame is better; the noise part obtained based on the pixel difference image is subtracted from the target frame, so that the obtained noise reduction image has a better noise reduction effect, the definition of the finally obtained image is improved, and the user experience is better.

In a possible implementation manner, the determining a noise part and/or a motion blur part in the pixel difference map based on the pixel difference map formed by the pixel difference values includes: obtaining a high-frequency signal in the pixel difference map as the noise part through a high-pass filter; and/or obtaining a low-frequency signal in the pixel difference map as the dynamic blurring part through a low-pass filter.

In the embodiment of the disclosure, the noise part in the image is considered to be a high-frequency signal, so that the high-frequency signal in the pixel difference map can be obtained by filtering through a high-pass filter, and the noise part in the pixel difference map can be accurately obtained; considering that the motion blur part in the image is often a low-frequency signal, the low-frequency signal in the pixel difference map can be obtained by filtering through a low-pass filter, and the motion blur part in the pixel difference map can be accurately obtained.

In a possible implementation manner, a total image frame number of the image frame sequence is n, and the determining the pixel difference value between the fusion frame corresponding to the previous i frames and the target frame includes: and under the condition that the ith frame is the nth frame, determining the pixel difference value between the fusion frame corresponding to the previous n frames and the target frame.

In the embodiment of the present disclosure, the length of the image sequence is determined, and then, when the ith frame is the nth frame, the pixel difference value between the fusion frame corresponding to the n previous frames and the target frame is determined, and subsequent image processing is performed, so that the operation of determining the pixel difference value and performing subsequent image processing only needs to be performed once, the amount of calculation for performing image processing on the entire image frame sequence is reduced, and processing resources are saved.

In one possible implementation, the sequence of image frames is a sequence of image frames taken in real time; the determining the pixel difference value between the fusion frame corresponding to the previous i frame and the target frame includes: under the condition that the ith frame meets the image processing condition, determining the pixel difference value of the fusion frame corresponding to the previous i frame and the target frame; the image processing conditions include: the ith frame is an image frame received in real time at present; and/or the ith frame is the last frame image frame in a preset single processing period.

In the embodiment of the present disclosure, when the image processing condition includes that the ith frame is an image frame received currently in real time, a scene with a high real-time requirement can be satisfied, and when the image processing condition includes that the ith frame is a last image frame in a preset single processing period, the processed image can be output according to a certain processing period, thereby further saving occupied memory resources.

In an aspect of the present disclosure, there is provided an image processing apparatus including: the device comprises an acquisition unit, a processing unit and a processing unit, wherein the acquisition unit is used for acquiring the ith frame in an image frame sequence, i is an integer larger than 1, and the image frame sequence comprises a plurality of frame images; the fusion unit is used for fusing the image frames of the previous i frames to obtain fusion frames corresponding to the previous i frames; the target frame determining unit is used for determining a target frame with the highest definition in the previous i frames; and the image processing unit is used for determining the pixel difference value between the fusion frame corresponding to the previous i frame and the target frame, and performing image processing on the fusion frame and/or the target frame corresponding to the previous i frame based on the pixel difference value to obtain a processed image.

In a possible implementation manner, the fusion unit is configured to obtain a pre-stored fusion frame of a previous (i-1) frame; aligning the fusion frame corresponding to the ith frame and the previous (i-1) frame; and fusing the pixel points which have the alignment relation in the fusion frame corresponding to the ith frame and the previous (i-1) frame to obtain a fusion frame corresponding to the previous i frame, and storing the fusion frame corresponding to the previous i frame.

In a possible implementation manner, the target frame determination unit is configured to obtain a previously stored sharpness value of a highest-sharpness image frame in a previous (i-1) frame; calculating a sharpness value of the ith frame; and comparing the definition value of the ith frame with the definition value of the image frame with the highest definition in the previous (i-1) frame to determine the image frame with the highest definition in the previous i frame.

In one possible implementation, the apparatus further includes: a recording unit, configured to record, when the target frame is the ith frame, a sharpness value of the ith frame as a highest sharpness value in a previous i frame; and storing the ith frame as a target frame.

In a possible implementation manner, the image processing unit is configured to perform an alignment operation on the fusion frame corresponding to the previous i frame and the target frame to obtain a pixel point having an alignment relationship in the fusion frame corresponding to the previous i frame and the target frame; and determining the difference value of the pixel values of the pixel points with the alignment relation in the fusion frame corresponding to the previous i frames and the target frame after alignment as the pixel difference value.

In a possible implementation manner, the image processing unit is configured to determine a noise part and/or a motion blur part in a pixel difference map based on the pixel difference map formed by the pixel difference values; subtracting the dynamic fuzzy part from the fusion frame corresponding to the previous i frames to obtain a de-ghosting image; and/or subtracting the noise part from the target frame to obtain a noise-reduced image.

In a possible implementation manner, the image processing unit is configured to obtain a high-frequency signal in the pixel difference map as the noise portion through a high-pass filter; and/or obtaining a low-frequency signal in the pixel difference map as the dynamic blurring part through a low-pass filter.

In a possible implementation manner, the total number of image frames of the image frame sequence is n, and the image processing unit is configured to determine, when the ith frame is the nth frame, a pixel difference value between a fusion frame corresponding to the n previous frames and the target frame.

In one possible implementation, the sequence of image frames is a sequence of image frames taken in real time; the image processing unit is used for determining pixel difference values of a fusion frame corresponding to the previous i frame and the target frame under the condition that the ith frame meets an image processing condition; the image processing conditions include: the ith frame is an image frame received in real time at present; and/or the ith frame is the last frame image frame in a preset single processing period.

In an aspect of the present disclosure, an electronic device is provided, including: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to invoke the memory-stored instructions to perform the above-described method.

In one aspect of the disclosure, a computer-readable storage medium is provided, having stored thereon computer program instructions, which when executed by a processor, implement the above-described method.

In the embodiment of the disclosure, after an ith frame in an image frame sequence is acquired, a fusion frame corresponding to the ith frame is obtained by fusing image frames of the ith frame, then a target frame with the highest definition in the ith frame is determined, a pixel difference value between the fusion frame corresponding to the ith frame and the target frame is determined, and image processing is performed on the fusion frame and/or the target frame corresponding to the ith frame based on the pixel difference value, so as to obtain a processed image. Because the pixel difference value can embody fine difference information in the fusion frame and the target frame, image processing such as denoising and/or ghost removing is carried out based on the pixel difference value, the processed image quality is better, the definition is higher, and the user experience is higher.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. Other features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments, which proceeds with reference to the accompanying drawings.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the disclosure.

Fig. 1 shows a flowchart of an image processing method of an embodiment of the present disclosure.

Fig. 2 illustrates a scene schematic diagram of determining a pixel difference map according to an embodiment of the disclosure.

Fig. 3 illustrates a scene schematic diagram of a decomposed pixel difference map according to an embodiment of the disclosure.

Fig. 4 shows an effect diagram of the processed image and the initial 1 st frame according to the embodiment of the disclosure.

Fig. 5 shows a flow chart of an image processing method of an embodiment of the present disclosure.

Fig. 6 shows a block diagram of an image processing apparatus of an embodiment of the present disclosure.

Fig. 7 shows a block diagram of an electronic device of an embodiment of the disclosure.

Fig. 8 shows a block diagram of another electronic device of an embodiment of the disclosure.

Detailed Description

Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. In the drawings, like reference numbers can indicate functionally identical or similar elements. While the various aspects of the embodiments are presented in drawings, the drawings are not necessarily drawn to scale unless specifically indicated.

The word "exemplary" is used exclusively herein to mean "serving as an example, embodiment, or illustration. Any embodiment described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments.

The term "and/or" herein is merely an association describing an associated object, meaning that three relationships may exist, e.g., a and/or B, may mean: a exists alone, A and B exist simultaneously, and B exists alone. In addition, the term "at least one" herein means any one of a plurality or any combination of at least two of a plurality, for example, including at least one of A, B, C, and may mean including any one or more elements selected from the group consisting of A, B and C.

Furthermore, in the following detailed description, numerous specific details are set forth in order to provide a better understanding of the present disclosure. It will be understood by those skilled in the art that the present disclosure may be practiced without some of these specific details. In some instances, methods, means, elements and circuits that are well known to those skilled in the art have not been described in detail so as not to obscure the present disclosure.

Image processing technologies such as image multi-frame noise reduction and ghost removal have a very wide application scene, for example, with the development of an intelligent terminal technology, the use of mobile terminal devices (such as a smart phone, a tablet computer, and the like) is more and more popular. The camera is built in most mobile terminal equipment, and along with the enhancement of mobile terminal processing capacity and the development of camera technology, the performance of built-in camera is more and more powerful, and the quality of shooting images is also more and more high. The intelligent mobile terminal brings convenience to daily photographing of people, and meanwhile, the requirements of people on the quality of photographed images are higher and higher, and particularly in a special scene of a night scene, the image quality is lower. At present, multiple frames of original images are usually collected to perform high-dynamic synthesis to improve the quality of night scene images, but noise and/or ghost images are introduced in the process of collecting the multiple frames of original images, so that the finally synthesized images are unclear. Therefore, it is an urgent problem to perform noise reduction and/or ghost removal on multi-frame images while preserving image details to the maximum.

In addition, for continuous image frames in a shooting scene, for example, for cameras in urban roads, and cameras in cells and shopping malls, continuous image frames are often shot, and for these continuous videos, certain noise and dynamic blur may also exist, so that the continuous image frames shot in the shooting scene also have the requirement of noise reduction and/or ghost removal, and these image frames are often transmitted to a user terminal in a video stream form for viewing by a user, and the requirement on real-time performance is high.

In the embodiment of the disclosure, an ith frame in an image frame sequence is acquired, wherein i is an integer greater than 1, and the image frame sequence comprises a plurality of frame images; fusing the image frames of the previous i frames to obtain fused frames corresponding to the previous i frames; determining a target frame with highest definition in the previous i frames; and determining pixel difference values of the fusion frame corresponding to the previous i frame and the target frame, and performing image processing on the fusion frame and/or the target frame corresponding to the previous i frame based on the pixel difference values to obtain a processed image. Because the pixel difference value can embody fine difference information in the fusion frame and the target frame, image processing such as denoising and/or ghost removing is carried out based on the pixel difference value, the processed image quality is better, the definition is higher, and the user experience is higher.

In one possible implementation, the image processing method may be performed by an electronic device such as a terminal device or a server, the terminal device may be a User Equipment (UE), a mobile device, a User terminal, a cellular phone, a cordless phone, a Personal Digital Assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, or the like, and the method may be implemented by a processor calling a computer readable instruction stored in a memory.

For convenience of description, in one or more embodiments of the present specification, an execution subject of the image processing method may be an image processing apparatus, and hereinafter, an embodiment of the method will be described with an example in which the execution subject is the image processing apparatus. It is to be understood that the method is carried out by the image processing apparatus as an exemplary illustration only, and should not be construed as limiting the method.

Fig. 1 illustrates a flowchart of an image processing method according to an embodiment of the present disclosure, which includes, as illustrated in fig. 1: in S11, an ith frame in an image frame sequence is acquired, where i is an integer greater than 1, and the image frame sequence includes a plurality of frame images.

The image frame sequence includes a plurality of frames, and the image frame sequence may be embodied in a locally stored video file or may be an image frame sequence in a video stream acquired in real time.

In an example, in a case that an execution subject of the image processing method is a terminal device (e.g., a mobile phone, a tablet computer), the image frame sequence may be a plurality of image frames that are continuously captured by an image capture module of the terminal device in a photographing mode, and is used for generating a frame of noise reduction picture for output, where an application scene of the example is commonly found in a night-view photographing mode of the terminal device; or, the video file may be a video file shot by an image capture module of the terminal device in a video recording mode, or an image frame sequence captured by the image capture module of the terminal device in real time.

In another example, in a case where the execution subject of the image processing method is a server, the image frame sequence may be an image frame sequence acquired by a remote image acquisition device (e.g., a road camera, a mall camera), for example, an image frame sequence acquired by the remote image acquisition device in real time, or a video file stored in a storage medium after being acquired by the remote image acquisition device.

For a video file stored in a storage medium, image frames can be sequentially read from the 1 st frame of the file, and then the ith frame is the ith image frame in the read video file. For the image frames transmitted in the form of a video stream, the image frames in the video stream may be read in real time, and for the noise reduction of the video stream, the noise reduction may be performed on the video stream in an iterative loop according to a noise reduction period, for example, one noise reduction period may be 10 frames. After the processed image is obtained, a next noise reduction period can be started, in the iteration of the next noise reduction period, the value of i of the image frame to be subjected to noise reduction is reset to zero, then the image frames of the 1 st frame, the 2 nd frame, the 3 rd frame, … … the ith frame, … … the nth frame are sequentially received, the ith frame is the ith image frame received in the noise reduction period, n is the total number of the image frames in one noise reduction period, and i and n are integers more than 1.

In S12, the image frames of the previous i frames are fused to obtain a fused frame corresponding to the previous i frame.

There are various ways to fuse the previous i frame image frames, and in one possible implementation, i image frames of the previous i frame may be directly read and then fused to obtain a fused frame.

In addition, the image frames may also be fused frame by frame, and in a possible implementation manner, specifically, the fusing the image frame of the previous i frame to obtain a fused frame corresponding to the previous i frame includes: acquiring a pre-stored fusion frame of a previous (i-1) frame; aligning the fusion frame corresponding to the ith frame and the previous (i-1) frame; and fusing the pixel points which have the alignment relation in the fusion frame corresponding to the ith frame and the previous (i-1) frame to obtain a fusion frame corresponding to the previous i frame, and storing the fusion frame corresponding to the previous i frame.

In this implementation, the fusion frame corresponding to the previous (i-1) frame is obtained by fusing the previous (i-1) frame, and when i =2, that is, when the previous (i-1) frame is the 1 st frame, the fusion frame corresponding to the previous (i-1) frame is the 1 st frame itself.

In the embodiment of the present disclosure, in order to save occupied memory resources, it is not necessary to obtain all image frames in an image frame sequence and then fuse all image frames, that is, when a previous (i-1) frame is fused, a fused frame corresponding to the previous (i-1) frame is obtained, and when an ith frame is fused, only the fused frame corresponding to the ith frame and the fused frame corresponding to the previous (i-1) frame are fused, so that a fused frame corresponding to the previous i frame can be obtained.

In the process of aligning two image frames, one image frame can be kept still as a reference, the other image frame is superposed on the previous image frame, and then the other image frame is subjected to rotation and translation operation, so that pixel points representing the same thing in the two image frames are overlapped as much as possible, and the image frames are aligned. For example, keeping the ith frame still, superimposing the fusion frame corresponding to the previous (i-1) frame onto the ith frame, and then performing operations such as rotation and translation on the fusion frame, so that the pixels representing the same thing in the fusion frame corresponding to the ith frame and the previous (i-1) frame are overlapped as much as possible, that is, the alignment of the image frames is realized.

After alignment, the coincident pixel points in the fusion frame corresponding to the ith frame and the previous (i-1) frame are the pixel points with the alignment relationship, and then the pixel points with the alignment relationship in the fusion frame corresponding to the ith frame and the previous (i-1) frame are fused to obtain the fusion frame corresponding to the previous i frame.

After the fusion frame corresponding to the ith frame is obtained, the fusion frame corresponding to the ith frame can be stored, the fusion frame is a single-frame video, and the storage space occupied by the fusion frame is the storage space of the single-frame video, so that the occupied memory resource can be saved.

The image frames may be fused by aligning two image frames and then averaging pixel values having an alignment relationship, and the specific fusion process may refer to a possible implementation manner provided in the present disclosure, which is not described herein again.

In S13, the target frame with the highest definition in the previous i frames is determined.

The information level of the image frame may be determined in various ways, and in one possible implementation, the definition of the image may be determined based on the sharpness value of the image, or in one possible implementation, the definition of the image may also be determined based on the intensity of a high-frequency signal in the image. The sharpness value is an index that characterizes the sharpness of an image and the sharpness of the edges of the image. The higher the sharpness value of the image, the higher the contrast of the details on the image, and the clearer the appearance. For example, in the case of a high sharpness value, not only the wrinkles and spots of the face on the screen are clearer, but also the swelling or sinking of the facial muscles can be lifelike. For dark or black lines in the image or places with sudden changes of the black and white image, under the condition of higher sharpness value, the edges of the lines or the joints of the sudden changes of the black and white image are sharper, and the whole picture is clearer. Therefore, the higher the sharpness value, the higher the sharpness of the image tends to be.

The sharpness value is used to indicate sharpness of the image frame, and may be, for example, the sharpness value described above, or may also be an intensity value of a high-frequency signal, and so on, which is not limited by the present disclosure.

The highest definition value in the previous (i-1) frame is the highest definition value in the plurality of image frames of the previous (i-1) frame, and if the highest definition value in the plurality of image frames of the previous (i-1) frame is smaller than the definition value of the ith frame, the ith frame is taken as the target frame with the highest definition value in the previous i frame; and if the highest definition value in a plurality of image frames of the previous (i-1) frame is larger than the definition value of the ith frame, keeping the highest definition value unchanged, and still taking the image frame with the highest definition value determined before as the target frame with the highest definition value in the previous i frame.

In the embodiment of the present disclosure, in order to save occupied memory resources, instead of determining the sharpness values of all image frames after acquiring all image frames in an image frame sequence, a highest sharpness value is recorded in the memory, and the highest sharpness value in the previous i frames can be determined by comparing the highest sharpness value with the sharpness value of the current i frame. In the process, the memory only needs to record the highest definition value in the previous (i-1) frame, the current ith frame image frame and the definition value thereof, and does not need to load all the image frames in the image sequence, thereby saving the occupied memory resource.

In one possible implementation, after determining the target frame with the highest sharpness value in the previous i frames, the method further includes: under the condition that the target frame is the ith frame, recording the definition value of the ith frame as the highest definition value in the previous ith frame; and storing the ith frame as a target frame. Therefore, the highest sharpness value in the previous (i +1) frame can be obtained according to the highest sharpness value in the recorded previous i frame and the sharpness value of the (i +1) th frame. In the process, the memory only needs to record the highest sharpness value in the previous i frames, the current (i +1) th frame image frame and the sharpness value thereof, and all the image frames in the image sequence are not required to be loaded, so that the occupied memory resource can be saved.

In S14, determining a pixel difference value between the fused frame corresponding to the previous i frame and the target frame, and performing image processing on the fused frame and/or the target frame corresponding to the previous i frame based on the pixel difference value to obtain a processed image.

The pixel difference value is used for representing the difference degree between the pixels, and specifically may be the difference of the pixel values, or may also be the square of the difference of the pixel values, or may also be a normalized value of the difference of the pixel values.

After the pixel difference value is obtained, image processing may be performed based on the pixel difference value, where the image processing includes noise reduction processing and/or de-ghosting processing. For a specific implementation process, reference may be made to possible implementation manners provided by the present disclosure, and details are not described here.

The obtained processed image can be displayed in time through the display equipment so as to meet scenes with requirements on real-time performance, such as scenes of image pictures acquired and displayed in real time by the image acquisition equipment; alternatively, the resulting processed image may be persistently stored by a storage medium for subsequent use.

In a possible implementation manner, a total image frame number of the image frame sequence is n, and the determining of the pixel difference values of the fusion frame corresponding to the previous i frames and the target frame includes; and under the condition that the ith frame is the nth frame, determining the pixel difference value between the fusion frame corresponding to the previous n frames and the target frame.

In this implementation, the length of the image sequence is determined, for example, the image sequence is a locally stored video file, and if a processed image needs to be output to the video file, then, if the ith frame is the last frame (nth frame) in the video file, the pixel difference value between the fusion frame corresponding to the previous i frame and the target frame may be determined, and the subsequent operation may be performed. For another example, for a plurality of image frames continuously captured in the photographing mode, the last frame of the plurality of image frames continuously captured corresponds to the image processing condition. Then, when the ith frame is the last frame of the plurality of continuously shot image frames, the pixel difference value between the fusion frame corresponding to the previous i frame and the target frame can be determined, and the subsequent operation is performed.

In the embodiment of the present disclosure, the length of the image sequence is determined, and then, when the ith frame is the nth frame, the pixel difference value between the fusion frame corresponding to the n previous frames and the target frame is determined, and the subsequent image processing is performed, so that the operation of determining the pixel difference value and performing the image processing in step S14 only needs to be performed once, the amount of calculation for performing the image processing on the entire image frame sequence is reduced, and the processing resource is saved.

In this implementation, the image frame sequence is an image frame sequence captured in real time, for example, an image frame sequence in a video stream acquired in real time, and then, in order to implement real-time noise reduction and/or ghost removal on the image frame sequence, so that a user can see a clearer image frame in real time, the ith frame may be each image frame received currently in real time, and thus, for each image frame, an operation of calculating a pixel difference value to obtain a processed image can be performed, and the user can see the processed clearer image frame in real time.

The implementation mode is particularly suitable for scenes with high real-time requirements on the pictures acquired by the camera, for example, continuous image frames can be often shot for the camera in the road and the cameras in a community and a market, certain noise and dynamic blurring can exist for the continuous videos, in order to enable a user to see the noise reduction pictures of the acquired images in real time, image processing operation can be executed for each image frame to obtain processed images, and then the processed images are output through the display equipment, so that the user can conveniently check the processed images.

In addition, in order to further save occupied memory resources, the processed image may be output according to a certain processing period, that is, the image processing operation conforming to the image processing condition is not performed on each image frame, and the processing period may be defined by a user, for example, the operation of determining the pixel difference value to obtain the processed image may be performed every 5 frames, or the operation of determining the pixel difference value to obtain the processed image may be performed every 10 frames. That is, the ith frame is the last frame image frame in a preset single processing period, that is, the image frame meeting the image processing condition.

In one possible implementation, after determining a target frame with the highest definition in the previous i frames, the method further includes: and under the condition that the ith frame is not the last frame in the image frame sequence, saving the fusion frame corresponding to the previous ith frame, updating the value of i to i +1, and re-executing the processing for determining the ith frame and the subsequent frames in the image frame sequence to be subjected to noise reduction.

And under the condition that the image frame sequence is a plurality of continuously shot image frames in the shooting mode, if the ith frame is the last frame of the plurality of continuously shot image frames, the noise reduction process can be ended after the processed image is output, and if the ith frame is not the last frame of the plurality of continuously shot image frames, the value of i is updated to i +1, and the next iteration process is carried out.

For another example, in the case that the image frame sequence is a video file stored in the storage medium, if the ith frame is the last frame of the video file, the noise reduction process may be ended after the processed image is output. And if the ith frame is not the last frame of the video file, updating the value of i to i +1, and carrying out the next iteration process.

For another example, when the image frame sequence is a video stream acquired in real time, the image processing condition is the current image frame received in real time, and if the ith frame is not the last frame in the image frame sequence to be denoised, the value of i may be updated to i +1, and the next iteration process is performed.

For another example, when the image frame sequence is a video stream acquired in real time, the image processing condition is that the ith frame is the last frame image frame in a preset single processing period, and if the ith frame is not the last frame in the image frame sequence to be denoised, the value of i may be updated to i +1, and the next iteration process is performed.

For the image frames received in real time, the image frame sequence may be the image frames received in real time in a single processing period, that is, for the image frames received in real time, the image may be performed according to a preset processing period, the length of the single processing period is the preset number of frames, and when the processing period is finished, the i value is zeroed, and the noise reduction processing of the next noise reduction period is started. Therefore, under the condition that the pictures in the image frames change, the situation that the image frames with different frames are overlapped can be reduced, and a good noise reduction effect can still be obtained after the pictures in the image frames received in real time change.

The process of aligning the fusion frame corresponding to the i-th frame with the target frame may refer to the process of aligning the fusion frame corresponding to the i-th frame with the fusion frame corresponding to the (i-1) th frame, which is not described herein again. And in the fusion frame and the target frame corresponding to the aligned previous i frames, the pixel points have an alignment relation.

The pixel difference value is used for representing the difference degree between the fusion frame and the target frame, and the difference degree is embodied in the difference between two pixel points representing the same thing in the two frames of images, namely the difference between the pixel points with the alignment relation. Then, the difference value of the pixel values of the pixel points having the alignment relationship in the fusion frame and the target frame corresponding to the previous i frames after alignment is determined to be used as the pixel difference value, so as to accurately represent the difference degree between the fusion frame and the target frame.

In a possible implementation manner, denoising a fusion frame corresponding to the previous i frame based on the pixel difference value to obtain a processed image includes: determining a noise part and/or a dynamic fuzzy part in the pixel difference map based on the pixel difference map formed by the pixel difference values; and subtracting the dynamic fuzzy part from the fusion frame corresponding to the previous i frame to obtain a de-ghosting image, and/or subtracting the noise part from the target frame to obtain a de-noised image.

The pixel difference value is obtained from a difference value between the fusion frame of the previous i frames and the target frame, and therefore, the pixel difference value has image coordinates, for example, a difference value between two points of which the image coordinates after the fusion frame and the target frame are aligned are (1, 1) is a, and then a is a pixel value of a point of which the image coordinates in the pixel difference map are (1, 1). Then, the coordinates of the difference value in the pixel difference map are the coordinates of the two pixels with alignment relationship for determining the difference value.

Fig. 2 illustrates a scene schematic diagram of determining a pixel difference map according to an embodiment of the disclosure. As shown in fig. 2, the images of the previous i frames (1-i frames) are fused to obtain a fused frame of the previous i frames, then the fused frame is differentiated from the target frame with the highest sharpness value to obtain a pixel difference value, and then a pixel difference map is constructed from the pixel difference value.

The pixel difference image can have a noise part and/or a dynamic fuzzy part, image noise refers to unnecessary or redundant interference information existing in image data, the noise is often represented as isolated pixel points on the image, the noise is often caused by instability of a photosensitive chip of image acquisition equipment, and the common noise of the image basically has the following four types: gaussian noise, poisson noise, multiplicative noise, salt and pepper noise. The motion blur may also be referred to as motion blur (motion blur), which is a blurred dragging trace caused by a fast moving object in the picture and may also be referred to as a ghost in the image. When a camera takes a picture, it not only represents a single instant picture, but due to technical limitations or artistic requirements, the picture represents a scene within a period of exposure time. As objects in a scene move, the image of the scene must represent the complete combination of all the positions of those objects within the exposure time (depending on the shutter speed), and in such pictures, moving objects will appear blurred or dragged.

For the noise part and/or the dynamic fuzzy part in the fusion frame, the noise part and/or the dynamic fuzzy part in the pixel difference map can be determined by determining the noise part and/or the dynamic fuzzy part in the pixel difference map, and the position coordinates of the noise part and the dynamic fuzzy part in the pixel difference map can be reserved. Fig. 3 illustrates a scene schematic diagram of a decomposed pixel difference map according to an embodiment of the disclosure. As shown in fig. 3, after the pixel difference map is decomposed into a noise part and a motion blur part, a noise distribution map and a motion blur distribution map can be obtained, and both the noise and the motion blur in the image will retain their original position information in the pixel difference map.

After the noise part and/or the dynamic blur part are/is obtained, the dynamic blur part can be subtracted from the fusion frame corresponding to the previous i frame to obtain a de-ghosted image, and specifically, the dynamic blur part at the position with the same image coordinate is subtracted from the fusion frame to obtain the de-ghosted image. In addition, the noise part may be subtracted from the target frame corresponding to the previous i frames to obtain a noise-reduced image, specifically, the noise part at the position with the same image coordinates may be subtracted from the target frame to obtain the noise-reduced image.

Fig. 4 shows an effect diagram of the processed image and the initial 1 st frame according to the embodiment of the disclosure. As shown in fig. 4, the effect of the processed image (noise reduction frame) provided by the present disclosure and the initial frame 1 is schematically shown, and it can be seen from this schematic diagram that the sharpness of the processed image is significantly higher than that of the frame 1.

The noise portion in the image is often a high-frequency signal, and therefore, the high-frequency signal in the pixel difference map may be obtained as the noise portion by filtering with a high-pass filter. The high-pass filter can pass high-frequency signals in the image and block signals of other frequencies, so that a noise part in the image can be accurately obtained. The high frequency signal threshold used when the high pass filter specifically filters the high frequency signal can be determined by those skilled in the art according to practical situations, and the disclosure does not limit this.

The motion blur part in the image is often a low-frequency signal, so the low-frequency signal in the pixel difference map can be obtained by filtering through a low-pass filter as the motion blur part. The low-pass filter can pass low-frequency signals in the image and block signals of other frequencies, so that dynamic blurring parts in the image can be accurately obtained. The threshold of the low frequency signal used when the low pass filter specifically filters the low frequency signal can be determined by those skilled in the art according to practical situations, and the disclosure does not limit this.

For the parts not described in detail in this embodiment, reference may be made to the foregoing description, and details are not repeated here. In the implementation mode, the image frame sequence is a plurality of image frames which are continuously shot in a shooting mode, the plurality of image frames are denoised, and a processed image is output. The implementation flowchart of this implementation is shown in fig. 5, and fig. 5 shows a flowchart of an image processing method according to an embodiment of the present disclosure, which is described in detail below.

After receiving an input ith frame image frame, firstly judging whether the frame is a 1 st frame, if the frame is the 1 st frame, recording the 1 st frame as a fusion frame corresponding to a previous i (i =1) frame and a target frame with the highest sharpness value, and recording the sharpness value of the 1 st frame as the highest sharpness value in the previous i (i =1) frame; if the frame is not the 1 st frame, aligning the image frame with the fusion frame, calculating a sharpness value, judging whether the sharpness value of the i-th frame is higher than the highest sharpness value in the pre-recorded previous (i-1) frames, wherein the previous (i-1) frames are the 1 st to (i-1) frames before the i-th frame, for example, under the condition that i =10, the previous (i-1) frames are the 1 st frame, the 2 nd frame, the 3 rd frame, the 4 th frame, the 5 th frame, the 6 th frame, the 7 th frame, the 8 th frame and the 9 th frame, if so, updating the highest sharpness value to the sharpness value of the i-th frame, updating the target frame with the highest sharpness value, and if not, not updating; then fusing the aligned ith frame with a fused frame corresponding to the previous (i-1) frame to obtain a fused frame corresponding to the previous i frame; judging whether the ith frame is the last frame or not, if not, saving the fusion frame corresponding to the previous i frame and continuing to execute the process on the (i +1) th frame; if the frame is the last frame, calculating the pixel difference between the fused frame corresponding to the previous i frame and the target frame, decomposing the pixel difference into a noise part and a dynamic fuzzy part, subtracting the dynamic fuzzy part from the fused frame corresponding to the previous i frame to obtain a de-ghosting image, subtracting the noise part from the target frame to obtain a noise-reduced image, and outputting the de-ghosting image and the noise-reduced image as processed images.

It is understood that the above-mentioned method embodiments of the present disclosure can be combined with each other to form a combined embodiment without departing from the logic of the principle, which is limited by the space, and the detailed description of the present disclosure is omitted. Those skilled in the art will appreciate that in the above methods of the specific embodiments, the specific order of execution of the steps should be determined by their function and possibly their inherent logic.

In addition, the present disclosure also provides an image processing apparatus, an electronic device, a computer-readable storage medium, and a program, which can be used to implement any one of the image processing methods provided by the present disclosure, and the descriptions and corresponding descriptions of the corresponding technical solutions and the corresponding descriptions in the methods section are omitted for brevity.

Fig. 6 shows a block diagram of an image processing apparatus 60 according to an embodiment of the present disclosure, and as shown in fig. 6, the apparatus 60 includes: the acquiring unit 61 is configured to acquire an ith frame in an image frame sequence, where i is an integer greater than 1, and the image frame sequence includes multiple frame images; the fusion unit 62 is configured to fuse the image frames of the previous i frames to obtain fusion frames corresponding to the previous i frames; a target frame determining unit 63 that determines a target frame with the highest definition in the previous i frames; and the image processing unit 64 is configured to determine a pixel difference value between the fusion frame corresponding to the previous i frame and the target frame, and perform image processing on the fusion frame and/or the target frame corresponding to the previous i frame based on the pixel difference value to obtain a processed image.

In a possible implementation manner, the fusion unit 62 is configured to obtain a pre-stored fusion frame of a previous (i-1) frame; aligning the fusion frame corresponding to the ith frame and the previous (i-1) frame; and fusing the pixel points which have the alignment relation in the fusion frame corresponding to the ith frame and the previous (i-1) frame to obtain a fusion frame corresponding to the previous i frame, and storing the fusion frame corresponding to the previous i frame.

In a possible implementation manner, the target frame determining unit 63 is configured to obtain a previously stored sharpness value of a highest-sharpness image frame in a previous (i-1) frame; calculating a sharpness value of the ith frame; and comparing the definition value of the ith frame with the definition value of the image frame with the highest definition in the previous (i-1) frame to determine the image frame with the highest definition in the previous i frame.

In a possible implementation manner, the image processing unit 64 is configured to perform an alignment operation on the fusion frame corresponding to the previous i frame and the target frame to obtain a pixel point having an alignment relationship in the fusion frame corresponding to the previous i frame and the target frame; and determining the difference value of the pixel values of the pixel points with the alignment relation in the fusion frame corresponding to the previous i frames and the target frame after alignment as the pixel difference value.

In a possible implementation manner, the image processing unit 64 is configured to determine a noise portion and/or a motion blur portion in a pixel difference map based on the pixel difference map formed by the pixel difference values; subtracting the dynamic fuzzy part from the fusion frame corresponding to the previous i frames to obtain a de-ghosting image; and/or subtracting the noise part from the target frame to obtain a noise-reduced image.

In a possible implementation manner, the image processing unit 64 is configured to obtain a high-frequency signal in the pixel difference map as the noise portion through a high-pass filter; and/or obtaining a low-frequency signal in the pixel difference map as the dynamic blurring part through a low-pass filter.

In a possible implementation manner, the total image frame number of the image frame sequence is n, and the image processing unit 64 is configured to determine the pixel difference value between the fusion frame corresponding to the n previous frames and the target frame if the ith frame is the nth frame.

In one possible implementation, the sequence of image frames is a sequence of image frames taken in real time; the image processing unit 64 is configured to determine pixel difference values of the fusion frame corresponding to the i-th frame and the target frame when the i-th frame meets the image processing condition; the image processing conditions include: the ith frame is an image frame received in real time at present; and/or the ith frame is the last frame image frame in a preset single processing period.

In some embodiments, functions of or modules included in the apparatus provided in the embodiments of the present disclosure may be used to execute the method described in the above method embodiments, and specific implementation thereof may refer to the description of the above method embodiments, and for brevity, will not be described again here.

Embodiments of the present disclosure also provide a computer-readable storage medium having stored thereon computer program instructions, which when executed by a processor, implement the above-mentioned method. The computer readable storage medium may be a volatile or non-volatile computer readable storage medium.

An embodiment of the present disclosure further provides an electronic device, including: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to invoke the memory-stored instructions to perform the above-described method.

The disclosed embodiments also provide a computer program product comprising computer readable code or a non-transitory computer readable storage medium carrying computer readable code, which when run in a processor of an electronic device, the processor in the electronic device performs the above method.

The electronic device may be provided as a terminal, server, or other form of device.

Fig. 7 illustrates a block diagram of an electronic device 800 of an embodiment of the disclosure. For example, the electronic device 800 may be a User Equipment (UE), a mobile device, a User terminal, a cellular phone, a cordless phone, a Personal Digital Assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, or other terminal device.

Referring to fig. 7, electronic device 800 may include one or more of the following components: a processing component 802, a memory 804, a power component 806, a multimedia component 808, an audio component 810, an input/output (I/O) interface 812, a sensor component 814, and a communication component 816.

The processing component 802 generally controls overall operation of the electronic device 800, such as operations associated with display, telephone calls, data communications, camera operations, and recording operations. The processing components 802 may include one or more processors 820 to execute instructions to perform all or a portion of the steps of the methods described above. Further, the processing component 802 can include one or more modules that facilitate interaction between the processing component 802 and other components. For example, the processing component 802 can include a multimedia module to facilitate interaction between the multimedia component 808 and the processing component 802.

The memory 804 is configured to store various types of data to support operations at the electronic device 800. Examples of such data include instructions for any application or method operating on the electronic device 800, contact data, phonebook data, messages, pictures, videos, and so forth. The memory 804 may be implemented by any type or combination of volatile or non-volatile memory devices such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic or optical disks.

The power supply component 806 provides power to the various components of the electronic device 800. The power components 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the electronic device 800.

The multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and a user. In some embodiments, the screen may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive an input signal from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front facing camera and/or a rear facing camera. The front camera and/or the rear camera may receive external multimedia data when the electronic device 800 is in an operation mode, such as a shooting mode or a video mode. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

The audio component 810 is configured to output and/or input audio signals. For example, the audio component 810 includes a Microphone (MIC) configured to receive external audio signals when the electronic device 800 is in an operational mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signals may further be stored in the memory 804 or transmitted via the communication component 816. In some embodiments, audio component 810 also includes a speaker for outputting audio signals.

The I/O interface 812 provides an interface between the processing component 802 and peripheral interface modules, which may be keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to: a home button, a volume button, a start button, and a lock button.

The sensor assembly 814 includes one or more sensors for providing various aspects of state assessment for the electronic device 800. For example, the sensor assembly 814 may detect an open/closed state of the electronic device 800, the relative positioning of components, such as a display and keypad of the electronic device 800, the sensor assembly 814 may also detect a change in the position of the electronic device 800 or a component of the electronic device 800, the presence or absence of user contact with the electronic device 800, orientation or acceleration/deceleration of the electronic device 800, and a change in the temperature of the electronic device 800. Sensor assembly 814 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 814 may also include a light sensor, such as a Complementary Metal Oxide Semiconductor (CMOS) or Charge Coupled Device (CCD) image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

The communication component 816 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices. The electronic device 800 may access a wireless network based on a communication standard, such as a wireless network (Wi-Fi), a second generation mobile communication technology (2G), a third generation mobile communication technology (3G), a fourth generation mobile communication technology (4G), a long term evolution of universal mobile communication technology (LTE), a fifth generation mobile communication technology (5G), or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 further includes a Near Field Communication (NFC) module to facilitate short-range communications. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID) technology, infrared data association (IrDA) technology, Ultra Wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

In an exemplary embodiment, the electronic device 800 may be implemented by one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), controllers, micro-controllers, microprocessors or other electronic components for performing the above-described methods.

In an exemplary embodiment, a non-transitory computer-readable storage medium, such as the memory 804, is also provided that includes computer program instructions executable by the processor 820 of the electronic device 800 to perform the above-described methods.

The disclosure relates to the field of augmented reality, and aims to detect or identify relevant features, states and attributes of a target object by means of various visual correlation algorithms by acquiring image information of the target object in a real environment, so as to obtain an AR effect combining virtual and reality matched with specific applications. For example, the target object may relate to a face, a limb, a gesture, an action, etc. associated with a human body, or a marker, a marker associated with an object, or a sand table, a display area, a display item, etc. associated with a venue or a place. The vision-related algorithms may involve visual localization, SLAM, three-dimensional reconstruction, image registration, background segmentation, key point extraction and tracking of objects, pose or depth detection of objects, and the like. The specific application can not only relate to interactive scenes such as navigation, explanation, reconstruction, virtual effect superposition display and the like related to real scenes or articles, but also relate to special effect treatment related to people, such as interactive scenes such as makeup beautification, limb beautification, special effect display, virtual model display and the like. The detection or identification processing of the relevant characteristics, states and attributes of the target object can be realized through the convolutional neural network. The convolutional neural network is a network model obtained by performing model training based on a deep learning framework.

Fig. 8 illustrates a block diagram of another electronic device 1900 of an embodiment of the disclosure. For example, the electronic device 1900 may be provided as a server. Referring to fig. 8, electronic device 1900 includes a processing component 1922 further including one or more processors and memory resources, represented by memory 1932, for storing instructions, e.g., applications, executable by processing component 1922. The application programs stored in memory 1932 may include one or more modules that each correspond to a set of instructions. Further, the processing component 1922 is configured to execute instructions to perform the above-described method.

The electronic device 1900 may also include a power component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input/output (I/O) interface 1958. Electronic device 1900 may operate based on operations stored in memory 1932Operating systems, e.g. Microsoft Server operating System (Windows Server)^TM) Apple Inc. of the present application based on the graphic user interface operating System (Mac OS X)^TM) Multi-user, multi-process computer operating system (Unix)^TM) Free and open native code Unix-like operating System (Linux)^TM) Open native code Unix-like operating System (FreeBSD)^TM) Or the like.

In an exemplary embodiment, a non-transitory computer readable storage medium, such as the memory 1932, is also provided that includes computer program instructions executable by the processing component 1922 of the electronic device 1900 to perform the above-described methods.

The present disclosure may be systems, methods, and/or computer program products. The computer program product may include a computer-readable storage medium having computer-readable program instructions embodied thereon for causing a processor to implement various aspects of the present disclosure.

The computer readable storage medium may be a tangible device that can hold and store the instructions for use by the instruction execution device. The computer readable storage medium may be, for example, but is not limited to, an electronic memory device, a magnetic memory device, an optical memory device, an electromagnetic memory device, a semiconductor memory device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a Static Random Access Memory (SRAM), a portable compact disc read-only memory (CD-ROM), a Digital Versatile Disc (DVD), a memory stick, a floppy disk, a mechanical coding device, such as punch cards or in-groove projection structures having instructions stored thereon, and any suitable combination of the foregoing. Computer-readable storage media as used herein is not to be construed as transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., optical pulses through a fiber optic cable), or electrical signals transmitted through electrical wires.

The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to a respective computing/processing device, or to an external computer or external storage device via a network, such as the internet, a local area network, a wide area network, and/or a wireless network. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers and/or edge servers. The network adapter card or network interface in each computing/processing device receives computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in the respective computing/processing device.

The computer program instructions for carrying out operations of the present disclosure may be assembler instructions, Instruction Set Architecture (ISA) instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider). In some embodiments, the electronic circuitry that can execute the computer-readable program instructions implements aspects of the present disclosure by utilizing the state information of the computer-readable program instructions to personalize the electronic circuitry, such as a programmable logic circuit, a Field Programmable Gate Array (FPGA), or a Programmable Logic Array (PLA).

Various aspects of the present disclosure are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the disclosure. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer-readable program instructions.

These computer-readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks. These computer-readable program instructions may also be stored in a computer-readable storage medium that can direct a computer, programmable data processing apparatus, and/or other devices to function in a particular manner, such that the computer-readable medium storing the instructions comprises an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks.

The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer, other programmable apparatus or other devices implement the functions/acts specified in the flowchart and/or block diagram block or blocks.

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

The computer program product may be embodied in hardware, software or a combination thereof. In an alternative embodiment, the computer program product is embodied in a computer storage medium, and in another alternative embodiment, the computer program product is embodied in a Software product, such as a Software Development Kit (SDK), or the like.

Having described embodiments of the present disclosure, the foregoing description is intended to be exemplary, not exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen in order to best explain the principles of the embodiments, the practical application, or improvements made to the technology in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. An image processing method, comprising:

acquiring an ith frame in an image frame sequence, wherein i is an integer greater than 1, and the image frame sequence comprises a plurality of frame images;

fusing the image frames of the previous i frames to obtain fused frames corresponding to the previous i frames;

determining a target frame with highest definition in the previous i frames;

determining pixel difference values of the fusion frame corresponding to the previous i frame and the target frame, and performing image processing on the fusion frame and/or the target frame corresponding to the previous i frame based on the pixel difference values to obtain a processed image; the pixel difference value is used for representing the difference between the pixel points which have the alignment relation between the fusion frame and the target frame;

based on the pixel difference value, performing image processing on the fusion frame and/or the target frame corresponding to the previous i frame to obtain a processed image, including:

determining a noise part and/or a dynamic fuzzy part in the pixel difference map based on the pixel difference map formed by the pixel difference values;

subtracting the dynamic fuzzy part from the fusion frame corresponding to the previous i frames to obtain a de-ghosting image; and/or subtracting the noise part from the target frame to obtain a noise-reduced image.

2. The image processing method according to claim 1, wherein the fusing the image frames of the previous i frames to obtain a fused frame corresponding to the previous i frames comprises:

acquiring a pre-stored fusion frame of a previous (i-1) frame;

aligning the fusion frame corresponding to the ith frame and the previous (i-1) frame;

and fusing the pixel points which have the alignment relation in the fusion frame corresponding to the ith frame and the previous (i-1) frame to obtain a fusion frame corresponding to the previous i frame, and storing the fusion frame corresponding to the previous i frame.

3. The image processing method according to claim 1, wherein the determining a target frame with the highest definition in the previous i frames comprises:

acquiring a definition value of an image frame with the highest definition in a pre-stored previous (i-1) frame;

calculating a sharpness value of the ith frame;

and comparing the definition value of the ith frame with the definition value of the image frame with the highest definition in the previous (i-1) frame to determine the image frame with the highest definition in the previous i frame.

4. The image processing method according to claim 3, wherein after determining a target frame with the highest sharpness among the previous i frames, the method further comprises:

under the condition that the target frame is the ith frame, recording the definition value of the ith frame as the highest definition value in the previous ith frame;

and storing the ith frame as a target frame.

5. The image processing method according to claim 1, wherein the determining pixel difference values of the fused frame corresponding to the previous i frame and the target frame comprises:

aligning the fusion frame corresponding to the previous i frame with the target frame to obtain pixel points with an alignment relationship in the fusion frame corresponding to the previous i frame and the target frame;

and determining the difference value of the pixel values of the pixel points with the alignment relation in the fusion frame corresponding to the previous i frames and the target frame after alignment as the pixel difference value.

6. The method according to claim 1, wherein determining the noise portion and/or the motion blur portion in the pixel difference map based on the pixel difference map formed by the pixel difference values comprises:

obtaining a high-frequency signal in the pixel difference map as the noise part through a high-pass filter; and/or the presence of a gas in the gas,

and obtaining a low-frequency signal in the pixel difference map through a low-pass filter as the dynamic fuzzy part.

7. The method according to claim 1, wherein a total image frame number of the image frame sequence is n, and the determining the pixel difference values of the fused frame corresponding to the previous i frames and the target frame comprises:

and under the condition that the ith frame is the nth frame, determining the pixel difference value between the fusion frame corresponding to the previous n frames and the target frame.

8. The image processing method according to claim 1, wherein the image frame sequence is an image frame sequence photographed in real time;

the determining the pixel difference value between the fusion frame corresponding to the previous i frame and the target frame includes:

under the condition that the ith frame meets the image processing condition, determining the pixel difference value of the fusion frame corresponding to the previous i frame and the target frame;

the image processing conditions include: the ith frame is an image frame received in real time at present; and/or the ith frame is the last frame image frame in a preset single processing period.

9. An image processing apparatus characterized by comprising:

the device comprises an acquisition unit, a processing unit and a processing unit, wherein the acquisition unit is used for acquiring the ith frame in an image frame sequence, i is an integer larger than 1, and the image frame sequence comprises a plurality of frame images;

the fusion unit is used for fusing the image frames of the previous i frames to obtain fusion frames corresponding to the previous i frames;

the target frame determining unit is used for determining a target frame with the highest definition in the previous i frames;

the image processing unit is used for determining pixel difference values of the fusion frame corresponding to the previous i frame and the target frame, and performing image processing on the fusion frame and/or the target frame corresponding to the previous i frame based on the pixel difference values to obtain a processed image; the pixel difference value is used for representing the difference between the pixel points which have the alignment relation between the fusion frame and the target frame;

the image processing unit is used for determining a noise part and/or a dynamic fuzzy part in the pixel difference map based on the pixel difference map formed by the pixel difference values; subtracting the dynamic fuzzy part from the fusion frame corresponding to the previous i frames to obtain a de-ghosting image; and/or subtracting the noise part from the target frame to obtain a noise-reduced image.

10. An electronic device, comprising:

a processor;

a memory for storing processor-executable instructions;

wherein the processor is configured to invoke the memory-stored instructions to perform the image processing method of any of claims 1 to 8.

11. A computer-readable storage medium having computer program instructions stored thereon, which when executed by a processor implement the image processing method of any one of claims 1 to 8.