CN114372932A - Image processing method and computer program product - Google Patents
Image processing method and computer program product Download PDFInfo
- Publication number
- CN114372932A CN114372932A CN202111672323.1A CN202111672323A CN114372932A CN 114372932 A CN114372932 A CN 114372932A CN 202111672323 A CN202111672323 A CN 202111672323A CN 114372932 A CN114372932 A CN 114372932A
- Authority
- CN
- China
- Prior art keywords
- frame
- light spot
- image
- map
- point
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000004590 computer program Methods 0.000 title claims abstract description 22
- 238000003672 processing method Methods 0.000 title abstract description 6
- 238000001514 detection method Methods 0.000 claims abstract description 359
- 238000000034 method Methods 0.000 claims description 121
- 238000012545 processing Methods 0.000 claims description 82
- 230000004927 fusion Effects 0.000 claims description 74
- 230000008569 process Effects 0.000 claims description 38
- 238000010586 diagram Methods 0.000 claims description 33
- 230000011218 segmentation Effects 0.000 claims description 33
- 238000009499 grossing Methods 0.000 claims description 13
- 238000012935 Averaging Methods 0.000 claims description 8
- 238000003860 storage Methods 0.000 claims description 8
- 238000003709 image segmentation Methods 0.000 claims description 4
- 238000009877 rendering Methods 0.000 abstract description 53
- 230000000694 effects Effects 0.000 abstract description 24
- 238000005457 optimization Methods 0.000 description 12
- 230000006870 function Effects 0.000 description 9
- 238000005516 engineering process Methods 0.000 description 6
- 238000010606 normalization Methods 0.000 description 6
- 230000036961 partial effect Effects 0.000 description 6
- 238000013473 artificial intelligence Methods 0.000 description 5
- 238000004422 calculation algorithm Methods 0.000 description 5
- 238000003286 fusion draw glass process Methods 0.000 description 4
- 238000005070 sampling Methods 0.000 description 4
- 230000009471 action Effects 0.000 description 3
- 238000013136 deep learning model Methods 0.000 description 3
- 230000007246 mechanism Effects 0.000 description 3
- 230000002829 reductive effect Effects 0.000 description 3
- 238000011160 research Methods 0.000 description 3
- 238000013459 approach Methods 0.000 description 2
- 230000003796 beauty Effects 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 2
- 238000006243 chemical reaction Methods 0.000 description 2
- 238000013135 deep learning Methods 0.000 description 2
- 238000001914 filtration Methods 0.000 description 2
- 230000001965 increasing effect Effects 0.000 description 2
- 238000010801 machine learning Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000009467 reduction Effects 0.000 description 2
- 239000000126 substance Substances 0.000 description 2
- 230000001502 supplementing effect Effects 0.000 description 2
- 238000012795 verification Methods 0.000 description 2
- 241001139947 Mida Species 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000003190 augmentative effect Effects 0.000 description 1
- 230000006399 behavior Effects 0.000 description 1
- 238000009529 body temperature measurement Methods 0.000 description 1
- 239000003086 colorant Substances 0.000 description 1
- 239000002131 composite material Substances 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000012937 correction Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000002708 enhancing effect Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000000670 limiting effect Effects 0.000 description 1
- 239000000203 mixture Substances 0.000 description 1
- 229910052754 neon Inorganic materials 0.000 description 1
- GKAOGPIIYCISHV-UHFFFAOYSA-N neon atom Chemical compound [Ne] GKAOGPIIYCISHV-UHFFFAOYSA-N 0.000 description 1
- 238000007781 pre-processing Methods 0.000 description 1
- 230000011514 reflex Effects 0.000 description 1
- 230000002441 reversible effect Effects 0.000 description 1
- 238000004088 simulation Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000009469 supplementation Effects 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
- 230000004584 weight gain Effects 0.000 description 1
- 235000019786 weight gain Nutrition 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/70—Denoising; Smoothing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/04—Context-preserving transformations, e.g. by using an importance map
- G06T3/053—Detail-in-context presentations
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/90—Dynamic range modification of images or parts thereof
- G06T5/94—Dynamic range modification of images or parts thereof based on local image properties, e.g. for local contrast enhancement
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/11—Region-based segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/194—Segmentation; Edge detection involving foreground-background segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10016—Video; Image sequence
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20212—Image combination
- G06T2207/20216—Image averaging
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20212—Image combination
- G06T2207/20221—Image fusion; Image merging
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Image Processing (AREA)
Abstract
An embodiment of the application provides an image processing method and a computer program product, wherein the image processing method comprises the following steps: obtaining a light spot weight map of a k frame according to M historical frame light spot detection maps and a light spot detection map of a k frame, wherein the M historical frame light spot detection maps are obtained by respectively carrying out light spot detection on M original images, and the k frame light spot detection map is obtained by carrying out light spot detection on the k frame original image; and obtaining a target object light spot blurring rendering image of the kth frame according to the light spot point weight image of the kth frame and the original image of the kth frame. The technical scheme of this application embodiment can solve the facula scintillation problem that exists when image facula blurring effect is poor, video blurring for video blurring facula effect is more stable.
Description
Technical Field
The present application relates to the field of video image blurring processing, and in particular, embodiments of the present application relate to a method and a computer program product for image processing.
Background
With the update iteration of electronic devices (for example, mobile terminal devices including mobile phones and the like), the functional requirements of users on the camera shooting of the electronic devices are increasing and better. For example, a neon spot video blurring technique based on a mobile terminal (an electronic device such as a mobile phone) is an effect similar to a single lens reflex of a camera generated by a computer vision algorithm, and mainly includes a portrait segmentation technique and a simulated depth-of-field rendering technique, wherein a mainstream method of the portrait segmentation technique is mainly based on a depth learning model at present, and the simulated depth-of-field blurring rendering technique is mainly used for simulation by a method similar to filtering blurring.
At present, there are many effect problems in image blurring (e.g., blurring effect in portrait shooting mode) or video blurring (e.g., portrait video blurring effect) on electronic devices, such as blurred and unclear light spots on images after image blurring or severe flicker of background light spots generated in videos, which seriously affect user experience.
Therefore, how to improve the problems existing in image blurring or video blurring becomes a technical problem to be solved urgently.
Disclosure of Invention
An object of the embodiments of the present application is to provide an image processing method and a computer program product, which can solve the problem of blurred and blurred light spots on an image after image blurring or the problem of light spot flickering existing during video blurring by using the technical scheme provided by the embodiments of the present application, so that the video blurring effect is more natural and the blurring light spot effect is more stable.
In a first aspect, some embodiments of the present application provide a method of image processing, the method comprising: obtaining a light spot weight map of a k frame according to M historical frame light spot detection maps and a light spot detection map of a k frame, wherein the M historical frame light spot detection maps are obtained by respectively carrying out light spot detection on M original images, the k frame light spot detection map is obtained by carrying out light spot detection on a k frame original image, the M original image is a frame or continuous multi-frame image which is positioned in front of the k frame original image and is adjacent to the k frame original image, and M is an integer greater than or equal to 1; and obtaining a k frame target object light spot virtual rendering image according to the k frame light spot weight image and the k frame original image, wherein the k frame target object light spot virtual rendering image is the obtained target image.
According to the embodiment of the application, when the facula point weight map of each frame is obtained, the historical frame facula point detection graph before the frame (namely the k-th frame of the current processing frame) is considered, so that the problem that facula flicker generated by electronic equipment in a video blurring mode is serious can be solved.
In some embodiments, before the deriving a kth frame spot weight map from the M historical frame spot detection maps and the kth frame spot detection map, the method further comprises: carrying out image segmentation on any one of the M frame original image and the k frame original image to obtain a background segmentation image; obtaining a single-channel image based on the maximum value of the multiple channels of each pixel point in the background segmentation image; and acquiring a light spot detection graph corresponding to any frame of original image at least according to the value of each pixel point on the single-channel image.
Some embodiments of this application detect the facula point and then obtain facula point detection map through acquireing the single channel picture that corresponds with the background segmentation map, can promote the accuracy that detects the facula point that obtains.
In some embodiments, the obtaining a light spot detection map corresponding to any frame of the original image according to at least values of pixel points on a single-channel map includes: fuzzifying the single-channel image to obtain a fuzzy single-channel image; and if the value of any pixel point on the single-channel graph is larger than a first threshold value and the value of any pixel point on the fuzzy single-channel graph is confirmed to be smaller than or equal to a second threshold value, confirming that any pixel point belongs to a light spot, wherein the second threshold value is smaller than the first threshold value.
Some embodiments of the application utilize the attribute of the light spot (that is, the characteristic that the pixel value of the light spot located at the center of the light spot is large and the pixel value of the adjacent pixel point located at the center of the light spot is small) to comprehensively judge whether each pixel point belongs to the light spot through the constructed fuzzy single-channel image and the single-channel image, thereby improving the accuracy of the detected light spot.
In some embodiments, M is an integer greater than 1, where obtaining a kth frame flare point weight map from the M historical frame flare point detection maps and the kth frame flare point detection map includes: selecting X historical frame facula point detection maps from the M historical frame facula point detection maps, wherein the X historical frame facula point detection maps are adjacent to the k frame facula point detection map, and X is an integer which is greater than or equal to 1 and less than or equal to M; fusing the X historical frame light spot detection graphs and the k frame light spot detection graph to obtain a k frame light spot fusion graph; and obtaining the light spot point weight map of the kth frame according to the light spot point fusion map of the kth frame.
Some embodiments of the present application dynamically select at least a part of historical frame light spot detection maps from a plurality of historical frame light spot detection maps adjacent to a current processing frame (i.e., a k-th frame), and fuse the selected at least part of historical frame light spot detection maps with a current frame light spot detection map, thereby improving a light spot flicker phenomenon existing between a current frame original image and an adjacent frame original image in a video to be blurred.
In some embodiments, said selecting X historical frame spot detections from among said M historical frame spot detections comprises: and selecting the X historical frame light spot detection maps from the M historical frame light spot detection maps according to the distribution information of the light spots on different frame original images.
Some embodiments of the application determine the number of frames of the historical frames required for optimizing the flicker by using a light spot interframe difference detection mechanism, so that the problem of light spot flicker of adjacent frames can be solved by using as few historical frames as possible.
In some embodiments, said selecting X historical frame spot detections from among said M historical frame spot detections comprises: and if the difference value between the kth frame light spot detection diagram and the jth frame historical frame light spot detection diagram meets a difference threshold value, taking the jth frame to (k-1) th frame light spot detection diagrams as the X historical frame light spot detection diagrams, wherein the jth frame light spot detection diagram belongs to one frame of images in the M historical frame light spot detection diagrams.
According to the embodiments of the application, the difference of the light spots distributed on different frames is determined according to the current frame light spot detection graph and the historical frame light spot detection graph, and the accuracy of the obtained difference result is improved.
In some embodiments, if j takes on a value of
i is an integer greater than or equal to 0, wherein the confirming that a difference value between the kth frame facula point detection map and the jth frame history frame facula point detection map satisfies a difference threshold includes: and i, repeating the following process from 0 until the difference value obtained in the following way meets the difference threshold value: calculating the k frame facula point detection chart and the
Detecting absolute values of differences of corresponding pixel values of all pixel points on the image by using the frame light spot points to obtain ith differences; calculating the sum of the k frame light spot detection diagram and the k frame light spot detection diagram
The absolute value of the difference value of the pixel values corresponding to all the pixel points on the frame light spot detection image is obtained to obtain the (i + 1) th difference value; calculating a difference between the ith difference and the (i + 1) th difference, and calculating a sum of the ith difference and the (i + 1) th difference; obtaining the difference value according to the ratio of the difference to the sum; wherein "[ 2 ]]An "operand symbol represents a rounding operation.
According to some embodiments of the application, the farthest historical frame light spot detection image stored away from the current frame, namely the kth frame, or the intermediate frame included in the historical frame is selected to obtain the difference value, so that the calculation amount of the obtained value is reduced, the speed of calculating the difference value is increased, and the accuracy of obtaining the difference value is improved.
In some embodiments, said selecting X historical frame spot detections from among said M historical frame spot detections further comprises: and if the difference value between the k-frame light spot detection image and the j-frame historical frame light spot detection image does not meet the difference threshold value, taking the k-1-frame light spot detection image as the X historical frame light spot detection images, wherein the j-frame light spot detection image belongs to one of the M historical frame light spot detection images.
Some embodiments of the present application, if it is determined that the difference between the stored historical frame light spot detection map and the current frame light spot detection map (i.e. the kth frame) is not great, select a historical frame light spot detection map closest to the current frame (i.e. the kth frame) to adjust the kth frame light spot detection map to obtain a kth frame light spot weight map, so as to improve the light spot flicker problem and minimize the data processing amount on the one hand.
In some embodiments, the obtaining the kth frame light spot weight map according to the kth frame light spot fusion map includes: performing curve stretching on the kth frame light spot point fusion graph to obtain a kth frame light spot point fusion stretching graph; and obtaining the light spot point weight map of the kth frame according to the light spot point fusion stretch map of the kth frame.
Some embodiments of the present application enhance the brightness of the overall image by stretching and pull the levels of brightness apart so that the resulting convoluted spots are more hierarchical.
In some embodiments, the obtaining the kth frame light spot weight map according to the kth frame light spot fusion map includes: and smoothing the kth frame light spot point fusion image or a kth frame light spot point fusion stretching image obtained by stretching the kth frame light spot point fusion image to obtain the kth frame light spot point weight image.
Some embodiments of the present application further enhance the speckle display effect of the obtained video by smoothing.
In some embodiments, the smoothing the k frame spot fusion map to obtain the k frame spot weight map includes: carrying out weighted averaging of different proportions on values of corresponding pixel points on a k-1 th frame light spot point fusion image and the k-1 th frame light spot point fusion image to obtain a weighted average value of each pixel point, and taking the weighted average value of each pixel point as a weight value of the corresponding pixel point on the k-1 th frame light spot point weight image; carrying out weighted averaging of different proportions on values of the same pixel points on a k-1 th frame light spot point fusion tensile graph and the k-1 th frame light spot point fusion graph to obtain a weighted average value of each pixel point, and taking the weighted average value of each pixel point as a weight value of a corresponding pixel point on the k-1 th frame light spot point weight graph; or, the step of performing smoothing processing on the k-th frame light spot point fusion stretch graph obtained by stretching the k-th frame light spot point fusion graph to obtain the k-th frame light spot point weight graph includes: carrying out weighted averaging of different proportions on values of the same pixel points on the k-1 th frame light spot point fusion drawing and the k-1 th frame light spot point fusion drawing to obtain a weighted average value of each pixel point, and taking the weighted average value of each pixel point as a weight value of a corresponding pixel point on the k-1 th frame light spot point weight drawing; or carrying out weighted average of different proportions on the values of the same pixel points on the k-1 th frame light spot point fused and stretched graph and the k-1 th frame light spot point fused and stretched graph to obtain a weighted average value of each pixel point, and taking the weighted average value of each pixel point as the weight value of the corresponding pixel point on the k-1 th frame light spot point weight graph.
According to some embodiments of the application, smoothing processing is performed on the current frame light spot detection image according to the historical frame light spot detection image by introducing the weight average algorithm with different proportions, so that the light spot display effect in the synthesized video is further improved, and the phenomenon of light spot flicker between adjacent frames in the synthesized target video is reduced.
In some embodiments, the method further comprises: and repeating the process of obtaining the target image to obtain a target video, wherein the target video comprises a plurality of target images.
Some embodiments of the present application obtain a video file with significantly improved speckle flicker effects by enabling acquisition.
In a second aspect, some embodiments of the present application provide a computer program product comprising a computer program, wherein the computer program, when executed by a processor, is adapted to implement the method according to any of the embodiments of the first aspect.
In a third aspect, some embodiments of the present application provide a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, may implement the method according to any of the embodiments of the first aspect.
In a fourth aspect, some embodiments of the present application provide an electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor, when executing the program, may implement the method according to any of the embodiments of the first aspect.
In a fifth aspect, some embodiments of the present application provide an apparatus for speckle flicker processing in a video blurring mode, the apparatus comprising: the light spot weight map acquisition module of the kth frame is configured to acquire a light spot weight map of the kth frame according to M historical frame light spot detection maps and the k frame light spot detection map, wherein the M historical frame light spot detection maps are obtained by respectively performing light spot detection on M original images, the k frame light spot detection map is obtained by performing light spot detection on the k frame original image, the M original image is one or more continuous multi-frame images which are located before the k frame original image and adjacent to the k frame original image, M is an integer greater than or equal to 1, and the target light spot blurring rendering map of the kth frame is an acquired target image.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are required to be used in the embodiments of the present application will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present application and therefore should not be considered as limiting the scope, and that those skilled in the art can also obtain other related drawings based on the drawings without inventive efforts.
FIG. 1 is a schematic diagram of an image processing system according to an embodiment of the present disclosure;
FIG. 2 is a flowchart of a method for image processing according to an embodiment of the present disclosure;
fig. 3 is a second flowchart of a method for image processing according to an embodiment of the present application;
FIG. 4 is a block diagram of an apparatus for image processing according to an embodiment of the present disclosure;
fig. 5 is a schematic composition diagram of an electronic device according to an embodiment of the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application.
It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, it need not be further defined and explained in subsequent figures. Meanwhile, in the description of the present application, the terms "first", "second", and the like are used only for distinguishing the description, and are not to be construed as indicating or implying relative importance.
In recent years, technical research based on artificial intelligence, such as computer vision, deep learning, machine learning, image processing, and image recognition, has been actively developed. Artificial Intelligence (AI) is an emerging scientific technology for studying and developing theories, methods, techniques and application systems for simulating and extending human Intelligence. The artificial intelligence subject is a comprehensive subject and relates to various technical categories such as chips, big data, cloud computing, internet of things, distributed storage, deep learning, machine learning and neural networks. Computer vision is used as an important branch of artificial intelligence, particularly a machine is used for identifying the world, and the computer vision technology generally comprises the technologies of face identification, living body detection, fingerprint identification and anti-counterfeiting verification, biological feature identification, face detection, pedestrian detection, target detection, pedestrian identification, image processing, image identification, image semantic understanding, image retrieval, character identification, video processing, video content identification, behavior identification, three-dimensional reconstruction, virtual reality, augmented reality, synchronous positioning and map construction (SLAM), computational photography, robot navigation and positioning and the like. With the research and progress of artificial intelligence technology, the technology is applied to various fields, such as security, city management, traffic management, building management, park management, face passage, face attendance, logistics management, warehouse management, robots, intelligent marketing, computational photography, mobile phone images, cloud services, smart homes, wearable equipment, unmanned driving, automatic driving, smart medical treatment, face payment, face unlocking, fingerprint unlocking, testimony verification, smart screens, smart televisions, cameras, mobile internet, live webcasts, beauty treatment, medical beauty treatment, intelligent temperature measurement and the like.
In the related art, a video blurring process based on a mobile terminal (e.g., an electronic device such as a mobile phone) needs to perform target object blurring (e.g., portrait video blurring) on each frame of image included in a video, and a target object blurring technique (e.g., portrait video blurring) mainly includes a target object segmentation technique (e.g., portrait segmentation technique) and a simulated depth rendering technique, where a currently mainstream method of the target object segmentation technique is mainly based on a depth learning model, and the simulated depth blurring rendering technique is mainly simulated by a method similar to gaussian blur.
In order to complete the video blurring process, each frame image in the video needs to be blurred respectively to obtain a blurred image of each frame, and obtaining the blurred image of each frame needs to obtain a light spot point weight map of the corresponding frame image first, then render the background segmentation image based on the light spot point weight map to obtain a background light spot blurring rendering map, and fuse the background light spot blurring rendering map and the target object segmentation image (for example, a portrait segmentation image) to obtain a blurred image corresponding to the frame (i.e., obtain a target object light spot blurring rendering map). The inventor of the present application finds in research that, when obtaining a speckle weight map of each frame on a current mobile terminal (e.g., a mobile phone), only the speckle distribution of the frame is considered (i.e., only a speckle detection map of the frame is passed), so that there are many effects on a video blurring effect (e.g., a portrait blurring effect), such as a video blurring effect that is not beautiful, and a relatively serious speckle flicker phenomenon between different frames.
Some embodiments of the application utilize time sequence information to optimize each frame of facula point detection map, namely utilize historical frame facula point detection map to optimize the facula point detection map of the current processing frame (namely k frame facula point detection map), obtain and treat the current processing frame facula point weight map (namely k frame facula point weight map) that the blurred image corresponds has fused historical frame facula distribution characteristic at present, can effectively improve the stability and the scintillation degree of sequence facula and can solve the serious problem of video facula scintillation at least, let the video facula that the algorithm generated more stable, more approximate true facula effect.
Referring to fig. 1, fig. 1 is a diagram illustrating an image processing system 100 according to some embodiments of the present disclosure, where the image processing system 100 includes: a mobile phone terminal 10 and a server 20.
The mobile phone terminal 10 has an image or video capturing unit 11, and in some embodiments of the present application, a segment of video 22 to be blurred (e.g., portrait video to be blurred) may be captured by the image capturing unit 11, and then the mobile phone terminal 10 transmits the video 22 to be blurred to the server 20. In some embodiments of the present application, an image to be blurred (not shown) may be captured by the image capturing unit 11, and then the image to be blurred may be sent to the server 20.
The server 20 may perform background blurring operation on each frame of image of the received video 22 to be blurred, obtain a target object light spot blurring rendering map with a better light spot blurring effect for each frame, and form a target video from the target object light spot blurring rendering maps and feed the target video back to the mobile phone terminal 10, so that the mobile phone terminal 10 obtains a target video with a good light spot display effect (for example, a light spot flickering phenomenon between different frames in the target video is improved).
It should be noted that the video 22 to be blurred includes multiple frames of original images, and the target video includes a target object light spot blurred rendering map obtained by blurring each frame of original image respectively. It can be understood that, in some embodiments of the present application, when the target object light spot blurring rendering map of the current processing frame is obtained, the light spot weight map according to which is determined by the light spot detection map of the current processing frame and the light spot detection map of the previous frame or multiple frames of history frames adjacent to the current processing frame, so that the flicker defect of the target object light spot blurring rendering map of the adjacent frame in the target video is effectively improved.
The following describes an exemplary process for acquiring a target object spot blurring rendering map and acquiring a target video, which is provided by some embodiments of the present application, with a specific frame. For example, in order to perform target object light spot rendering on a q-th frame of original image (belonging to any frame of original image in a video to be blurred), a q-th frame of light spot point weight map corresponding to the q-th frame of original image needs to be acquired, then, light spots are rendered according to the q-th frame of light spot point weight map to obtain a background light spot blurred rendering map of the frame, then, the background light spot blurred rendering map of the frame is fused with a target object segmentation map of the frame to obtain a target object light spot blurred rendering map of the frame, and the process is repeated to acquire the target object light spot blurred rendering map of each frame in the video to be blurred to obtain the target video. It should be noted that, in the related art, only the flare point detection map of the q-th frame is considered when the flare point weight map of the q-th frame is acquired, and in some embodiments of the present application, in order to acquire the flare point weight map of the q-th frame, it is also necessary to consider an influence of the flare point detection map of a frame before the q-th frame on the flare point weight map of the q-th frame, so that the target video obtained by using these embodiments of the present application can improve a flare phenomenon existing in an adjacent frame in the video.
It is understood that the shooting unit 11 of the mobile phone terminal 10 is located at the back of the mobile phone, in other embodiments, the camera may also be located at other positions, such as the front of the mobile phone terminal 10, and the embodiment of the present application does not limit the specific position of the image shooting unit 11 on the mobile phone terminal 10. The mobile phone terminal 10 may be any type of electronic device with an image capturing function, for example, the mobile phone terminal 10 may also be a notebook computer with an image capturing function, and the embodiment of the present application is not limited thereto.
The server 20 and the mobile phone terminal 10 are connected through a wired network (not shown in the figure) or a wireless network, and the image to be blurred and the target object light spot blurring rendering map are transmitted through the connection.
It should be noted that, in some embodiments of the present application, the mobile phone terminal 10 may also utilize its own processor to execute the method for processing an image provided in some embodiments of the present application, so as to obtain a target video or obtain a target image. It can be understood that, in these embodiments, the mobile phone terminal 10 does not output the video to be blurred to the user, but adjusts, by the processor of the mobile phone terminal 10, the light spot detection map corresponding to each frame image in the video to be blurred according to the historical frame light spot detection map (i.e., performs the method of light spot flicker processing in the video blurring mode provided in some embodiments of the present application), and finally outputs the target video or the target image to the user. It should be understood that, in this case, it is not necessary to separately provide one server 20 to execute the method for processing the flare flicker in the video blurring mode according to the embodiment of the present application.
A method of flare flicker processing in the video blurring mode performed by the server 20 or by a terminal device such as the mobile phone terminal 10 is exemplarily described below with reference to fig. 2.
As shown in fig. 2, some embodiments of the present application provide a method of image processing, the method comprising: s101, obtaining a light spot weight map of a kth frame according to the M historical frame light spot detection maps and the light spot detection map of the kth frame; and S102, obtaining a target object light spot blurring rendering image of the kth frame according to the light spot point weight image of the kth frame and the original image of the kth frame. It can be understood that, in order to obtain a target video with a better blurring effect, the image processing method according to the embodiment of the present application further includes: and S103, repeating the processes corresponding to the S101 and the S102 to obtain a multi-frame target object light spot blurring rendering map, and obtaining a target video according to the multi-frame target object light spot blurring rendering map. That is, a target video including a plurality of target images is obtained by repeating the process of acquiring the target images.
It is understood that, since the embodiment of the application considers the speckle detection map located before each frame when acquiring the speckle weight map of each frame, the problem that the speckle flicker generated by the electronic device in the video blurring mode is serious can be solved.
It can be understood that before S101 is executed, light spot detection maps corresponding to each of the M-frame original image and the k-th frame original image need to be acquired.
The steps involved in the above process are exemplarily set forth below.
First, a process of acquiring a spot detection map corresponding to each frame of an original image is exemplarily explained. It should be noted that, in some embodiments of the present application, a mobile phone terminal executes a process of acquiring each frame of light spot detection map.
That is, in some embodiments of the present application, before S101, the method for flare flicker processing in the video blurring mode further includes:
firstly, carrying out image segmentation on any one frame of original image in the M frame of original image and the k frame of original image to obtain a background segmentation image. For example, the first step includes the following steps a) to C). It should be noted that, in some embodiments of the present application, the speckle detection map corresponding to each frame of the original image may also be obtained by using the method disclosed in the related art, that is, the method for acquiring each frame of the speckle detection map is not limited to the method described in the embodiments of the present application. In other embodiments of the present application, the flare point detection map corresponding to each of the M-frame original image and the k-th frame original image is obtained by the first step method.
A) And the camera of the mobile phone captures images.
The user clicks the video to record by focusing the portrait, the mobile phone equipment obtains a frame of original image, namely obtains a frame of RGB format image I to be blurredimageObtaining an image I to be blurredimage。
B) Treating the blurred image I according to the portrait segmentation modelimageAnd separating the foreground and the background to obtain a portrait binary image, namely segmenting a portrait (namely a foreground) and a background of the image to be blurred according to an image segmentation model (namely a deep learning model) of the related technology to obtain a portrait segmentation mask.
The image I to be blurred is treated by a deep learning model (e.g., unet or resnet network) of the related artimageSegmenting the portrait to obtain a single-channel portrait binary image ImaskIn the single-channel portrait binary image, the values of all the pixel points in the main area (i.e. the area corresponding to the portrait) are 1, and the values of all the pixel points in the background area except the portrait are 0.
C) By means of Iimage×(1-Imask) Segmentation to background except for human figure IbgThe area corresponding to the portrait on the background segmentation map needs to be filled with 0.
And secondly, obtaining a single-channel image based on the maximum value of the plurality of channels of each pixel point in the background segmentation image. That is, for each pixel point, a single-channel map is obtained by selecting a maximum value from a plurality of channels of the background segmentation map. For example, the second step includes the following step D).
D) Segmentation chart I based on backgroundbgR, G and B, traversing the three channels one by one according to pixels to obtain the maximum value of the three channels, and obtaining the single-channel graph Imaxch_bg。
And thirdly, acquiring a light spot detection graph corresponding to any frame of original image at least according to the value of each pixel point on the single-channel image. It should be noted that the third step exemplarily includes the following E), or exemplarily includes the following step E) and step F), or exemplarily includes the following step E), step F), and step G).
That is to say, some embodiments of this application detect the light spot and then obtain light spot point detection map through obtaining the single channel map that corresponds with the background segmentation map, can promote the accuracy of detecting the light spot point that obtains. Specifically, in some embodiments of the present application, the third step exemplarily includes: fuzzifying the single-channel image to obtain a fuzzy single-channel image; and for any pixel point, confirming that the value of the pixel point on the single channel graph is larger than a first threshold value, and confirming that the value of the pixel point on the fuzzy single channel graph is smaller than or equal to a second threshold value, and confirming that the pixel point belongs to a light spot, wherein the second threshold value is smaller than the first threshold value. It should be noted that the value of each pixel may be a pixel value or a weight value, where the weight value is a value of each pixel obtained by normalizing the pixel values of all pixels in the entire image. That is to say, some embodiments of the present application utilize the attribute of the light spot (that is, the characteristic that the pixel value at the center of the light spot is large and the pixel value of the adjacent pixel point at the center of the light spot is small) to comprehensively determine whether each pixel point belongs to the light spot through the constructed blurred single-channel image and the single-channel image, thereby improving the accuracy of the detected light spot.
E) For the single channel chart I obtained in the step D) abovemaxch_bgAnd (2) detecting light spots (or called light spots) to obtain a light spot detection map of the frame, wherein the detailed flow is as follows:
i. for single channel chart Imaxch_bgGaussian blur of N x N (e.g. 3 x 3) is performed to obtain a blurred single-channel map Ismooth_maxch_bg。
For a single channel plot Imaxch_bgAnd fuzzy single-channel diagram Ismooth_maxch_bgTraversing, if the value of the pixel point at the (x, y) position meets the following condition, the pixel point is considered as a candidate light spot (namely belongs to a light spot), and obtaining a light spot detection map I consisting of all candidate light spotsinit_spots：
The if statement is a conditional judgment statement in C language, and the function of the judgment statement is: and when the two inequalities are judged to be simultaneously satisfied, executing an assignment statement behind the semicolon. The value operation described above means a normalization operation on a parameter in parentheses, specifically, value (I)maxch_bg) (x, y) represents the value (or called normalized pixel value, or called weight value) of the pixel point (x, y) on the normalized graph after the normalization processing is performed on the values of all the pixel points on the single-channel graph, and it is understood that the value belongs to a numerical value greater than or equal to 0 and less than or equal to 1. value (I)maxch_smooth_bg) (x, y) represents the value (or called normalized pixel value, or called weight value) of the pixel point (x, y) on the blurred single-channel image after normalization processing of the values of all the pixel points on the image, and it is understood that the value belongs to a numerical value greater than or equal to 0 and less than or equal to 1. The if statement needs to judge that the normalized pixel value of the pixel point (x, y) on the single-channel graph is greater than the set first threshold value bokeh _ thresh1, andthe normalized pixel value of the pixel point on the fuzzy single-channel graph is smaller than a set second threshold value bokeh _ thresh2, and in combination with the above statements, it is not difficult to find that both the two threshold values belong to two value ranges smaller than 1, and the maximum value included in the second threshold value is smaller than the minimum value included in the first threshold value. In addition, it can be seen from the above statements that, when a pixel point (x, y) simultaneously satisfies two conditions of a judgment statement (i.e., if statement), it is determined that the pixel point belongs to a light spot, and for the light spot, the value of the pixel point (x, y) on the single-channel image needs to be assigned to the light spot detection map Iinit_spotsAnd the same pixel point.
It should be noted that, for all the pixel points that do not satisfy the two conditions of the if statement, it is determined that the pixel points do not belong to the light spot, and the values of the pixel points on the light spot detection map are 0, it is understood that a map composed of all the determined light spot points can be determined by the light spot detection map.
F) Map I for spot detectioninit_spotsOr spot detection map Iinit_spotsAnd carrying out sparsification operation on the included dense area to obtain a sparse facula point detection diagram.
For example, spot detection map I is based on image zoom operation resize operation carried by Open Source Computer Vision Libraryinit_spotsFirstly, sampling 4 times (namely a specific example of target sampling multiplying power) and then sampling 4 times, so that filtering points can be simply and directly made by utilizing bilinear interpolation of resize self-band, and light spot dense gathering areas are thinned, so that the problems of overexposure and no layering in large light spot clustering areas in subsequent rendering are prevented.
The process of obtaining dense areas is exemplarily set forth below.
Judgment of Iinit_spotsDense instances of the collection of light spots in the image, including: dividing an image to be blurred into 4 small images with the same size according to half of the width and the height; solving the centroid coordinate of each small graph, namely representing the center of the pixel set in the area, and specifically: solving the zeroth order moment M of each small graph00And a first moment M10、M01(ii) a Based on the following formulaTo the centroid coordinates:
and respectively taking the respective centroid coordinate points as centers, and calculating the number sum _ spots of non-zero values in the range of 100 x 100.
Obtaining a dense region sampling threshold thresh of four small graphs, sum _ spots/10000, and if at least one thresh is more than 10%, determining that I isinit_spotsThe existence of dense areas can cause the problems of spot clustering and overexposure and the like, and the requirement on I is satisfiedinit_spotsOr the dense areas comprised by the graph, are subjected to a thinning operation.
G) And D), performing point supplementing processing on the discrete light spots to update the light spot detection diagram obtained in the step F).
Since the operation of step F) on the dense dots reduces the number of originally sparse discrete dots and the weight value, it is necessary to perform a dot-filling operation on these spots. The point supplementing operation is to detect the sparse light spot obtained in the step d to obtain an image I'init_spotsThe following operations are carried out:
detection of spot points map Iinit_spotsPerforming pixel traversal, if the number of light spots (or called light spots) in a pixel point (x, y) of the traversal in a neighborhood with a radius of M (for example, M is 3) pixel points does not exceed L (for example, L is 8), then the pixel point is considered to be a discrete sparse point, and map I 'needs to be detected at the sparse light spot point'init_spotsAnd (5) adding points around the position (x, y) and the pixel point. It should be noted that point supplementation is to improve the values of the pixel points on the thinned light spot detection map, and the specific method is as follows.
Updating sparse light spot detection map I'init_spotsThe value of the discrete sparse point (x, y) above (the type of value includes a weight value or a pixel value), e.g. map Iinit_spotsAnd assigning the values of the pixel points corresponding to the discrete sparse points to the same pixel points on the sparse light spot detection graph, so as to perform weight gain on the sparse discrete points.
v. update sparse light spot detection map I'init_spots(x, y) period of discrete sparse points of (c)The value of the surrounding point (the type of the value includes a weight value or a pixel value) is assigned with the same weight in a neighborhood range with a radius of P (for example, the value of P is 1) away from the discrete sparse point, that is, the weight values of a plurality of (for example, the value is 8 when the value of P is 1) pixel points around the pixel point (x, y) corresponding to the discrete sparse point on the sparse spot detection map are improved, so that the discrete spot is brighter.
Light spot detection graph I of corresponding frame can be obtained based on the stepsspotsThe purpose of obtaining the graph is to perform weight factor enhancement on the spot generation area when convolution blurring is performed, and the graph is different from the common blurring effect of non-spots.
It should be noted that, in some embodiments of the present application, step E) may be performed to obtain Iinit_spotsAs a spot detection map for each frame required in S101. In other embodiments of the present application, the sparse speckle detection map obtained in step F) may also be used as each frame speckle detection map required in S101. In still other embodiments of the present application, the discrete light spots may be subjected to the point-complementing process, that is, the image obtained in step G) is used as the light spot detection map required in step S101. The above-mentioned steps a), B), C), D), E), F) and G) are steps that can be performed sequentially, and the same symbol indicates the same sub-image in these steps.
That is, in some embodiments of the present application, I obtained after performing steps A) to E) may beinit_spotsAs the speckle detection map required by S101, in other embodiments of the present application, the sparse speckle detection map obtained after performing steps a) to F)) may be used as each frame speckle detection map required by S101, and in still other embodiments of the present application, the image obtained after performing steps a) to G) may be used as the speckle detection map described in S101.
Next, an implementation process of S101 is exemplarily explained.
The M historical frame light spot detection maps described in S101 are obtained by performing light spot detection (for example, using the embodiments of steps a) to E)) on M frames of original images, respectively, the k-th frame light spot detection map is obtained by performing light spot detection (for example, using the embodiments of steps a) to E)) on a k-th frame of original image, where the M frames of original images and the k-th frame of original images are located in the same video to be blurred (for example, the video 22 to be blurred in fig. 1), the M frames of original images are one or more frames of images that are located before the k-th frame of original image and are adjacent to the k-th frame of original image, and M is an integer greater than or equal to 1. Note that the neighborhood includes original images adjacent to or closer to the k-th frame original image. The k-th frame original image is any frame original image in the video to be blurred, and a value of k is related to a number of a frame, for example, the value of k is an integer greater than or equal to 0 in some embodiments (it can be understood that when the value of k is 0, a corresponding history frame does not exist because the k belongs to an initial frame of the video, and the k-th frame is other frames after the initial frame in some embodiments of the present application), and it can be understood that the initial frame of the video to be blurred can also be defined as the 1 st frame. For example, if k is 5 and M is 2, that is, the current processing frame is the 5 th frame original image, and S101 is executed to adjust a current processing frame light spot detection map (i.e., the 5 th frame light spot detection map) according to two historical frame light spot detection maps to obtain a 5 th frame light spot weight map, where the 5 th frame light spot detection map is obtained by performing light spot detection on the 5 th frame original image in the video 22 to be blurred, the 4 th frame light spot detection map is obtained by performing light spot detection on the 4 th frame original image in the video to be blurred, and the 3 rd frame light spot detection map is obtained by performing light spot detection on the 3 rd frame original image in the video to be blurred.
It should be noted that, in some embodiments, the frame number corresponding to the k-th frame original image belongs to an absolute frame number of the frame original image in the video to be blurred, for example, the video to be blurred includes 100 frames of original images, and when the currently processed frame is the 20 th frame, the k-th frame original image is the 20 th frame original image. In other embodiments of the present application, the frame number corresponding to the k-th frame of the original image is a buffer frame number of the speckle detection image corresponding to the frame in the buffer queue. For example, the video sequence to be blurred includes 100 frames in total, the length of the queue for the buffer light spot detection map is 8, and the frame images in the queue are numbered sequentially from 0 in the morning to the evening according to the storage time, so it is understood that the frame number of the currently processed frame is k ═ 7 each time, that is, the frame number of the currently processed frame is 7, that is, the k-th frame light spot detection map referred to in S101, that is, the 7-th frame light spot detection map, it is understood that the absolute frame number of the original image corresponding to the 7-th frame light spot detection map in the video to be blurred may not be 7, and at this time, the frame number in the buffer queue is also used for numbering the original frame.
In some embodiments of the present application, when the values of k are different, a different number of partial frames are selected from M historical frame facula detection maps to correct the facula detection map of the current frame. In other embodiments of the present application, the same number of historical frame flare detection maps are selected to correct the flare detection map of the current frame no matter what the value of k is. That is, in some embodiments of the present application, S101 exemplarily includes: and directly fusing the M historical frame light spot detection images with the k frame light spot detection image to obtain a k frame light spot fusion detection image, and then obtaining a k frame light spot weight map according to the k frame light spot fusion detection. In other embodiments of the present application, S101 exemplarily includes: selecting X historical frame light spot detection maps from M historical frame light spot detection maps, fusing the selected X historical frame light spot detection maps with a k frame light spot detection map to obtain a k frame light spot fusion detection map, and then obtaining a k frame light spot weight map according to the k frame light spot fusion detection.
The following exemplary process dynamically selects the number of historical speckle detection images required for the currently processed frame (i.e., the k-th frame) from the M historical frame speckle detection images. For example, some embodiments of the present application dynamically determine the number of historical frames required for optimal flicker by an inter-frame difference detection mechanism of flare, so that the flare flicker problem of adjacent frames can be solved with as few historical frames as possible.
Specifically, in some embodiments of the present application, the S101 example includes:
the method comprises the following steps of firstly, selecting X pieces of historical frame light spot detection maps from M pieces of historical frame light spot detection maps, wherein the X pieces of historical frame light spot detection maps are adjacent to the k frame light spot detection map, X is an integer which is greater than or equal to 1 and less than or equal to M, and the adjacent meaning is the same as that described above. For example, in some embodiments of the present application this first step example comprises: and selecting at least part of historical frame light spot detection images from the M historical frame light spot detection images according to the difference of the distribution of light spots among different frame original images. Specifically, in some examples of the present application, the first step illustratively comprises: and if it is confirmed that a difference value between the kth frame light spot detection map (i.e., the light spot detection map corresponding to the current processing frame) and the jth frame historical frame light spot detection map meets a difference threshold value, taking the jth frame to (k-1) frame light spot detection maps as the X historical frame light spot detection maps, wherein the jth frame light spot detection map belongs to one frame of the M frame light spot detection maps. For example, j is greater than or equal to k-M and less than or equal to k-1.
It is understood that in some embodiments of the present application, the difference values between the kth frame of speckle detection map and each of the above M historical frame of speckle detection maps may be obtained until a difference value satisfying a difference threshold is found. In other embodiments of the present application, difference values between the kth frame of speckle detection map and a part of the M historical frame of speckle detection maps may be obtained until a difference value satisfying a difference threshold is found from the part of the historical frame of speckle detection maps. That is, in some embodiments of the present application, the above example of confirming that a difference value between the kth frame facula point detection map and the jth frame history frame facula point detection map satisfies a difference threshold includes: and confirming whether the difference value of each light spot detection image in the k-th frame light spot detection image and the M historical frame light spot detection images meets the requirement of a difference threshold value one by one. In some embodiments of the present application, the above example of confirming that a difference value between the kth frame facula point detection map and the jth frame history frame facula point detection map satisfies a difference threshold includes: and determining whether the difference value between the light spot detection image of the k-th frame and the partial frame light spot detection images in the M historical frame light spot detection images meets the requirement of a difference threshold value.
Taking the example of calculating the difference value between the currently processed frame light spot detection map (i.e. the k-th frame light spot detection map) and the partial history frame light spot detection map until the difference value meeting the difference threshold is found, the process of acquiring at least partial history frame light spot detection map from M history frame light spot detection maps is exemplarily described below.
For example, in some embodiments of the present application, j has a value of
i is an integer greater than or equal to 0. For example, if M is 8 and i is 0, the difference value between the k-th frame light spot detection map and the k-8 th frame history frame light spot detection map is determined, and if M is 8 and i is 1, the difference value between the k-th frame light spot detection map and the k-4 th frame history frame light spot detection map is determined. It should be noted that, in the following description,
is a number equal to or greater than 0, and therefore the maximum value of i can be determined.
It is exemplarily set forth below that when j takes on a value of
And how to obtain a difference value between the k frame light spot detection diagram and the j frame historical frame light spot detection diagram.
An exemplary method for confirming that a difference value between the kth frame light spot detection map and the jth frame historical frame light spot detection map meets a difference threshold value, which is described in S101, includes: and i, repeating the following process from 0 until the difference value obtained in the following way meets the difference threshold value: calculating the k frame facula point detection chart and the
Detecting absolute values of differences of corresponding pixel values of all pixel points on the image by using the frame light spot points to obtain ith differences; calculating the sum of the k frame light spot detection diagram and the k frame light spot detection diagram
The absolute value of the difference value of the pixel values corresponding to all the pixel points on the frame light spot detection image is obtained to obtain the (i + 1) th difference value; calculating a difference between the ith difference and the (i + 1) th difference, and calculating a sum of the ith difference and the (i + 1) th difference; obtaining the difference value according to the ratio of the difference to the sum; wherein "[ 2 ]]The "operand symbol represents a rounding operation. It should be noted that the rounding operation includes rounding up or rounding down, and these formulas are also included in the following concepts provided above in the present application: at least part of historical frame light spot detection graphs are selected from M historical frame light spot detection graphs by acquiring the difference of light spot distribution on different frame original images, so that the historical frame light spot detection graphs and the k frame light spot detection graphs are fused to obtain a k frame light spot weight graph.
For example, if M is 8, k is 16, and i is 0, the zeroth difference is obtained as follows: calculating the absolute value of the difference value of each pixel point on the 16 th frame of light spot detection graph and the 8 th frame of light spot detection graph corresponding to the 8 th frame of original image to obtain an ith difference value; calculating the absolute value of the difference value of each pixel point on the 16 th frame light spot detection graph and the 12 th frame light spot detection graph corresponding to the 12 th frame original image to obtain an i +1 th difference value; calculating a difference between the ith difference and the (i + 1) th difference, and calculating a sum of the ith difference and the (i + 1) th difference; and obtaining a zeroth difference value according to the ratio of the difference to the sum, and confirming that the zeroth difference value does not meet the difference threshold value. Further, when i is 1, the corresponding first difference value is calculated by the following method: calculating the absolute value of the difference value of each pixel point on the 16 th frame light spot detection graph and the 12 th frame light spot detection graph corresponding to the 12 th frame original image to obtain an ith difference value; calculating the absolute value of the difference value of each pixel point on the 16 th frame of light spot detection graph and the 14 th frame of light spot detection graph corresponding to the 14 th frame of original image to obtain an i +1 th difference value; calculating a difference between the ith difference and the (i + 1) th difference, and calculating a sum of the ith difference and the (i + 1) th difference; and obtaining a first difference value according to the ratio of the difference to the sum, and taking 3 historical frame facula point detection graphs from the 12 th frame to the 15 th frame as M historical frame facula point detection graphs related to S101 when the first difference value is confirmed to meet the difference threshold value.
It can be understood that, if it is finally determined through multiple cycles that the difference values between the kth frame of light spot detection map and all the jth frame of historical frame of light spot detection maps do not satisfy the difference threshold, at this time, a frame of historical frame of light spot detection map closest to the kth frame may be selected as the historical frame of light spot detection map selected from the M historical frame of light spot detection maps. For example, in some embodiments of the present application, the method for flare flicker processing in the video blurring mode further includes: and if the difference values between the k-th frame light spot detection image and all the j-th frame historical frame light spot detection images do not meet the difference threshold value, taking the k-1-th frame light spot detection image as the X historical frame light spot detection images, wherein the j-th frame light spot detection image belongs to one frame of image in the M frame light spot detection images. That is, if it is determined that the stored historical frame light spot detection map has little difference from the current frame light spot detection map, some embodiments of the present application select a historical frame light spot detection map closest to the current frame (i.e., the k-th frame) to adjust the k-th frame light spot detection map to obtain a k-th frame light spot weight map, so as to improve the light spot flicker problem and minimize the data processing amount.
It should be noted that, in some embodiments of the present application, the k-th frame and M pieces of history frame spot detection maps obtained in the above steps may be further placed in a queue with a length of M +1, and then a part of the history frame spot detection map is selected from the queue as a reference frame for adjusting the k-th frame spot detection map (the reference frame is at least a part of the history frame spot detection map selected from the M pieces of history frame spot detection maps). At this time, the difference value can be calculated according to the sorting number of each historical frame facula detection image in the queue. The following will explain the process with reference to an example, which is not described herein in detail to avoid repetition.
And secondly, fusing the X historical frame light spot detection graphs and the k frame light spot detection graph to obtain a k frame light spot fusion graph.
For example, weighted average processing is respectively performed on the values of the same pixel points on at least part of (i.e., X) historical frame light spot detection maps and the k-th frame light spot detection map obtained in the first step to obtain a weighted average value of each pixel point, and then the k-th frame light spot fusion map is obtained according to the weighted average value of each pixel point. It is understood that other fusion approaches may be used by those skilled in the art.
And thirdly, obtaining a light spot weight map of the kth frame according to the light spot fusion map of the kth frame. For example, in some embodiments of the present application, pixel value normalization processing is directly performed on the k-th frame light spot fusion map to obtain a k-th frame light spot weight map. In other embodiments of the present application, pixel value normalization and stretching are performed on the k-th frame light spot fusion image, and the stretched image is used as the k-th frame light spot weight image. In still other embodiments of the present application, pixel value normalization and stretching are performed on the k-th frame of light spot point fusion image, the stretched image is smoothed according to the historical frame processing result, and the smoothed image is used as the k-th frame of light spot point weight image.
The following exemplarily explains how S101 adjusts the k-th frame light spot detection map according to the selected X historical frame light spot detection maps to obtain a k-th frame light spot weight map.
It should be noted that, in some embodiments of the present application, a current frame (i.e., the k-th frame) light spot weight map is obtained by fusing pixel values on a current frame light spot detection map and pixel values on at least a partial image screened from M historical frame light spot detection maps, so that a light spot flicker phenomenon occurring between a previous frame and a next frame is effectively improved, and a video light spot display effect is improved.
For example, in some embodiments of the present application, S101 illustratively comprises:
and step one, fusing the X historical frame light spot detection graphs and the k frame light spot detection graph to obtain a k frame light spot fusion graph. For example, in some embodiments of the present application, the first step illustratively comprises: and respectively carrying out weighted average processing on values of corresponding pixel points on the M historical frame light spot detection graphs and the k frame light spot detection graph to obtain a weighted average value of each pixel point, and then obtaining the k frame light spot fusion graph according to the weighted average value of each pixel point. That is to say, some embodiments of the present application fuse the k-th frame light spot detection map and the M historical frame light spot detection maps through a weighted average algorithm, so that the light spot distribution characteristics of the obtained k-th frame light spot fusion map are as close as possible to the light spot distribution characteristics of the previous historical frame, and the light spot flicker phenomenon existing in the composite video is effectively improved.
And secondly, obtaining a light spot weight map of the kth frame according to the light spot fusion map of the kth frame.
For example, in some embodiments of the present application, the obtaining the kth frame light spot weight map according to the kth frame light spot fusion map includes: and smoothing the kth frame light spot point fusion image or a kth frame light spot point fusion stretching image obtained by stretching the kth frame light spot point fusion image to obtain the kth frame light spot point weight image.
In some embodiments of the present application, the above process of smoothing the k frame light spot fusion map to obtain the k frame light spot weight map exemplarily includes: carrying out weighted averaging of different proportions on values of corresponding pixel points on a k-1 th frame light spot point fusion image and the k-1 th frame light spot point fusion image to obtain a weighted average value of each pixel point, and taking the weighted average value of each pixel point as a weight value of the corresponding pixel point on the k-1 th frame light spot point weight image; or carrying out weighted average of different proportions on the values of the same pixel points on the k-1 th frame light spot point fusion tensile graph and the k-1 th frame light spot point fusion graph to obtain a weighted average value of each pixel point, and taking the weighted average value of each pixel point as the weight value of the corresponding pixel point on the k-1 th frame light spot point weight graph.
In some embodiments of the present application, the process of obtaining the light spot weight map of the kth frame by performing smoothing processing on the light spot point fusion and stretching map of the kth frame obtained by stretching the light spot point fusion and stretching map of the kth frame exemplarily includes: carrying out weighted averaging of different proportions on values of the same pixel points on the k-1 th frame light spot point fusion drawing and the k-1 th frame light spot point fusion drawing to obtain a weighted average value of each pixel point, and taking the weighted average value of each pixel point as a weight value of a corresponding pixel point on the k-1 th frame light spot point weight drawing; or carrying out weighted average of different proportions on the values of the same pixel points on the k-1 th frame light spot point fused and stretched graph and the k-1 th frame light spot point fused and stretched graph to obtain a weighted average value of each pixel point, and taking the weighted average value of each pixel point as the weight value of the corresponding pixel point on the k-1 th frame light spot point weight graph.
That is to say, some embodiments of the present application introduce different proportions of the weighted average algorithm to perform smoothing processing on the current frame light spot detection map according to the historical frame light spot detection map, further improve the light spot display effect in the synthesized video, and reduce the flicker phenomenon of light spots between adjacent frames in the synthesized target video.
For example, in some embodiments of the present application, the kth frame spot weight map is obtained by the following formula:
wherein the content of the first and second substances,
characterizing the light spot fusion stretch map of the (k-1) th frame,
characterizing the k-th frame spot fusion stretch chart, Iresult_spotAnd characterizing the light spot weight map of the kth frame, wherein the value range of n is more than or equal to 2 and less than or equal to 5. Some embodiments of the present application further improve the speckle flicker phenomenon occurring between adjacent frames in the target video by introducing a smoothing formula.
The following example S102 is an implementation process of obtaining a light spot blurring rendering map of a target object in a kth frame according to a light spot weight map in the kth frame and an original image in the kth frame. It can be understood that the kth frame spot weight map has been obtained through the execution of S101.
H) Performing saturation enhancement on the k frame original image (namely the image to be blurred) to obtain a saturation enhancement image I'imageThe purpose of obtaining the image is to increase the flamboyance degree of blurring the background portrait and the diversity of the spot colors.
I) Utilizing saturation enhancement map and formula I'image×(1-Imask) Obtaining a saturation enhanced background segmentation map I 'except for a target object (e.g., portrait)'bgOn the figure, the portrait area is filled with 0.
J) Segment map I 'against saturation enhancing background'bgPerforming gamma conversion (for example, gamma. 1.6), and nonlinearly stretching the brightness of the background to obtain a stretched background segmentation map I'gamma_bg。
K) The spot area brightness gain operation is performed on the background as follows:
vi. apply the original drawing IimageConverting into YUV format image, wherein single-channel Y image is named as Igray。
vii. mixing IgrayNormalized to [0,1 ]]To obtain Ilight_weightThen overall +1, the range of values is [1, 2 ]]。
According to Ilight_weight*I'gamma_bgObtaining an image I with brightness gainimage4renderObtaining the background image I to be rendered of the kth frameimage4render。
L) at least according to the light spot weight map of the kth frame and the background image I to be rendered of the kth frameimage4renderAnd executing a light spot rendering method to obtain a k frame background light spot blurring rendering image.
It should be noted that, in order to perform the spot rendering method, a depth image needs to be acquired, and the process exemplarily includes the following operations: based on IimagePerforming depth estimation, and obtaining a depth map I by using a depth learning method (e.g. mono depth, midas, etc. model)depthWhen the camera of the mobile phone is focused, the coordinate position (x) of the portrait in the image can be returned0,y0) Obtaining a depth image I by the following formulacoc；
Where coc represents the size of the depth of field, i.e. the radius of the spot required for rendering.
For example, map of spot weights, I, for the k-th frameimage4renderAnd IcocAs an input, a spot rendering method based on convolution is input, and gamma conversion is performed again (for example, gamma is 1/1.6), so that a k-th frame background spot blurring rendering map is finally obtained.
And O) fusing the target object (such as a portrait) with the saturation gain with the k frame background light spot blurring rendering image to obtain a k frame target object light spot blurring rendering image.
It is understood that, by repeatedly performing the above processes, a target object light spot blurring rendering map corresponding to each frame in the video to be blurred can be obtained, so that the target video is obtained.
A method of image processing performed by a mobile phone terminal is exemplarily set forth below with reference to fig. 3.
Some embodiments of the application can solve the problem that light spots generated in a mobile phone end video blurring mode flicker seriously, and the simulated light spot effect approaches the single-reflex shooting effect. For example, in some embodiments of the present application, a depth learning model (e.g., a unet or resnet frame segmentation model) is first used to perform portrait segmentation on each captured frame of original image, a disparity estimation map coc distribution map is obtained by using the depth estimation model, and then blurring rendering of a single image is performed in a convolution manner. The method of flare flash processing in the video blurring mode performed by the handset is exemplarily described below with reference to fig. 3.
S201, the camera of the mobile phone captures the original image of the current processing frame (namely the k frame original image).
It can be understood that after a user of a current frame of original image is pointed at a portrait and clicks a video recording, a certain frame of original image (i.e., a kth frame of original image) in a section of video acquired by a background of a mobile phone device.
Suppose that the mobile phone device at the current moment acquires an RGB (red, green and blue) format image I to be blurredimageThe RGB format image to be blurred is the k-th frame original image.
And S202, acquiring a light spot detection diagram of the current processing frame according to the original image of the current processing frame. It is understood that the spot detection point map of each frame is obtained by the spot detection method.
The following example sets forth the process of acquiring a currently processed frame speckle detection map, and it will be appreciated that the following example is equally applicable to the acquisition of other historical frame speckle detection maps of respective frames.
1. For the original image I of the current processing frameimagePerforming portrait segmentation, and obtaining single-channel binary image I by using deep learning model (such as unet and resnet frames)maskWherein, the pixel value of the corresponding pixel point of the main body (i.e. the human image) on the single-channel binary image is 1 and the background is 0.
2. By means of Iimage×(1-Imask) Obtaining a background picture I except for the portraitbgFilling 0 in the portrait area;
3. based on IbgR, G and B, and the maximum value of the three channels is obtained by traversing pixel by pixel, namely the single-channel image Imaxch_bg；
4. Based on Imaxch_bgAnd carrying out light spot detection to obtain a light spot detection map of a corresponding frame, wherein the detailed flow is as follows:
4-1 pairs of single channels Imaxch_bgBlurring (e.g. using a convolution of 3 x 3 and gaussian blurring of the single-channel image) results in a blurred single-channel image Ismooth_maxch_bg；
4-2 pairs of Imaxch_bgAnd Ismooth_maxch_bgTraversal is performed to consider a candidate spot (i.e., light) if the pixel at the (x, y) position satisfies the following conditionSpots) and obtains a current frame spot detection map Iinit_spots
It should be noted that the current frame light spot detection map obtained in step 4-2 may be further processed to obtain a more optimized current frame light spot detection map (for example, the current frame light spot detection map is subjected to sparsification and a point complementing operation, which is specifically referred to the above example of obtaining the light spot detection map and is not described herein for avoiding repetition), and the corresponding light spot detection map required by S101 belongs to the more optimized current frame light spot detection map.
And S203, acquiring a historical frame facula point detection map. It is understood that, the obtaining manner of each historical frame facula point detection map can refer to the above-mentioned step S202. Are not described in detail herein to avoid repetition.
And S204, creating a sequence frame buffer queue Q.
For example, with Q taken to 10, historical frame blob detect (9 frames) and the current processed frame blob detect are stored in a fixed-length queue container Q.
That is to say, a global cache queue Q is created for caching the generated speckle detection maps, the length of Q is 10, the first address always stores the speckle detection map highlightspots of the current processing frame, and by analogy, the last frame of Q stores the speckle detection map of the history frame farthest from the current frame.
And S205, selecting at least partial (corresponding to the X pieces of historical frame light spot detection maps) from the cache degree column Q according to the interframe difference.
That is, the number of frames M (M < len (Q)) required for optimizing flicker is determined by the interframe difference detection mechanism, len (Q) representing the length of the buffer queue Q, and Q ═ M.
The number of frames required by the flicker optimization processing per frame is uncertain, and self-adaptive selection needs to be performed through difference information of continuous sequences in Q, and the method specifically comprises the following two steps:
the first step is as follows: taking the length of Q as 10, calculating a plurality of difference maps by four times as follows:
making the absolute value of the difference between the head frame and the tail frame pre-stored in Q for the first time to generate a new mat graph
The following equation:
that is to say, the maximum difference facula point distribution graph is obtained according to the difference between the facula point detection graph of the current processing frame and the value of the same position point (or called as the same pixel point) on the facula point detection graph of the earliest historical frame in the cache queue
The maximum difference light spot distribution diagram
The value of each pixel point above is positive or 0.
Then, making absolute difference between the intermediate frame and the current frame in Q for the second time to generate a new mat graph
The following graphic formula:
wherein N iscurIs represented as the spot detection point sequence number of the current frame in the queue (namely the stored sequence number in the buffer queue, and the spot detection maps of the historical frames are stored in turn according to the generation sequence), N0The sequence number of the subtracted object when the absolute difference value is made for the first time can be simplified as the following formula:
by analogy in the following way,
the third absolute difference reduction is as follows:
the fourth absolute difference reduction is as follows:
second, a fixed threshold parameter thresh _ diff is set to be 0.35 (as a specific example of the difference threshold, the value range of the difference threshold is greater than or equal to 0.2 or less than or equal to 0.45), and the following inequality needs to be satisfied:
if s is equal to 0, then information of 0-9 ten frames of serial numbers is required to be provided to a flicker optimization module (that is, all the light spot detection maps of the 9 historical frames are selected, and the 0-9 ten frame light spot detection maps are fused to obtain a light spot weight map of the 9 th frame of the current processing frame), that is, the subsequent flicker optimization module is at least configured to obtain the light spot weight map corresponding to the 9 th frame by fusing the light spot detection maps of the 10 frames. If s is equal to 1, information of six frames with serial numbers of 4-9 is required to be provided for a flicker optimization module (namely, 5 recent historical frame images are selected from 9 historical frame light spot detection images, and the 4-9 six frame light spot detection images are fused to obtain a light spot weight image of the 9 th frame of the current processing frame), namely, the subsequent flicker optimization module is at least configured to obtain the light spot weight image corresponding to the 9 th frame by fusing the 6 frame light spot detection images; if s is 2, information of four frames with serial numbers of 6-9 is required to be provided for a flicker optimization module (namely, 3 recent historical frame images are selected from 9 historical frame light spot detection images, and the 6-9 four frame light spot detection images are fused to obtain a light spot weight image of the 9 th frame of the current processing frame), namely, the subsequent flicker optimization module is at least configured to obtain the light spot weight image corresponding to the 9 th frame by fusing the 4 frames of light spot detection images; if s is 3, information of three frames with serial numbers of 7-9 is required to be provided for a flicker optimization module (namely, 3 recent historical frame images are selected from 9 historical frame light spot detection images, and the 7-9 three frame light spot detection images are fused to obtain a light spot weight image of the 9 th frame of the current processing frame), namely, the subsequent flicker optimization module is at least configured to obtain the light spot weight image corresponding to the 9 th frame by fusing the 3 frame light spot detection images; if the current frame does not meet the requirement, only the information of 8 frames and 9 frames is needed to be provided for a flicker optimization module (namely, the latest frame of historical frame image is selected from the 9 frames of historical frame light spot detection images, and the 8-9 frames of light spot detection images are fused to obtain the light spot weight image of the 9 th frame of the current processing frame), namely, the subsequent flicker optimization module is at least configured to obtain the light spot weight image corresponding to the 9 th frame by fusing the light spot detection images of the two frames. It will be appreciated that in these examples frame 9 is the currently processed frame and the frame number of the spot detection map for frame 9 is the number in the buffer queue.
It should be noted that, in some embodiments of the present application, when it is determined that s is equal to 0, a difference value between the currently processed frame light spot detection map and the 0 th frame light spot detection map satisfies a difference threshold (a frame number in the buffer queue, not an absolute frame number in the video to be blurred), and then no calculation is required to be performed
And
and fusing all the light spot detection images corresponding to the frames from 0 th frame to the current processing frame to obtain a light spot fused image of the current processing frame. Computing
Value-namely, calculating the k frame facula point detection map and the
And detecting the absolute value of the difference value of the corresponding pixel values of each pixel point on the image by the frame light spot point to obtain the ith difference value. Computing
Value is calculated by summing the light spot detection map of the k frame and the light spot detection map of the k frame
And detecting the absolute value of the difference value of the corresponding pixel values of each pixel point on the image by using the frame light spot point to obtain the (i + 1) th difference value.
S206, carrying out weighted average on at least part of historical frame light spot detection maps and the current frame light spot detection map to generate a current frame light spot fusion detection map.
For example, according to S205, light spots of consecutive m +1 frames including the light spot detection map at the present time may be acquired, and the light spot detection map of the m +1 frame is used as a weight map for acquiring flicker optimization, that is, all of the consecutive m +1 frames are weighted and averaged to generate a light spot fusion detection map of the current processing frame including information of the m +1 frame.
And S207, performing curve stretching enhancement on the light spot point fusion detection image of the current processing frame to obtain a light spot point fusion stretching image of the current processing frame.
That is to say, the light spot point fusion detection map obtained in S206 is subjected to primary curve stretching to increase the overall value, and the obtained light spot point fusion stretching map of the current processing frame is referred to as the "light spot point fusion stretching map" of the current processing frame
And S208, acquiring a light spot point fusion stretch map of the previous frame (namely, the previous frame adjacent to the current processing frame). It can be understood that the process of obtaining the light spot fusion stretch chart of the previous frame is the same as the process of obtaining the fusion stretch chart of the currently processed frame, and please refer to the above specifically, so that redundant description is not repeated here to avoid repetition.
S209, carrying out weighted average processing on the light spot point fused and stretched graph of the current processing frame and the light spot point fused and stretched graph of the previous frame in different proportions to obtain a light spot point weight graph of the current frame.
For example, the weighted average processing of different proportions is performed using the following formula:
wherein the content of the first and second substances,
showing a fusion stretch map of spot points of the light of the current processing frame,
showing the spot point fusion stretch map of the previous frame.
And S210, separating the foreground and background of the original image of the current processing frame according to the portrait segmentation model to obtain a background segmentation image and a portrait segmentation image.
S211, preprocessing such as intensity enhancement, segmentation, gamma correction and stretching is carried out on the original image of the current frame, and finally a background image I to be rendered of the current frame to be processed is obtainedimage4renderThe specific implementation process may refer to the above-mentioned background image I to be rendered for obtaining the kth frameimage4renderThe process of (a) is not described herein in detail to avoid repetition.
And S212, obtaining a target object light spot blurring rendering map of the current processing frame by adopting a light spot rendering method. For example, the portrait segmentation map of the current processing frame and the background light spot rendering map of the current processing frame are fused and merged to obtain a final portrait light spot blurring rendering map (that is, a specific example of the target object light spot blurring rendering map). That is, the background image I to be rendered according to the currently processed frameimage4renderAnd obtaining a target object light spot blurring rendering map of the current processing frame by adopting a light spot rendering method according to the light spot weight map of the current processing frame, wherein the specific process can refer to the exemplary explanation aiming at S102.
That is, the light spot weight map of the current processing frame obtained in S209 and the preprocessed current processing frame are to be renderedBackground image Iimage4renderAnd sending the image into a light spot rendering method to obtain a background light spot rendering image of the current processing frame. It can be understood that image rendering the S101 speckle weight map significantly improves the speckle flicker problem in the target video.
And S213, repeating the above process to obtain a multi-frame target object light spot blurring rendering image, and obtaining a target video according to the multi-frame target object light spot blurring rendering image.
Referring to fig. 4, fig. 4 shows an apparatus for image processing provided in an embodiment of the present application, it should be understood that the apparatus corresponds to the above-described embodiment of the method of fig. 2, and can perform various steps related to the above-described embodiment of the method, and specific functions of the apparatus can be referred to the above description, and detailed descriptions are appropriately omitted herein to avoid repetition. The device comprises at least one software functional module which can be stored in a memory in the form of software or firmware or solidified in an operating system of the device, and the device for processing the facula flicker in the video blurring mode comprises: a kth frame light spot weight map obtaining module 101 and a kth frame target object light spot blurring rendering module 102. In some embodiments of the present application, the apparatus further comprises a target video acquisition module 103.
The k-th frame light spot weight map obtaining module 101 is configured to obtain a k-th frame light spot weight map according to M history frame light spot detection maps and a k-th frame light spot detection map, where the M history frame light spot detection maps are obtained by respectively performing light spot detection on M frames of original images, the k-th frame light spot detection map is obtained by performing light spot detection on a k-th frame of original images, the M frames of original images and the k-th frame of original images are located in the same video to be blurred, the M frames of original images are one or more frames of images located before the k-th frame of original images and adjacent to the k-th frame of original images, and M is an integer greater than or equal to 1.
The kth frame target object light spot blurring rendering module 102 is configured to obtain a kth frame target object light spot blurring rendering according to the kth frame light spot weight map and the kth frame original image.
The target video obtaining module 103 is configured to obtain a multi-frame target object light spot blurring rendering map, and obtain a target video according to the multi-frame target object light spot blurring rendering map.
It is clear to those skilled in the art that, for convenience and brevity of description, the specific working process of the apparatus described above may refer to the corresponding process in the foregoing method, and will not be described in too much detail herein.
Some embodiments of the present application provide a computer program product comprising a computer program, wherein the computer program when executed by a processor may implement the method according to any of the embodiments described above.
Some embodiments of the present application provide a computer readable storage medium having stored thereon a computer program which, when executed by a processor, may implement a method as in any of the embodiments described above.
As shown in fig. 5, some embodiments of the present application provide an electronic device 500, which includes a memory 510, a processor 520, and a computer program stored on the memory 510 and executable on the processor 520, wherein the processor 520 reads the program from the memory 510 through a bus 530 and executes the program to implement the method according to any of the embodiments.
In the embodiments provided in the present application, it should be understood that the disclosed apparatus and method can be implemented in other ways. The apparatus embodiments described above are merely illustrative, and for example, the flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
In addition, functional modules in the embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.
The functions, if implemented in the form of software functional modules and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application or portions thereof that substantially contribute to the prior art may be embodied in the form of a software product stored in a storage medium and including instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
The above description is only an example of the present application and is not intended to limit the scope of the present application, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, improvement and the like made within the spirit and principle of the present application shall be included in the protection scope of the present application. It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, it need not be further defined and explained in subsequent figures.
The above description is only for the specific embodiments of the present application, but the scope of the present application is not limited thereto, and any person skilled in the art can easily conceive of the changes or substitutions within the technical scope of the present application, and shall be covered by the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.
It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
Claims (15)
1. A method of image processing, the method comprising:
obtaining a light spot weight map of a k frame according to M historical frame light spot detection maps and a light spot detection map of a k frame, wherein the M historical frame light spot detection maps are obtained by respectively carrying out light spot detection on M original images, the k frame light spot detection map is obtained by carrying out light spot detection on a k frame original image, the M original image is a frame or continuous multi-frame image which is positioned in front of the k frame original image and is adjacent to the k frame original image, and M is an integer greater than or equal to 1;
and obtaining a k frame target object light spot virtual rendering image according to the k frame light spot weight image and the k frame original image, wherein the k frame target object light spot virtual rendering image is the obtained target image.
2. The method of claim 1, wherein prior to said deriving a kth frame spot weight map from the M historical frame spot detection maps and the kth frame spot detection map, the method further comprises:
carrying out image segmentation on any one of the M frame original image and the k frame original image to obtain a background segmentation image;
obtaining a single-channel image based on the maximum value of the multiple channels of each pixel point in the background segmentation image;
and acquiring a light spot detection graph corresponding to any frame of original image at least according to the value of each pixel point on the single-channel image.
3. The method of claim 2, wherein the obtaining of the speckle detection map corresponding to any frame of the original image according to at least the pixel values of the pixels on the single-channel map comprises:
fuzzifying the single-channel image to obtain a fuzzy single-channel image;
and if the value of any pixel point on the single-channel graph is greater than a first threshold value and the value of any pixel point on the fuzzy single-channel graph is less than or equal to a second threshold value, determining that any pixel point belongs to a light spot, wherein the second threshold value is less than the first threshold value.
4. The method of any one of claims 1-3, wherein M is an integer greater than 1, wherein,
the obtaining of the kth frame light spot weight map according to the M historical frame light spot detection maps and the kth frame light spot detection map includes:
selecting X historical frame facula point detection maps from the M historical frame facula point detection maps, wherein the X historical frame facula point detection maps are adjacent to the k frame facula point detection map, and X is an integer which is greater than or equal to 1 and less than or equal to M;
fusing the X historical frame light spot detection graphs and the k frame light spot detection graph to obtain a k frame light spot fusion graph;
and obtaining the light spot point weight map of the kth frame according to the light spot point fusion map of the kth frame.
5. The method of claim 4, wherein said selecting X historical frame speckle detection maps from said M historical frame speckle detection maps comprises:
and selecting the X historical frame light spot detection maps from the M historical frame light spot detection maps according to the distribution information of the light spots on different frame original images.
6. The method of any of claims 4-5, wherein said selecting X historical frame speckle detection maps from among the M historical frame speckle detection maps comprises:
and if the difference value between the kth frame light spot detection diagram and the jth frame historical frame light spot detection diagram meets a difference threshold value, taking the jth frame to (k-1) th frame light spot detection diagrams as the X historical frame light spot detection diagrams, wherein the jth frame light spot detection diagram belongs to one frame of image in the M historical frame light spot detection diagrams.
7. The method of claim 6, wherein j is a value of
i is an integer greater than or equal to 0, wherein,
the confirming that a difference value between the kth frame light spot detection graph and the jth frame historical frame light spot detection graph meets a difference threshold value comprises:
and i, repeating the following process from 0 until the difference value obtained in the following way meets the difference threshold value:
calculating the k frame facula point detection chart and the
Detecting absolute values of differences of corresponding pixel values of all pixel points on the image by using the frame light spot points to obtain ith differences;
calculating the sum of the k frame light spot detection diagram and the k frame light spot detection diagram
The absolute value of the difference value of the pixel values corresponding to all the pixel points on the frame light spot detection image is obtained to obtain the (i + 1) th difference value;
calculating a difference between the ith difference and the (i + 1) th difference, and calculating a sum of the ith difference and the (i + 1) th difference;
obtaining the difference value according to the ratio of the difference to the sum;
wherein the operation symbol of "[ ]" represents the operation of rounding.
8. The method of any one of claims 4-7, wherein said selecting X historical frame speckle detection maps from among the M historical frame speckle detection maps further comprises:
and if the difference values between the k-frame light spot detection image and the j-frame historical frame light spot detection image do not meet a difference threshold value, taking the k-1-frame light spot detection image as the X historical frame light spot detection images, wherein the j-frame light spot detection image belongs to one of the M historical frame light spot detection images.
9. The method according to any one of claims 4-8, wherein said obtaining the k frame spot weight map from the k frame spot fusion map comprises:
performing curve stretching on the kth frame light spot point fusion graph to obtain a kth frame light spot point fusion stretching graph;
and obtaining the light spot point weight map of the kth frame according to the light spot point fusion stretch map of the kth frame.
10. The method of any one of claims 4 to 8,
the obtaining of the light spot weight map of the kth frame according to the light spot fusion map of the kth frame includes:
and smoothing the kth frame light spot point fusion image or a kth frame light spot point fusion stretching image obtained by stretching the kth frame light spot point fusion image to obtain the kth frame light spot point weight image.
11. The method of claim 10,
the step of smoothing the light spot point fusion map of the kth frame to obtain a light spot point weight map of the kth frame includes:
carrying out weighted averaging of different proportions on values of corresponding pixel points on a k-1 th frame light spot point fusion image and the k-1 th frame light spot point fusion image to obtain a weighted average value of each pixel point, and taking the weighted average value of each pixel point as a weight value of the corresponding pixel point on the k-1 th frame light spot point weight image; or
Carrying out weighted averaging of different proportions on values of corresponding pixel points on a k-1 th frame light spot point fused tensile graph and the k-1 th frame light spot point fused graph to obtain a weighted average value of each pixel point, and taking the weighted average value of each pixel point as a weight value of the corresponding pixel point on the k-1 th frame light spot point weight graph;
alternatively, the first and second electrodes may be,
the step of performing smoothing processing on the k frame light spot point fusion stretch graph obtained by stretching the k frame light spot point fusion graph to obtain the k frame light spot point weight graph comprises the following steps:
carrying out weighted averaging of different proportions on values of corresponding pixel points on a k-1 th frame light spot point fusion graph and the k-1 th frame light spot point fusion stretch graph to obtain a weighted average value of each pixel point, and taking the weighted average value of each pixel point as a weight value of the corresponding pixel point on the k-1 th frame light spot point weight graph; or
And carrying out weighted average of different proportions on the values of the same pixel points on the k-1 th frame light spot point fused and stretched graph and the k-1 th frame light spot point fused and stretched graph to obtain a weighted average value of each pixel point, and taking the weighted average value of each pixel point as the weight value of the corresponding pixel point on the k-1 th frame light spot point weight graph.
12. The method of any one of claims 1-11, further comprising: and repeating the process of obtaining the target image to obtain a target video, wherein the target video comprises a plurality of target images.
13. A computer program product, characterized in that the computer program product comprises a computer program, wherein the computer program when executed by a processor is adapted to perform the method of any of claims 1-12.
14. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, is adapted to carry out the method of any one of claims 1 to 12.
15. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor when executing the program is adapted to implement the method of any of claims 1-12.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111672323.1A CN114372932A (en) | 2021-12-31 | 2021-12-31 | Image processing method and computer program product |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111672323.1A CN114372932A (en) | 2021-12-31 | 2021-12-31 | Image processing method and computer program product |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114372932A true CN114372932A (en) | 2022-04-19 |
Family
ID=81141382
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111672323.1A Pending CN114372932A (en) | 2021-12-31 | 2021-12-31 | Image processing method and computer program product |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114372932A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115619678A (en) * | 2022-10-31 | 2023-01-17 | 锋睿领创(珠海)科技有限公司 | Image deformation correction method and device, computer equipment and storage medium |
| CN115984944A (en) * | 2023-01-20 | 2023-04-18 | 北京字跳网络技术有限公司 | Expression information identification method, device, equipment, readable storage medium and product |
-
2021
- 2021-12-31 CN CN202111672323.1A patent/CN114372932A/en active Pending
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115619678A (en) * | 2022-10-31 | 2023-01-17 | 锋睿领创(珠海)科技有限公司 | Image deformation correction method and device, computer equipment and storage medium |
| CN115619678B (en) * | 2022-10-31 | 2024-04-19 | 锋睿领创(珠海)科技有限公司 | Correction method and device for image deformation, computer equipment and storage medium |
| CN115984944A (en) * | 2023-01-20 | 2023-04-18 | 北京字跳网络技术有限公司 | Expression information identification method, device, equipment, readable storage medium and product |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Lv et al. | Attention guided low-light image enhancement with a large scale low-light simulation dataset | |
| Gur et al. | Single image depth estimation trained via depth from defocus cues | |
| CN110598610B (en) | Target significance detection method based on neural selection attention | |
| CN111741211B (en) | Image display method and apparatus | |
| CN104517110B (en) | The binarization method and system of a kind of image in 2 D code | |
| CN110889851B (en) | Robust use of semantic segmentation for depth and disparity estimation | |
| CN111292264A (en) | Image high dynamic range reconstruction method based on deep learning | |
| CN113888437A (en) | Image processing method, image processing device, electronic equipment and computer readable storage medium | |
| CN114372932A (en) | Image processing method and computer program product | |
| Xiao et al. | Single image dehazing based on learning of haze layers | |
| CN113505768A (en) | Model training method, face recognition method, electronic device and storage medium | |
| CN111127476A (en) | Image processing method, device, equipment and storage medium | |
| CN110033424A (en) | Method, apparatus, electronic equipment and the computer readable storage medium of image procossing | |
| CN112836625A (en) | Face living body detection method and device and electronic equipment | |
| CN113781370A (en) | Image enhancement method and device and electronic equipment | |
| CN113658197B (en) | Image processing method, device, electronic equipment and computer readable storage medium | |
| CN114708173A (en) | Image fusion method, computer program product, storage medium, and electronic device | |
| CN116664603B (en) | Image processing method, device, electronic equipment and storage medium | |
| CN114372931A (en) | Target object blurring method and device, storage medium and electronic equipment | |
| CN111738964A (en) | Image data enhancement method based on modeling | |
| Bhattacharya et al. | D2bgan: A dark to bright image conversion model for quality enhancement and analysis tasks without paired supervision | |
| CN115471413A (en) | Image processing method and device, computer readable storage medium and electronic device | |
| CN116263942A (en) | Method for adjusting image contrast, storage medium and computer program product | |
| Kang et al. | Specular highlight region restoration using image clustering and inpainting | |
| CN113706402A (en) | Neural network training method and device and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |