CN114119413A - Image processing method and device, readable medium and mobile terminal - Google Patents
Image processing method and device, readable medium and mobile terminal Download PDFInfo
- Publication number
- CN114119413A CN114119413A CN202111418051.2A CN202111418051A CN114119413A CN 114119413 A CN114119413 A CN 114119413A CN 202111418051 A CN202111418051 A CN 202111418051A CN 114119413 A CN114119413 A CN 114119413A
- Authority
- CN
- China
- Prior art keywords
- image
- target
- motion
- current frame
- brightness difference
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 26
- 238000012937 correction Methods 0.000 claims abstract description 66
- 238000000034 method Methods 0.000 claims abstract description 60
- 238000012545 processing Methods 0.000 claims abstract description 37
- 230000004927 fusion Effects 0.000 claims description 13
- 238000004590 computer program Methods 0.000 claims description 3
- 238000012935 Averaging Methods 0.000 claims description 2
- 238000004891 communication Methods 0.000 description 13
- 230000006870 function Effects 0.000 description 13
- 238000012986 modification Methods 0.000 description 9
- 230000004048 modification Effects 0.000 description 9
- 238000007726 management method Methods 0.000 description 8
- 238000010586 diagram Methods 0.000 description 7
- 230000002123 temporal effect Effects 0.000 description 6
- 238000001514 detection method Methods 0.000 description 4
- 238000007499 fusion processing Methods 0.000 description 4
- 238000010295 mobile communication Methods 0.000 description 4
- 238000013528 artificial neural network Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 238000004364 calculation method Methods 0.000 description 2
- 238000013500 data storage Methods 0.000 description 2
- 239000000835 fiber Substances 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 101150064138 MAP1 gene Proteins 0.000 description 1
- 230000001133 acceleration Effects 0.000 description 1
- 230000006978 adaptation Effects 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 238000013529 biological neural network Methods 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 210000000988 bone and bone Anatomy 0.000 description 1
- 210000004556 brain Anatomy 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 210000002569 neuron Anatomy 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 238000009877 rendering Methods 0.000 description 1
- 230000003252 repetitive effect Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 239000000126 substance Substances 0.000 description 1
- 238000012549 training Methods 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
- 238000013519 translation Methods 0.000 description 1
Images
Classifications
-
- G06T5/70—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/50—Image enhancement or restoration by the use of more than one image, e.g. averaging, subtraction
-
- G06T5/90—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20212—Image combination
- G06T2207/20221—Image fusion; Image merging
Abstract
The disclosure provides an image processing method, an image processing device, a computer readable medium and a mobile terminal, and relates to the technical field of image processing. The method comprises the following steps: acquiring a current frame image, and determining the maximum brightness difference value of the current frame image and a pre-set frame denoised image; determining a target motion state correction curve according to the current scene label of the current frame image and the current ambient light intensity; the target motion state correction curve is a corresponding relation curve of a maximum brightness difference value and a motion brightness difference value under the current scene label and the current ambient light intensity; determining a motion brightness difference value corresponding to the current frame image according to the target motion state correction curve; and fusing the denoised image of the current frame and the previously preset frame according to the motion brightness difference value to obtain a target denoised image. The technical scheme of the disclosure can perform denoising processing on the image under different ambient light conditions.
Description
Technical Field
The present disclosure relates to the field of image processing technologies, and in particular, to an image processing method, an image processing apparatus, a computer-readable medium, and a mobile terminal.
Background
With the rapid development of communication technology, mobile terminals have become an indispensable tool in people's life and also become a main medium for users to acquire and browse images.
However, images displayed in the mobile terminal may have different losses of brightness or color gamut under different viewing conditions of ambient light, which may reduce the comfort of human eyes in viewing the images.
Therefore, image processing such as denoising is performed on the image under different ambient light conditions, which becomes a necessary process for displaying the image of the mobile terminal.
Disclosure of Invention
An object of the present disclosure is to provide an image processing method, an image processing apparatus, a computer readable medium, and a mobile terminal, which provide an image processing method for denoising an image under different ambient light conditions.
According to a first aspect of the present disclosure, there is provided an image processing method including: acquiring a current frame image, and determining the maximum brightness difference value of the current frame image and a denoised image of a previous preset frame; determining a target motion state correction curve according to the current scene label of the current frame image and the current ambient light intensity; the target motion state correction curve is a corresponding relation curve of the maximum brightness difference value and the motion brightness difference value under the current scene label and the current ambient light intensity; determining the motion brightness difference corresponding to the current frame image according to the maximum brightness difference and the target motion state correction curve; and fusing the current frame image and the denoised image of the previous preset frame according to the motion brightness difference value to obtain a target denoised image.
According to a second aspect of the present disclosure, there is provided an image processing apparatus comprising: the brightness difference determining module is used for acquiring a current frame image and determining the maximum brightness difference between the current frame image and a pre-set frame denoised image; the target curve determining module is used for determining a target motion state correction curve according to the current scene label of the current frame image and the current ambient light intensity; the target motion state correction curve is a corresponding relation curve of the maximum brightness difference value and the motion brightness difference value under the current scene label and the current ambient light intensity; a motion difference value determining module, configured to determine the motion luminance difference value corresponding to the current frame image according to the maximum luminance difference value and the target motion state correction curve; and the image fusion module is used for fusing the denoised image of the current frame and the denoised image of the previous preset frame according to the motion brightness difference value to obtain a target denoised image.
According to a third aspect of the present disclosure, a computer-readable medium is provided, on which a computer program is stored, which computer program, when being executed by a processor, is adapted to carry out the above-mentioned method.
According to a fourth aspect of the present disclosure, there is provided a mobile terminal, comprising:
a processor; and
a memory for storing executable instructions of the processor;
wherein the processor is configured to perform the above-described method via execution of the executable instructions.
By the image processing method provided by the exemplary embodiment, in the image denoising process, the brightness errors in different scenes can be adjusted more accurately by combining the scene labels and the ambient light intensity; in addition, in the process of determining the target motion state correction curve, the corresponding relation between the maximum brightness difference value and the motion brightness difference value is included, wherein the motion brightness difference value is a brightness value related to the motion state, that is, the motion degree judgment is also combined in the process of brightness adjustment, so that the fuzzy condition of the image in the fusion process can be reduced, the image denoising effect under different ambient light conditions is improved, and the comfort degree of the human eyes for watching the image is improved.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and together with the description, serve to explain the principles of the disclosure. It is to be understood that the drawings in the following description are merely exemplary of the disclosure, and that other drawings may be derived from those drawings by one of ordinary skill in the art without the exercise of inventive faculty. In the drawings:
FIG. 1 illustrates a schematic diagram of an exemplary system architecture to which embodiments of the present disclosure may be applied;
FIG. 2 shows a schematic diagram of a mobile terminal to which embodiments of the present disclosure may be applied;
FIG. 3 schematically illustrates a flow chart of a method of image processing in an exemplary embodiment of the disclosure;
FIG. 4 is a block flow diagram schematically illustrating a temporal denoising process in an image processing method according to an exemplary embodiment of the present disclosure;
fig. 5 schematically illustrates a curve of the maximum luminance difference value and the motion luminance difference value under different ambient light intensities in a conventional scene in an exemplary embodiment of the disclosure;
fig. 6 schematically illustrates a corresponding relationship curve of a maximum luminance difference value and a motion luminance difference value under different ambient light intensities in a sky scene in an exemplary embodiment of the disclosure;
fig. 7 schematically illustrates a corresponding relationship curve of a maximum luminance difference value and a motion luminance difference value under different ambient light intensities in a face scene in an exemplary embodiment of the disclosure;
FIG. 8 is a partial block diagram schematically illustrating a temporal denoising process in an image processing apparatus according to an exemplary embodiment of the present disclosure;
fig. 9 schematically illustrates a block diagram of an image processing apparatus in an exemplary embodiment of the present disclosure.
Detailed Description
Example embodiments will now be described more fully with reference to the accompanying drawings. Example embodiments may, however, be embodied in many different forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.
Furthermore, the drawings are merely schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and thus their repetitive description will be omitted. Some of the block diagrams shown in the figures are functional entities and do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in the form of software, or in one or more hardware modules or integrated circuits, or in different networks and/or processor devices and/or microcontroller devices.
Fig. 1 is a schematic diagram illustrating a system architecture of an exemplary application environment to which an image processing method and apparatus according to an embodiment of the present disclosure may be applied.
As shown in fig. 1, the system architecture 100 may include one or more of terminals 101, 102, 103, a network 104, and a server 105. The network 104 is a medium used to provide communication links between the terminals 101, 102, 103 and the server 105. Network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, to name a few. The terminals 101, 102, 103 may be various terminals having image processing capabilities including, but not limited to, portable computers, smart phones, tablet computers, and the like. It should be understood that the number of terminals, networks, and servers in fig. 1 are merely illustrative. There may be any number of terminals, networks, and servers, as desired for an implementation. For example, server 105 may be a server cluster comprised of multiple servers, or the like.
The image processing method provided by the embodiment of the present disclosure is generally executed by the terminals 101, 102, 103, and accordingly, the image processing apparatus is generally provided in the terminals 101, 102, 103. However, it is easily understood by those skilled in the art that the image processing method provided in the embodiment of the present disclosure may also be executed by the server 105, and accordingly, the image processing apparatus may also be disposed in the server 105, which is not particularly limited in the exemplary embodiment. For example, in an exemplary embodiment, the terminals 101, 102, and 103 may calculate a maximum brightness difference from a denoised image of a previous preset frame according to an acquired current frame image, and then process the current frame image based on a target motion state correction curve according to a current scene label and a current ambient light intensity of the current frame image to obtain a target denoised image; the server 105 may also obtain the current frame image of the terminals 101, 102, and 103, and process the current frame image according to an image processing method to obtain the target denoising image, which is not particularly limited in the embodiment of the present disclosure.
An exemplary embodiment of the present disclosure provides a mobile terminal for implementing an image processing method, which may be the terminal 101, 102, 103 or the server 105 in fig. 1. The mobile terminal comprises at least a processor and a memory for storing executable instructions of the processor, the processor being configured to perform the image processing method via execution of the executable instructions.
The following takes the mobile terminal 200 in fig. 2 as an example, and exemplifies the configuration of the mobile terminal. In other embodiments, mobile terminal 200 may include more or fewer components than shown, or some components may be combined, some components may be split, or a different arrangement of components. The illustrated components may be implemented in hardware, software, or a combination of software and hardware. The interfacing relationship between the components is only schematically illustrated and does not constitute a structural limitation of the mobile terminal 200. In other embodiments, the mobile terminal 200 may also interface differently than shown in fig. 2, or a combination of multiple interfaces.
As shown in fig. 2, the mobile terminal 200 may specifically include: a processor 210, an internal memory 221, an external memory interface 222, a Universal Serial Bus (USB) interface 230, a charging management module 240, a power management module 241, a battery 242, an antenna 1, an antenna 2, a mobile communication module 250, a wireless communication module 260, an audio module 270, a speaker 271, a microphone 272, a microphone 273, an earphone interface 274, a sensor module 280, a display 290, a camera module 291, an indicator 292, a motor 293, a button 294, and a Subscriber Identity Module (SIM) card interface 295. Wherein the sensor module 280 may include a depth sensor 2801, a pressure sensor 2802, a gyroscope sensor 2803, and the like.
The NPU is a Neural-Network (NN) computing processor, which processes input information quickly by using a biological Neural Network structure, for example, by using a transfer mode between neurons of a human brain, and can also learn by itself continuously. The NPU can implement applications such as intelligent recognition of the mobile terminal 200, for example: image recognition, face recognition, speech recognition, text understanding, and the like.
A memory is provided in the processor 210. The memory may store instructions for implementing six modular functions: detection instructions, connection instructions, information management instructions, analysis instructions, data transmission instructions, and notification instructions, and execution is controlled by processor 210.
The charge management module 240 is configured to receive a charging input from a charger. The power management module 241 is used for connecting the battery 242, the charging management module 240 and the processor 210. The power management module 241 receives the input of the battery 242 and/or the charging management module 240, and supplies power to the processor 210, the internal memory 221, the display screen 290, the camera module 291, the wireless communication module 260, and the like.
The wireless communication function of the mobile terminal 200 may be implemented by the antenna 1, the antenna 2, the mobile communication module 250, the wireless communication module 260, a modem processor, a baseband processor, and the like. Wherein, the antenna 1 and the antenna 2 are used for transmitting and receiving electromagnetic wave signals; the mobile communication module 250 may provide a solution including wireless communication of 2G/3G/4G/5G, etc. applied to the mobile terminal 200; the modem processor may include a modulator and a demodulator;
the Wireless communication module 260 may provide a solution for Wireless communication applied to the mobile terminal 200, including Wireless Local Area Network (WLAN) (e.g., Wireless Fidelity (Wi-Fi) network), Bluetooth (BT), Ultra Wide Band (UWB) technology, and the like. In some embodiments, antenna 1 of the mobile terminal 200 is coupled to the mobile communication module 250 and antenna 2 is coupled to the wireless communication module 260, such that the mobile terminal 200 may communicate with networks and other devices via wireless communication techniques.
The mobile terminal 200 implements a display function through the GPU, the display screen 290, the application processor, and the like. The GPU is a microprocessor for image processing, and is connected to the display screen 290 and an application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 210 may include one or more GPUs that execute program instructions to generate or alter display information.
The mobile terminal 200 may implement a photographing function through the ISP, the camera module 291, the video codec, the GPU, the display screen 290, the application processor, and the like. The ISP is used for processing data fed back by the camera module 291; the camera module 291 is used for capturing still images or videos; the digital signal processor is used for processing digital signals, and can process other digital signals besides digital image signals; the video codec is used to compress or decompress digital video, and the mobile terminal 200 may also support one or more video codecs.
The external memory interface 222 may be used to connect an external memory card, such as a Micro SD card, to extend the memory capability of the mobile terminal 200. The external memory card communicates with the processor 210 through the external memory interface 222 to implement a data storage function. For example, files such as music, video, etc. are saved in an external memory card.
Internal memory 221 may be used to store computer-executable program code, which includes instructions. The internal memory 221 may include a program storage area and a data storage area. The storage program area may store an operating system, an application program (such as a sound playing function, an image playing function, etc.) required by at least one function, and the like. The storage data area may store data (e.g., audio data, a phonebook, etc.) created during use of the mobile terminal 200, and the like. In addition, the internal memory 221 may include a high-speed random access memory, and may further include a nonvolatile memory, such as at least one magnetic disk Storage device, a Flash memory device, a Universal Flash Storage (UFS), and the like. The processor 210 executes various functional applications of the mobile terminal 200 and data processing by executing instructions stored in the internal memory 221 and/or instructions stored in a memory provided in the processor.
The mobile terminal 200 may implement an audio function through the audio module 270, the speaker 271, the receiver 272, the microphone 273, the earphone interface 274, the application processor, and the like. Such as music playing, recording, etc.
The depth sensor 2801 is used to obtain depth information of the scene; the pressure sensor 2802 is used for sensing a pressure signal and converting the pressure signal into an electrical signal; the gyro sensor 2803 may be used to determine a motion gesture of the mobile terminal 200. In addition, other functional sensors, such as an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, etc., may be provided in the sensor module 280 according to actual needs. In the embodiment of the present disclosure, the ambient light sensor is used to detect an ambient light intensity value when the mobile terminal 200 takes a picture or previews a picture.
Other devices for providing auxiliary functions may also be included in mobile terminal 200. For example, the keys 294 include a power-on key, a volume key, and the like, and a user can generate key signal inputs related to user settings and function control of the mobile terminal 200 through key inputs. Further examples include indicator 292, motor 293, SIM card interface 295, etc.
In addition, the terminal 200 may further include an NFC module 296 and an auxiliary device of the NFC module, and is configured to perform near field communication with other devices. The NFC module 296 may perform frequency division or frequency multiplication on the input clock signal, then perform matching with the NFC antenna through the matching circuit, and convert the processed clock signal into a radio frequency signal for output.
The following specifically describes an image processing method and an image processing apparatus according to exemplary embodiments of the present disclosure.
The embodiment of the disclosure provides an image processing method, which can be applied to a mobile terminal with an image processing function. Referring to fig. 3, the image processing method may include the following steps S310 to S340:
in step S310, a current frame image is obtained, and a maximum brightness difference between the current frame image and a pre-set frame denoised image is determined.
In an exemplary embodiment of the present disclosure, the current frame image may be an existing image in the mobile terminal, and the current frame image may be acquired when the user needs to browse the existing image. Or, the current frame image may be obtained as the current frame image to be processed when the user uses the mobile terminal to take a picture. Still alternatively, the current frame image may be an image that has undergone other processing. For example, if the image denoising is divided into a spatial domain denoising part and a temporal domain denoising part, the process of obtaining the target denoised image from the current frame image provided by the embodiment of the present disclosure belongs to the temporal domain denoising, and then the current frame image may be an image that has been subjected to the spatial domain denoising. The present exemplary embodiment does not particularly limit the specific acquisition mode of the current frame image.
In the exemplary embodiment of the present disclosure, after the image is processed according to the image processing method provided by the embodiment of the present disclosure, the image may be stored in the Buffer. For example, at least the previously pre-set frame denoised image is stored in Buffer.
In practical applications, the denoised image of the previous preset frame and the image of the current frame may be images continuously shot in the same scene, adjacent images in the same video sequence, images selected by a user, and the like.
It should be noted that, if the image is not processed before the current frame image, the pre-specified image may be used as the pre-set frame denoised image. The number of the previous preset frames may be determined according to actual situations, for example, the previous frame, the previous two frames, the previous three frames, and the like, and the exemplary embodiment of the present disclosure is not particularly limited to the specific number of the previous preset frames.
In an exemplary embodiment of the present disclosure, after determining the denoised images of the current frame image and the previous preset frame, it is required to calculate a maximum brightness difference value between the denoised images of the current frame image and the previous preset frame. Taking the former two frames as an example, the maximum luminance difference value MotionMap1 can be the luminance value L of the current frame image curSnrcurSnrLuminance value L of denoised images preTnr1 and preTnr2 of the previous two framespreTnr1、LpreTnr2The maximum values after respective subtraction are shown in equation (1):
MotionMap1＝max(abs(LcurSnr-LpreTnr1),abs(LcurSnr-LpreTnr2)) (1)
it should be noted that, in the maximum brightness difference calculation process, the brightness value currsnr of the current frame image may be an average brightness value of the current frame image, that is, an overall brightness value of the current frame image; or the brightness value of each pixel in the current frame image, that is, the brightness value of the pixel in the current frame image, that is, the pixel in the current frame image is subjected to the denoising process separately.
Referring to fig. 4, a flow chart of a temporal denoising process in an image processing method according to an exemplary embodiment of the present disclosure is shown. The process of determining the maximum brightness difference may be performed in the Motion region detection module (Motion Map Cal)410 in fig. 4, where the Motion region detection module 410 acquires the brightness value L of the current frame image cusnrcurSnrAnd after the two previous frames of denoised images are preTnr1 and preTnr2, determining a maximum brightness difference value corresponding to the current frame of image for subsequent denoising processing.
In step S320, determining a target motion state correction curve according to the current scene label of the current frame image and the current ambient light intensity; the target motion state correction curve is a corresponding relation curve of the maximum brightness difference value and the motion brightness difference value under the current scene label and the current ambient light intensity.
In the exemplary embodiment of the present disclosure, in the process of determining the current scene tag of the current frame image, scene judgment may be performed on all regions of the current frame image, or only a target region of the current frame image may be subjected to scene judgment. The target area may be an area in which the image occupies a relatively large amount, or may be an area specified by the user and intended for comparison.
In practical applications, the scene is, for example, portrait, pet, baby, green, blue sky, beach, white cloud, spotlight, etc., and the corresponding scene tag may include: a face tag, a pet tag, a baby tag, a greenfield tag, a sky tag, a beach tag, a white cloud tag, a spotlight tag, etc., and the exemplary embodiments of the present disclosure are not particularly limited to the scene tag. In addition, in the scene tag determination process, the scene features may be extracted by using a conventional algorithm and determined in combination with a network training mode, and the determination process of the scene tag in the exemplary embodiment of the present disclosure is not repeated.
In the exemplary embodiment of the present disclosure, after the scene tag is determined, the denoising process in the embodiment of the present disclosure may be performed only on the target region corresponding to the scene tag, and the denoising process may not be performed on other regions of the current frame image. For example, a certain current frame image includes a blue sky and a face, and if the determined scene tag is a face tag, only the face region of the current frame image is denoised in the subsequent denoising process.
In practical application, a region frame can be set for a region indicated by a scene tag, so that a region to be subjected to denoising processing is positioned in a subsequent denoising processing process.
In an exemplary embodiment of the present disclosure, the current ambient light intensity Lux may be detected by an ambient light sensor on the mobile terminal 200.
After the current scene label and the current ambient light intensity of the current frame image are determined, a target motion state correction curve can be selected from a plurality of preset motion state correction curves. This process can be completed by the Curve modification module (Curve Adjust)420 in fig. 4, that is, the current Scene Tag and the current ambient light intensity Lux are input into the Curve modification module 420, and the Curve modification module 420 determines a target motion state modification Curve from a plurality of preset motion state modification curves.
In the exemplary embodiment of the present disclosure, the preset motion state correction curve is a corresponding relationship curve of the maximum luminance difference MotionMap1 and the motion luminance difference MotionMap under different scene tags and different ambient light intensities. The target motion state correction curve is a corresponding relation curve of the maximum brightness difference value MotionMap1 and the motion brightness difference value MotionMap under the current scene label and the current ambient light intensity.
It should be noted that the preset motion state modification curve is pre-stored in the curve modification module 420, or the curve modification module 420 may be directly called or accessed.
In an exemplary embodiment of the present disclosure, a specific manner of determining the preset motion state correction curve includes: firstly, acquiring video sequences in different scenes and different ambient light intensities and in different shooting states; wherein, the different shooting states may include: a stationary state, a uniform motion state, and a violent motion state. A video sequence herein refers to a plurality of frames of images arranged in order in a video.
Secondly, a reference motion state correction curve corresponding to the reference ambient light intensity in different scenes can be determined according to video sequences in different shooting states. If the reference ambient light intensity is Lux-50 nit, the multi-frame images in different shooting states under the reference ambient light intensity are experimentally debugged until an image with better brightness is obtained, and the corresponding relationship between the maximum brightness difference MotionMap1 and the motion brightness difference MotionMap can be represented by a curve in a curve mapping manner, as shown in fig. 5 to 7. The motion brightness difference MotionMap is a brightness value corresponding to an image with better brightness obtained after experimental debugging. The maximum luminance difference value MotionMap1 and the motion luminance difference value MotionMap are values corresponding to the same pixel if the unit of debugging is pixel.
In the process of determining the motion brightness difference MotionMap, the images in different shooting states are subjected to experimental debugging processing, so that the motion brightness difference MotionMap is related to the motion state and contains the brightness value of the motion characteristic. That is to say, the above time domain denoising process also combines the judgment of the motion degree, so that in the image processing process, the blur condition of the image in the fusion process can be reduced, the image denoising effect under different ambient light conditions can be improved, and the comfort level of the human eyes watching the image can be improved.
Fig. 5 shows a corresponding relationship between the maximum luminance difference MotionMap1 and the motion luminance difference MotionMap for different ambient light intensities Lux being 0, 50 and 100 nits in a conventional scene Normal case; wherein, the Normal scene Normal case refers to a scene without special processing in the room. Fig. 6 shows a corresponding relationship curve of the maximum luminance difference MotionMap1 and the motion luminance difference MotionMap for different ambient light intensities Lux of 0, 50 and 100 nits in the Sky scene Sky case. Fig. 7 shows a corresponding relationship curve of the maximum luminance difference MotionMap1 and the motion luminance difference MotionMap for different ambient light intensities Lux of 0, 50, and 100 nits in the Face scene case. It should be noted that different scenes correspond to different scene tags, a face scene corresponds to a face tag, and a sky scene corresponds to a sky tag, etc.
After obtaining the reference motion state correction curve corresponding to the reference ambient light intensity, for example, after obtaining the corresponding relationship curve between the maximum luminance difference MotionMap1 and the motion luminance difference MotionMap at Lux of 50 nit shown in fig. 5-7, the motion state correction curve at other ambient light intensities may be adjusted based on the corresponding relationship curve, that is, based on the reference motion state correction curve, to determine the preset motion state correction curve at other ambient light intensities.
Specifically, in the process of determining the preset motion state correction curve under other ambient light intensities, the maximum brightness difference value of the other ambient light intensities and the maximum brightness difference value of the reference ambient light intensity may be compared, and after the difference between the two maximum brightness difference values is determined, the reference motion state correction curve is adjusted based on the difference between the two maximum brightness difference values, so as to obtain the preset motion state correction curve under other ambient light intensities.
As shown in fig. 5, under the Normal scenario, taking Lux-50 nit as an example of the reference ambient light intensity, the difference between the maximum luminance difference value under other ambient light intensities Lux-0 and 100 nit and the maximum luminance difference value under Lux-50 nit is determined; and adjusting the reference motion state correction curve corresponding to the Lux-50 nit to obtain preset motion state correction curves under other ambient light intensities Lux-0 and 100 nit.
In a specific curve adjusting process, there may be a plurality of manners, for example, if a difference between a maximum luminance difference value at Lux ═ 0 nit and a maximum luminance difference value at Lux ═ 50 nit is-5 nit, a reference motion state correction curve corresponding to Lux ═ 50 nit may be translated to the left by 5 nit to obtain a preset motion state correction curve at Lux ═ 0 nit, and the like, where the translation manner is merely an example, and in the specific curve adjusting process, detailed adjustment may be performed according to each maximum luminance difference value, and the specific adjusting manner of the curve in the exemplary embodiment of the present disclosure is not particularly limited.
It should be noted that the above ambient light intensity Lux is 0, 50, and 100 nits, which is only an example, and the preset motion state correction curve under any ambient light intensity may be obtained by adjusting according to actual needs, and details are not repeated here.
As described above, after the preset motion state correction curves under different scenes and different ambient light intensities are determined, the target motion state correction curve can be selected from the plurality of preset motion state correction curves according to the current scene label of the current frame image and the current ambient light intensity.
In step S330, a motion luminance difference corresponding to the current frame image is determined according to the maximum luminance difference and the target motion state correction curve.
In the exemplary embodiment of the present disclosure, after the target Motion state correction curve is determined, according to the maximum luminance difference Motion Map1 between the current frame image and the pre-set frame de-noised image, the corresponding Motion luminance difference Motion Map is determined from the target Motion state correction curve.
In step S340, the current frame image and the pre-set frame denoised image are fused according to the motion brightness difference, so as to obtain a target denoised image.
After determining the Motion luminance difference Motion Map corresponding to the current frame image, the Motion region detection module 410 transmits the Motion luminance difference Motion Map to the fusion module 430, and the fusion module 430 also inputs the current frame image currsnr and the previously denoised frame image, for example, preTnr1 and preTnr 2. Then, the fusion module 430 fuses the current frame image curSnr and the pre-set frame denoised images preTnr1 and preTnr2 according to the motion brightness difference value, so as to obtain a target denoised image.
In an exemplary embodiment of the present disclosure, the manner of fusing the image by the fusion module 430 to obtain the target denoised image includes:
under the condition that the motion brightness difference value is 0, the brightness values of the current frame image curSnr and the denoised images preTnr1 and preTnr2 of the previous preset frame can be averaged to obtain a target denoised image; under the condition that the motion brightness difference is 255, only the current frame image curSnr is determined as the target de-noising image, that is, the brightness value of the target de-noising image is consistent with the brightness value of the current frame image curSnr.
Then, under the condition that the motion brightness difference value is between 0 and 255, the brightness interpolation can be carried out on the target denoising image under the condition that the motion brightness difference value is between 0 and 255 according to the target denoising image under the condition that the motion brightness difference value is between 0 and 255, and the corresponding target denoising image under the condition that the motion brightness difference value is between 0 and 255 is obtained. The interpolation method may be linear interpolation or other interpolation methods, and the exemplary embodiment of the present disclosure is not particularly limited in this respect.
In the interpolation process, weights of a current frame image curSnr and previously preset frame denoised images preTnr1 and preTnr2 corresponding to the motion brightness difference values of 0 and 255 are determined, and the weights are adjusted according to different motion intensity difference values to determine a corresponding target denoised image under the condition that the motion brightness difference value is between 0 and 255.
In the exemplary embodiment of the present disclosure, the process of fusing the images may be to fuse the entire image of the current frame image, or may be to fuse only the target area of the current frame image. In the specific fusion process, the fusion may be performed on a pixel-by-pixel basis, where each pixel in the target region is fused, or the target region may be subjected to overall averaging fusion, which is not particularly limited in the exemplary embodiment of the present disclosure.
Referring to fig. 4, after fusing the input current frame image currsnr with the pre-set frame denoised images preTnr1 and preTnr2, the fusion module 430 outputs a target denoised image tnrouut, and stores the output target denoised image tnrouut in Buffer for denoising the next frame image.
In summary, in the exemplary embodiment, in the image denoising process, by combining the scene label and the ambient light intensity, the brightness errors in different scenes can be adjusted more accurately; in addition, in the process of determining the target motion state correction curve, the corresponding relation between the maximum brightness difference value and the motion brightness difference value is included, wherein the motion brightness difference value is a brightness value related to the motion state, that is, the motion degree judgment is also combined in the process of brightness adjustment, so that the blurring condition of the image in the fusion process can be reduced, the miscellaneous points in the image are removed, the image denoising effect under different ambient light conditions is improved, and the comfort degree of human eyes for watching the image is improved.
The image processing method provided by the embodiment of the disclosure can perform spatial domain denoising, brightness enhancement, color enhancement, and the like according to actual conditions, besides performing the time domain denoising process. Specifically, the method may include: firstly, acquiring an image to be processed, and carrying out airspace denoising on the image to be processed to obtain an airspace denoised image; taking the space domain denoised image as a current frame image, and performing time domain denoising on the space domain denoised image according to the time domain denoising process to obtain a target denoised image; and then, performing brightness enhancement and/or color enhancement on the target de-noised image to obtain and output a target processing image.
If the above-mentioned brightness enhancement and color enhancement are performed on the target de-noised image at the same time, the brightness enhancement may be performed first and then the color enhancement may be performed, or the color enhancement may be performed first and then the brightness enhancement may be performed, which is not particularly limited in the exemplary embodiment of the present disclosure.
It should be noted that, the above-mentioned performing spatial domain denoising on the image to be processed, performing brightness enhancement on the target denoised image, and performing color enhancement on the brightness enhanced image may refer to the existing image processing method, and this is not described in detail in the exemplary embodiment of the present disclosure.
It is noted that the above-mentioned figures are merely schematic illustrations of processes involved in methods according to exemplary embodiments of the present disclosure, and are not intended to be limiting. It will be readily understood that the processes shown in the above figures are not intended to indicate or limit the chronological order of the processes. In addition, it is also readily understood that these processes may be performed synchronously or asynchronously, e.g., in multiple modules.
Further, referring to fig. 8, an image processing apparatus 800 applied to a mobile terminal having an image processing function is further provided in the present exemplary embodiment, and includes: a luminance difference determination module 810, a target curve determination module 820, a motion difference determination module 830, and an image fusion module 840. Wherein:
the brightness difference determining module 810 may be configured to obtain a current frame image, and determine a maximum brightness difference between the current frame image and a pre-set frame denoised image;
a target curve determining module 820, configured to determine a target motion state correction curve according to the current scene tag of the current frame image and the current ambient light intensity; the target motion state correction curve is a corresponding relation curve of a maximum brightness difference value and a motion brightness difference value under the current scene label and the current ambient light intensity;
the motion difference determining module 830, configured to determine a motion luminance difference corresponding to the current frame image according to the maximum luminance difference and the target motion state correction curve;
the image fusion module 840 may be configured to fuse the current frame image and the pre-set frame denoised image according to the motion brightness difference, so as to obtain a target denoised image.
In an exemplary embodiment, the target curve determining module 820 may be configured to select a target motion state correction curve from a plurality of preset motion state correction curves according to the current scene tag and the current ambient light intensity; the preset motion state correction curve is obtained by acquiring video sequences in different scenes and different ambient light intensities and in different shooting states; determining a reference motion state correction curve corresponding to reference ambient light intensity in different scenes according to video sequences in different shooting states; and determining preset motion state correction curves under other ambient light intensities by taking the reference motion state correction curve as a reference.
In an exemplary embodiment, the target curve determining module 820 may be configured to compare the maximum brightness difference of the other ambient light intensities with the maximum brightness difference of the reference ambient light intensity, and adjust the reference motion state correction curve to obtain the preset motion state correction curve at the other ambient light intensities.
In an exemplary embodiment, the image fusion module 840 may be configured to average luminance values of the current frame image and the pre-set frame denoised image when the motion luminance difference is 0, so as to obtain a target denoised image; under the condition that the motion brightness difference value is 255, determining the current frame image as a target de-noising image; and under the condition that the motion brightness difference value is between 0 and 255, performing brightness interpolation on the target denoising image under the conditions that the motion brightness difference value is between 0 and 255 to obtain the target denoising image.
In an exemplary embodiment, the target curve determining module 820 may be configured to perform scene judgment on a target area of the current frame image to determine a current scene tag of the current frame image.
In an exemplary embodiment, the image fusion module 840 may be configured to fuse the target region according to the motion brightness difference to obtain the target denoised image.
It should be noted that, in the image processing apparatus 800 provided in the present exemplary embodiment, the luminance difference value determining module 810 is disposed in the motion region detecting module 410, the target curve determining module 820 and the motion difference value determining module 830 are disposed in the curve modifying module 420, and the image fusing module 840 is disposed in the fusing module 430, corresponding to fig. 4.
In an exemplary embodiment, the image processing apparatus provided in the embodiments of the present disclosure, as shown in fig. 9, can be summarized in upper level as follows: a spatial domain denoising module 910, a temporal domain denoising module 920, a luminance enhancement module 930, and a color enhancement module 940; wherein the content of the first and second substances,
the spatial domain denoising module 910 may be configured to obtain an image to be processed, perform spatial domain denoising on the image to be processed, and obtain a spatial domain denoised image;
the time domain denoising module 920 may be configured to use the spatial domain denoised image as a current frame image to obtain a target denoised image;
a brightness enhancement module 930, configured to perform brightness enhancement on the target denoised image to obtain a brightness enhanced image;
and a color enhancement module 940, configured to perform color enhancement on the luminance enhanced image, obtain a color enhanced image, and output the color enhanced image.
The specific details of each module in the above apparatus have been described in detail in the method section, and details that are not disclosed may refer to the method section, and thus are not described again.
As will be appreciated by one skilled in the art, aspects of the present disclosure may be embodied as a system, method or program product. Accordingly, various aspects of the present disclosure may be embodied in the form of: an entirely hardware embodiment, an entirely software embodiment (including firmware, microcode, etc.) or an embodiment combining hardware and software aspects that may all generally be referred to herein as a "circuit," module "or" system.
Exemplary embodiments of the present disclosure also provide a computer-readable storage medium having stored thereon a program product capable of implementing the above-described method of the present specification. In some possible embodiments, various aspects of the disclosure may also be implemented in the form of a program product including program code for causing a terminal device to perform the steps according to various exemplary embodiments of the disclosure described in the above-mentioned "exemplary methods" section of this specification, when the program product is run on the terminal device, for example, any one or more of the steps in fig. 3 to 6 may be performed.
It should be noted that the computer readable media shown in the present disclosure may be computer readable signal media or computer readable storage media or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
In the present disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In contrast, in the present disclosure, a computer readable signal medium may comprise a propagated data signal with computer readable program code embodied therein, either in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.
Furthermore, program code for carrying out operations of the present disclosure may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., through the internet using an internet service provider).
Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. This application is intended to cover any variations, uses, or adaptations of the disclosure following, in general, the principles of the disclosure and including such departures from the present disclosure as come within known or customary practice within the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.
It will be understood that the present disclosure is not limited to the precise arrangements described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is to be limited only by the terms of the appended claims.
Claims (10)
1. An image processing method, comprising:
acquiring a current frame image, and determining the maximum brightness difference value of the current frame image and a denoised image of a previous preset frame;
determining a target motion state correction curve according to the current scene label of the current frame image and the current ambient light intensity; the target motion state correction curve is a corresponding relation curve of the maximum brightness difference value and the motion brightness difference value under the current scene label and the current ambient light intensity;
determining the motion brightness difference corresponding to the current frame image according to the maximum brightness difference and the target motion state correction curve;
and fusing the current frame image and the denoised image of the previous preset frame according to the motion brightness difference value to obtain a target denoised image.
2. The method of claim 1, wherein determining an object motion state correction curve according to the current scene label and the current ambient light intensity of the current frame image comprises:
selecting the target motion state correction curve from a plurality of preset motion state correction curves according to the current scene label and the current ambient light intensity;
wherein, the step of obtaining the preset motion state correction curve comprises the following steps:
acquiring video sequences in different scenes and different ambient light intensities and in different shooting states;
determining a reference motion state correction curve corresponding to the reference ambient light intensity in different scenes according to the video sequences in different shooting states;
and determining the preset motion state correction curve under other ambient light intensities by taking the reference motion state correction curve as a reference.
3. The method according to claim 2, wherein the determining the preset motion state correction curve at other ambient light intensities by taking the reference motion state correction curve as a reference comprises:
and comparing the maximum brightness difference of the other ambient light intensities with the maximum brightness difference of the reference ambient light intensity, and adjusting the reference motion state correction curve to obtain the preset motion state correction curve under the other ambient light intensities.
4. The method according to claim 1, wherein the fusing the denoised image of the current frame and the denoised image of the previous preset frame according to the motion brightness difference to obtain a target denoised image comprises:
under the condition that the motion brightness difference value is 0, averaging the brightness values of the denoised images of the current frame and the previous preset frame to obtain the target denoised image;
determining the current frame image as the target denoising image under the condition that the motion brightness difference value is 255;
and under the condition that the motion brightness difference value is between 0 and 255, performing brightness interpolation on the target denoising image under the conditions that the motion brightness difference value is between 0 and 255 to obtain the target denoising image.
5. The method according to any one of claims 1-4, further comprising:
and carrying out scene judgment on the target area of the current frame image to determine the current scene label of the current frame image.
6. The method as claimed in claim 5, wherein the fusing the denoised image of the current frame and the denoised image of the previous preset frame according to the motion brightness difference to obtain a target denoised image, further comprises:
and fusing the target area according to the motion brightness difference value to obtain the target denoising image.
7. The method of claim 1, wherein the obtaining the current frame image comprises:
acquiring an image to be processed, carrying out spatial domain denoising on the image to be processed to obtain a spatial domain denoised image,
taking the space domain denoised image as the current frame image;
the image processing method further includes:
and performing brightness enhancement and/or color enhancement on the target de-noised image to obtain and output a target processing image.
8. An image processing apparatus characterized by comprising:
the brightness difference determining module is used for acquiring a current frame image and determining the maximum brightness difference between the current frame image and a pre-set frame denoised image;
the target curve determining module is used for determining a target motion state correction curve according to the current scene label of the current frame image and the current ambient light intensity; the target motion state correction curve is a corresponding relation curve of the maximum brightness difference value and the motion brightness difference value under the current scene label and the current ambient light intensity;
a motion difference value determining module, configured to determine the motion luminance difference value corresponding to the current frame image according to the maximum luminance difference value and the target motion state correction curve;
and the image fusion module is used for fusing the denoised image of the current frame and the denoised image of the previous preset frame according to the motion brightness difference value to obtain a target denoised image.
9. A computer-readable medium, on which a computer program is stored which, when being executed by a processor, carries out the image processing method of any one of claims 1 to 7.
10. A mobile terminal, comprising:
a processor; and
a memory for storing executable instructions of the processor;
wherein the processor is configured to perform the image processing method of any of claims 1 to 7 via execution of the executable instructions.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111418051.2A CN114119413A (en) | 2021-11-25 | 2021-11-25 | Image processing method and device, readable medium and mobile terminal |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111418051.2A CN114119413A (en) | 2021-11-25 | 2021-11-25 | Image processing method and device, readable medium and mobile terminal |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114119413A true CN114119413A (en) | 2022-03-01 |
Family
ID=80373848
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111418051.2A Pending CN114119413A (en) | 2021-11-25 | 2021-11-25 | Image processing method and device, readable medium and mobile terminal |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114119413A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115002516A (en) * | 2022-04-18 | 2022-09-02 | 北京旷视科技有限公司 | System, method, electronic device, storage medium, and program product for video processing |
-
2021
- 2021-11-25 CN CN202111418051.2A patent/CN114119413A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115002516A (en) * | 2022-04-18 | 2022-09-02 | 北京旷视科技有限公司 | System, method, electronic device, storage medium, and program product for video processing |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111179282B (en) | Image processing method, image processing device, storage medium and electronic apparatus | |
| CN111598776B (en) | Image processing method, image processing device, storage medium and electronic apparatus | |
| CN111866483B (en) | Color restoration method and device, computer readable medium and electronic device | |
| CN110706310B (en) | Image-text fusion method and device and electronic equipment | |
| CN107690804B (en) | Image processing method and user terminal | |
| CN112562019A (en) | Image color adjusting method and device, computer readable medium and electronic equipment | |
| CN113706414B (en) | Training method of video optimization model and electronic equipment | |
| CN111741303B (en) | Deep video processing method and device, storage medium and electronic equipment | |
| CN111161176B (en) | Image processing method and device, storage medium and electronic equipment | |
| CN113705665B (en) | Training method of image transformation network model and electronic equipment | |
| KR20200117695A (en) | Electronic device and method for controlling camera using external electronic device | |
| CN112954251A (en) | Video processing method, video processing device, storage medium and electronic equipment | |
| CN113706440A (en) | Image processing method, image processing device, computer equipment and storage medium | |
| CN116113976A (en) | Image processing method and device, computer readable medium and electronic equipment | |
| CN110807769B (en) | Image display control method and device | |
| CN117274109B (en) | Image processing method, noise reduction model training method and electronic equipment | |
| CN113902636A (en) | Image deblurring method and device, computer readable medium and electronic equipment | |
| CN113658065A (en) | Image noise reduction method and device, computer readable medium and electronic equipment | |
| CN114119413A (en) | Image processing method and device, readable medium and mobile terminal | |
| CN113066020A (en) | Image processing method and device, computer readable medium and electronic device | |
| CN112967193A (en) | Image calibration method and device, computer readable medium and electronic equipment | |
| CN111447360A (en) | Application program control method and device, storage medium and electronic equipment | |
| US20220245920A1 (en) | Object display method and apparatus, electronic device, and computer readable storage medium | |
| CN113362260A (en) | Image optimization method and device, storage medium and electronic equipment | |
| CN112950516A (en) | Method and device for enhancing local contrast of image, storage medium and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |