CN111738951B - Image processing method and device - Google Patents
Image processing method and device Download PDFInfo
- Publication number
- CN111738951B CN111738951B CN202010572461.1A CN202010572461A CN111738951B CN 111738951 B CN111738951 B CN 111738951B CN 202010572461 A CN202010572461 A CN 202010572461A CN 111738951 B CN111738951 B CN 111738951B
- Authority
- CN
- China
- Prior art keywords
- image
- resolution
- super
- frame
- brightness
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 20
- 238000012545 processing Methods 0.000 claims abstract description 37
- 238000000034 method Methods 0.000 claims abstract description 34
- 230000009467 reduction Effects 0.000 claims description 17
- 230000006870 function Effects 0.000 claims description 12
- 230000008569 process Effects 0.000 claims description 9
- 230000003595 spectral effect Effects 0.000 claims description 5
- 238000001228 spectrum Methods 0.000 claims description 5
- 230000008439 repair process Effects 0.000 abstract description 6
- 238000004590 computer program Methods 0.000 description 7
- 238000010586 diagram Methods 0.000 description 7
- 238000004891 communication Methods 0.000 description 6
- 230000006835 compression Effects 0.000 description 5
- 238000007906 compression Methods 0.000 description 5
- 230000003287 optical effect Effects 0.000 description 5
- 238000005516 engineering process Methods 0.000 description 3
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 239000013307 optical fiber Substances 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 239000004065 semiconductor Substances 0.000 description 2
- 238000012935 Averaging Methods 0.000 description 1
- 241000023320 Luma <angiosperm> Species 0.000 description 1
- 230000004075 alteration Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 230000015572 biosynthetic process Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- JJWKPURADFRFRB-UHFFFAOYSA-N carbonyl sulfide Chemical compound O=C=S JJWKPURADFRFRB-UHFFFAOYSA-N 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 230000035622 drinking Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 239000011159 matrix material Substances 0.000 description 1
- OSWPMRLSEDHDFF-UHFFFAOYSA-N methyl salicylate Chemical compound COC(=O)C1=CC=CC=C1O OSWPMRLSEDHDFF-UHFFFAOYSA-N 0.000 description 1
- 238000003825 pressing Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000003786 synthesis reaction Methods 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/70—Denoising; Smoothing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/77—Retouching; Inpainting; Scratch removal
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/90—Dynamic range modification of images or parts thereof
- G06T5/92—Dynamic range modification of images or parts thereof based on global image properties
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10016—Video; Image sequence
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10024—Color image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20024—Filtering details
Landscapes
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Image Processing (AREA)
Abstract
The present disclosure provides an image processing method, an image processing apparatus, an electronic device, and a computer-readable storage medium. The method comprises the following steps: acquiring a frame of image from a video; when the resolution of the image is larger than the preset resolution, denoising the image to obtain a denoised image; when the resolution of the image is smaller than or equal to the preset resolution, performing super-resolution reconstruction on the image to obtain a high-resolution image corresponding to the image; and performing color restoration on the high-resolution image or the denoising image to obtain a final image. The embodiment of the disclosure is suitable for renewing old videos, and a user does not need to manually operate, so that the video repair is more intelligent and quicker.
Description
Technical Field
The present disclosure relates to the field of image processing technologies, and in particular, to an image processing method, an image processing apparatus, and a computer readable storage medium.
Background
Film art is an important representation of human culture and a valuable wealth of human civilization. Some of the losses due to noise, compression and the like lead to the conditions of film picture blurring, dense noise points, ripple and the like, so that the film quality is poor, and the film needs to be repaired at the moment.
At present, the repair of movies is mostly performed manually, which is time-consuming and labor-consuming.
Disclosure of Invention
This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.
The technical problem to be solved by the present disclosure is to provide an image processing method, so as to at least partially solve the technical problem of time and effort consumption in manual repair in the prior art. Further, an image processing apparatus, an image processing hardware apparatus, a computer-readable storage medium, and an image processing terminal are provided.
In order to achieve the above object, according to one aspect of the present disclosure, there is provided the following technical solutions:
an image processing method, comprising:
acquiring a frame of image from a video;
when the resolution of the image is larger than the preset resolution, denoising the image to obtain a denoised image;
when the resolution of the image is smaller than or equal to the preset resolution, performing super-resolution reconstruction on the image to obtain a high-resolution image corresponding to the image;
And performing color restoration on the high-resolution image or the denoising image to obtain a final image.
In order to achieve the above object, according to one aspect of the present disclosure, there is provided the following technical solutions:
an image processing apparatus comprising:
the image acquisition module is used for acquiring a frame of image from the video;
the noise reduction processing module is used for carrying out noise reduction processing on the image to obtain a denoising image when the resolution of the image is larger than the preset resolution;
the super-resolution reconstruction module is used for performing super-resolution reconstruction on the image when the resolution of the image is smaller than or equal to the preset resolution, so as to obtain a high-resolution image corresponding to the image;
and the color restoration module is used for carrying out color restoration on the high-resolution image or the denoising image to obtain a final image.
In order to achieve the above object, according to one aspect of the present disclosure, there is provided the following technical solutions:
an electronic device, comprising:
a memory for storing non-transitory computer readable instructions; and
and a processor for executing the computer readable instructions such that the processor, when executing, implements the image processing method described above.
In order to achieve the above object, according to one aspect of the present disclosure, there is provided the following technical solutions:
a computer readable storage medium storing non-transitory computer readable instructions which, when executed by a computer, cause the computer to perform the above-described image processing method.
In order to achieve the above object, according to still another aspect of the present disclosure, there is further provided the following technical solutions:
an image processing terminal includes any one of the image processing apparatuses described above.
According to the embodiment of the disclosure, a frame of image is obtained from a video, when the resolution of the image is larger than the preset resolution, the image is subjected to noise reduction treatment to obtain a denoising image, when the resolution of the image is smaller than or equal to the preset resolution, the image is subjected to super-resolution reconstruction to obtain a high-resolution image corresponding to the image, and the high-resolution image or the denoising image is subjected to color restoration to obtain a final image, so that the method is suitable for renovating an old video, and a user is not required to manually operate, so that video restoration is more intelligent and faster.
The foregoing description is only an overview of the disclosed technology, and may be implemented in accordance with the disclosure of the present disclosure, so that the above-mentioned and other objects, features and advantages of the present disclosure can be more clearly understood, and the following detailed description of the preferred embodiments is given with reference to the accompanying drawings.
Drawings
The above and other features, advantages, and aspects of embodiments of the present disclosure will become more apparent by reference to the following detailed description when taken in conjunction with the accompanying drawings. The same or similar reference numbers will be used throughout the drawings to refer to the same or like elements. It should be understood that the figures are schematic and that elements and components are not necessarily drawn to scale.
FIG. 1 is a flow diagram of an image processing method according to one embodiment of the present disclosure;
FIG. 2 is a flow diagram of an image processing apparatus according to one embodiment of the present disclosure;
fig. 3 is a schematic structural view of an electronic device according to an embodiment of the present disclosure.
Detailed Description
Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While certain embodiments of the present disclosure have been shown in the accompanying drawings, it is to be understood that the present disclosure may be embodied in various forms and should not be construed as limited to the embodiments set forth herein, but are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustration purposes only and are not intended to limit the scope of the present disclosure.
It should be understood that the various steps recited in the method embodiments of the present disclosure may be performed in a different order, or in parallel. Furthermore, method embodiments may include additional steps or omit performing the illustrated steps. The scope of the present disclosure is not limited in this respect.
The term "including" and variations thereof as used herein are intended to be open-ended, i.e., including, but not limited to. The term "based on" is based at least in part on. The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments. Related definitions of other terms will be given in the description below.
Example 1
In order to solve the technical problems of time and labor consumption of a plurality of manual repairs in the prior art, an embodiment of the disclosure provides an image processing method. As shown in fig. 1, the image processing method mainly includes the following steps S11 to S14.
Step S11: a frame of image is acquired from the video.
Wherein the video may be a short video or a movie.
Specifically, basic configuration information of a video is first acquired, the basic configuration information including the number of frames per second transmission (Frames Per Secondfps, FPS) of pictures. And performing frame extraction processing on the video according to the FPS to obtain a frame-by-frame image. In addition, the basic configuration information also comprises a playing format, a playing proportion, a pressing format, a code rate, a resolution and the like.
Before the video is frame-extracted, saw-tooth noise and water drinking ripple noise can be removed.
Step S12: and when the resolution of the image is larger than the preset resolution, carrying out noise reduction on the image to obtain a denoising image.
The preset resolution may be 360p or 720p. In particular, the resolution may be obtained from the underlying configuration information.
Step S13: and when the resolution of the image is smaller than or equal to the preset resolution, performing super-resolution reconstruction on the image to obtain a high-resolution image corresponding to the image.
The high-resolution image is an image with resolution exceeding the preset resolution.
Step S14: and performing color restoration on the high-resolution image or the denoising image to obtain a final image.
For each frame of image in the video, the processing from step S11 to step S14 is carried out, and then the image to video compression and the video synthesis are carried out according to the basic configuration information.
According to the embodiment, one frame of image is obtained from the video, when the resolution of the image is larger than the preset resolution, the image is subjected to noise reduction treatment to obtain a denoising image, when the resolution of the image is smaller than or equal to the preset resolution, the image is subjected to super-resolution reconstruction to obtain a high-resolution image corresponding to the image, and the high-resolution image or the denoising image is subjected to color restoration to obtain a final image, so that the method is suitable for renovating an old video, and a user does not need to manually operate, so that the video restoration is more intelligent and faster.
In a first alternative embodiment, step S12 specifically includes:
step S121: and when the resolution of the image is larger than the preset resolution, determining the power spectrum density of Gaussian noise of the image.
Where gaussian noise refers to a type of noise whose probability density function follows a gaussian distribution (i.e., normal distribution). Common gaussian noise includes heave noise, cosmic noise, thermal noise, shot noise, and the like.
Step S122: and determining the blurring degree of the image according to the power spectral density of the Gaussian noise.
Step S123: and carrying out self-adaptive denoising on the image according to the blurring degree to obtain a denoised image.
Different blur degrees correspond to different denoising parameters, and the image can be denoised in a targeted manner.
In a second alternative embodiment, step S13 specifically includes:
step S131: and acquiring m frames of images positioned in front of the image and n frames of images positioned behind the image.
Wherein the values of m and n can be determined according to the position of the image in the low-resolution video or film. For example, when the image is the first frame image in a low resolution video or movie, the value of m is 0, and the value of n is a preset integer, for example, 1 or 2. For another example, when the image is the last frame image in a low resolution video or movie, the value of m is a preset integer, for example, 1 or 2, and the value of n is 0. For another example, when the image is a low resolution video or an intermediate frame image in a movie, the value of m is a preset integer, for example, 1 or 2, and the value of n is a preset integer, for example, 1 or 2.
Step S132: and inputting the image, the m-frame image and the n-frame image into a super-resolution reconstruction network.
The super-resolution reconstruction network is used for carrying out high-resolution reconstruction on the image. In this step, the image, the m-frame image and the n-frame image are not subjected to downsampling, that is, the original size of the image is kept and input into a super-resolution reconstruction network for high-resolution reconstruction, so that the texture information of the picture can be increased, and the content of the picture is richer.
Step S133: and in the super-resolution reconstruction network, performing high-resolution reconstruction according to the image, the m-frame image and the n-frame image to obtain a high-resolution image corresponding to the image.
For example, feature fusion is performed on the image, the m-frame image and the n-frame image, so as to obtain a high-resolution image corresponding to the image.
In a third alternative embodiment, step S133 specifically includes:
step S1331: and in the super-resolution reconstruction network, performing high-resolution reconstruction according to the image, the m-frame image and the n-frame image to obtain an initial high-resolution image.
Step S1332: and in the super-resolution reconstruction network, determining the smoothness among pixels in the initial high-resolution image until the smoothness meets the smoothness constraint condition among the pixels, ending the high-resolution reconstruction process, and obtaining a final high-resolution image.
In this embodiment, by setting the smoothness constraint condition, it is possible to maintain the smoothness of the reconstructed high-resolution image, eliminate artifacts that may be caused by the high-resolution image, and adapt to video data in which various noise interferences exist.
In a fourth alternative embodiment, step S14 specifically includes:
step S141: and determining the original brightness of each pixel point in the high-resolution image or the denoising image.
Specifically, the luminance component of each pixel in the high resolution image or the denoising image may be extracted, and the original luminance may be determined according to the luminance component of each pixel. For example, if the high resolution image or the denoising image is a Red Green Blue (RGB) image, the high resolution image or the denoising image may be converted into an HSL image or a YUV image or a LAB image by color space conversion.
If the high resolution image or the denoising image is converted into an HSV image, where H is a Hue (Hue) component, S is a Saturation (Saturation) component, and L is a brightness (lighting) component, in this embodiment, only the L component is extracted, and the original brightness of each pixel point in the image is determined according to the L component.
If the high resolution image or the denoising image is converted into a YUV image, where Y represents brightness (luminence or Luma), i.e., a Luminance component, and U and V represent chromaticity (Chroma or Chroma), i.e., a Chrominance component, in this embodiment, only the Y component is extracted, and the original brightness of each pixel point in the image is determined according to the Y component.
If the high resolution image or the denoised image is converted into a LAB image, where L represents luminance (luminance), i.e. a luminance component, a represents a range from magenta to green and B represents a range from yellow to blue. In this embodiment, only the L component is extracted, and the original brightness of each pixel point in the image is determined according to the L component.
Step S142: and determining the average brightness and the maximum brightness of the high-resolution image or the denoising image according to the original brightness of each pixel point in the high-resolution image or the denoising image.
Specifically, the average brightness of the high-resolution image or the denoising image may be obtained by directly averaging the original brightness of all the pixel points, or may be obtained by using the method in the fifth alternative embodiment described below.
Step S143: and determining the final brightness of each pixel point according to the original brightness of each pixel point, the average brightness and the maximum brightness.
Specifically, the method of the step is adopted to calculate each pixel point in the high-resolution image or the denoising image one by one, so as to obtain the final brightness of each pixel point. The specific calculation method is referred to in the sixth alternative embodiment described below, and will not be described here again.
Step S144: and obtaining a final image according to the final brightness of each pixel point.
In a fifth alternative embodiment, step S142 specifically includes: by the formulaDetermining the high resolution image or the de-noised imageAverage brightness of (2); wherein L (x, y) is the original brightness of the pixel point (x, y), -/->For the average luminance, σ is a constant, m×n is the width and height of the high resolution image or the denoising image, exp () is an exponential function based on a natural constant e, and log () is a logarithmic operation.
In a sixth alternative embodiment, step S144 specifically includes: by the formulaDetermining the final brightness of each pixel point; wherein L is max For the maximum luminance, L is the final luminance, and L (x, y) is the original luminance of the pixel point (x, y).
In a seventh alternative embodiment, the method further comprises:
step S15: and obtaining the pressed format of the video.
Specifically, the compression format of the video may be obtained from the basic configuration information.
The compression format may be a reverse tape (IVTC) compression format or Parr (Phase Alteration Line, PAL).
Repairing video of de-interlacing method;
step S16: and when the pressed format is interlace scanning, de-interlacing the video in an anti-interlacing mode.
The de-interlacing mode eliminates interlacing and restores to the original frame rate by two processes, field Match and de-duplication frame (de-matrix).
In an eighth alternative embodiment, the method further comprises: and when the coding format of the video is the old coding format, transcoding the video to obtain the video with the latest coding format.
For example, the old encoding format may be MPEG, h.261, h.263, h.264, etc., and the latest encoding format may be h.265.
It will be appreciated by those skilled in the art that obvious modifications (e.g., overlapping of enumerated modes) or equivalent substitutions may be made on the basis of the various embodiments described above.
In the foregoing, although the steps in the embodiments of the image processing method are described in the above order, it should be clear to those skilled in the art that the steps in the embodiments of the present disclosure are not necessarily performed in the above order, but may be performed in reverse order, parallel, cross, etc., and other steps may be further added to those skilled in the art on the basis of the above steps, and these obvious modifications or equivalent manners are also included in the protection scope of the present disclosure and are not repeated herein.
The following is an embodiment of the disclosed apparatus, which may be used to perform steps implemented by an embodiment of the disclosed method, and for convenience of explanation, only those portions relevant to the embodiment of the disclosed method are shown, and specific technical details are not disclosed, referring to the embodiment of the disclosed method.
Example two
In order to solve the technical problems of time and labor consumption of a plurality of manual repairs in the prior art, an embodiment of the disclosure provides an image processing device. The apparatus may perform the steps of the image processing method embodiment described in the first embodiment. As shown in fig. 2, the apparatus mainly includes: an image acquisition module 21, a noise reduction processing module 22, a super resolution reconstruction module 23 and a color restoration module 24; wherein,
The image acquisition module 21 is used for acquiring a frame of image from the video;
the noise reduction processing module 22 is configured to perform noise reduction processing on the image to obtain a denoised image when the resolution of the image is greater than the preset resolution;
the super-resolution reconstruction module 23 is configured to perform super-resolution reconstruction on the image when the resolution of the image is less than or equal to a preset resolution, so as to obtain a high-resolution image corresponding to the image;
the color restoration module 24 is configured to perform color restoration on the high resolution image or the denoising image, so as to obtain a final image.
Further, the noise reduction processing module 22 includes: when the resolution of the image is larger than the preset resolution, determining the power spectrum density of Gaussian noise of the image; determining the blurring degree of the image according to the power spectral density of the Gaussian noise; and carrying out self-adaptive denoising on the image according to the blurring degree to obtain a denoised image.
Further, the super resolution reconstruction module 23 includes: an image frame acquisition unit 231, an image frame input unit 232, and a super resolution reconstruction unit 233; wherein,
the image frame acquisition unit 231 is configured to acquire m frame images located before the image and n frame images located after the image;
The image frame input unit 232 is configured to input the image, the m-frame image, and the n-frame image into a super-resolution reconstruction network;
the super-resolution reconstruction unit 233 is configured to perform high-resolution reconstruction according to the image, the m-frame image, and the n-frame image in the super-resolution reconstruction network, so as to obtain a high-resolution image corresponding to the image.
Further, the super-resolution reconstruction unit 233 is specifically configured to: in the super-resolution reconstruction network, performing high-resolution reconstruction according to the image, the m-frame image and the n-frame image to obtain an initial high-resolution image; and in the super-resolution reconstruction network, determining the smoothness among pixels in the initial high-resolution image until the smoothness meets the smoothness constraint condition among the pixels, ending the high-resolution reconstruction process, and obtaining a final high-resolution image.
Further, the color restoration module 24 includes: an original luminance determining unit 241, an average luminance determining unit 242, a final luminance determining unit 243, and an image reconstructing unit 244; wherein,
the original brightness determining unit 241 is configured to determine an original brightness of each pixel point in the high resolution image or the denoising image;
The average brightness determining unit 242 is configured to determine an average brightness and a maximum brightness of the high resolution image or the denoising image according to an original brightness of each pixel point in the high resolution image or the denoising image;
the final brightness determining unit 243 is configured to determine a final brightness of each pixel point according to the original brightness, the average brightness and the maximum brightness of each pixel point;
the image reconstruction unit 244 is configured to obtain a final image according to the final brightness of each pixel.
Further, the average brightness determining unit 242 is specifically configured to: by the formulaDetermining an average brightness of the high resolution image or the denoised image; wherein L (x, y) is the original brightness of the pixel point (x, y), -/->For the average luminance, σ is a constant, m×n is the width and height of the high resolution image or the denoising image, exp () is an exponential function based on a natural constant e, and log () is a logarithmic operation.
Further, the image reconstruction unit 244 is specifically configured to: by the formulaDetermining the final brightness of each pixel point; wherein L is max For the maximum luminance, L is the final luminance, and L (x, y) is the original luminance of the pixel point (x, y).
Further, the device further comprises: a de-interlacing processing module 25; wherein,
the de-interlacing processing module 25 is configured to obtain a compressed format of the video; and when the pressed format is interlace scanning, de-interlacing the video in an anti-interlacing mode.
Further, the device further comprises: a transcoding module 26; wherein,
the transcoding module 26 is configured to transcode the video to obtain the video in the latest encoding format when the encoding format of the video is the old encoding format.
For detailed descriptions of the working principles, the technical effects, etc. of the embodiments of the image processing apparatus, reference may be made to the related descriptions in the foregoing embodiments of the image processing method, which are not repeated herein.
Example III
Referring now to fig. 3, a schematic diagram of an electronic device 300 suitable for use in implementing embodiments of the present disclosure is shown. The terminal devices in the embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and the like, and stationary terminals such as digital TVs, desktop computers, and the like. The electronic device shown in fig. 3 is merely an example and should not be construed to limit the functionality and scope of use of the disclosed embodiments.
As shown in fig. 3, the electronic device 300 may include a processing means (e.g., a central processing unit, a graphics processor, etc.) 301 that may perform various suitable actions and processes in accordance with a program stored in a Read Only Memory (ROM) 302 or a program loaded from a storage means 308 into a Random Access Memory (RAM) 303. In the RAM 303, various programs and data required for the operation of the electronic apparatus 300 are also stored. The processing device 301, the ROM 302, and the RAM 303 are connected to each other via a bus 304. An input/output (I/O) interface 305 is also connected to bus 304.
In general, the following devices may be connected to the I/O interface 305: input devices 306 including, for example, a touch screen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; an output device 307 including, for example, a Liquid Crystal Display (LCD), a speaker, a vibrator, and the like; storage 308 including, for example, magnetic tape, hard disk, etc.; and communication means 309. The communication means 309 may allow the electronic device 300 to communicate with other devices wirelessly or by wire to exchange data. While fig. 3 shows an electronic device 300 having various means, it is to be understood that not all of the illustrated means are required to be implemented or provided. More or fewer devices may be implemented or provided instead.
In particular, according to embodiments of the present disclosure, the processes described above with reference to flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a non-transitory computer readable medium, the computer program comprising program code for performing the method shown in the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network via a communication device 309, or installed from a storage device 308, or installed from a ROM 302. The above-described functions defined in the methods of the embodiments of the present disclosure are performed when the computer program is executed by the processing means 301.
It should be noted that the computer readable medium described in the present disclosure may be a computer readable signal medium or a computer readable storage medium, or any combination of the two. The computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples of the computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this disclosure, a computer-readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present disclosure, however, the computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, with the computer-readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: electrical wires, fiber optic cables, RF (radio frequency), etc., or any suitable superposition of the above.
In some implementations, the clients, servers may communicate using any currently known or future developed network protocol, such as HTTP (HyperText Transfer Protocol ), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), the internet (e.g., the internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future developed networks.
The computer readable medium may be contained in the electronic device; or may exist alone without being incorporated into the electronic device.
The computer readable medium carries one or more programs which, when executed by the electronic device, cause the electronic device to: acquiring a frame of image from a video; when the resolution of the image is larger than the preset resolution, denoising the image to obtain a denoised image; when the resolution of the image is smaller than or equal to the preset resolution, performing super-resolution reconstruction on the image to obtain a high-resolution image corresponding to the image; and performing color restoration on the high-resolution image or the denoising image to obtain a final image.
Computer program code for carrying out operations of the present disclosure may be written in one or more programming languages, including but not limited to an object oriented programming language such as Java, smalltalk, C ++ and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of remote computers, the remote computer may be connected to the user computer through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (e.g., connected through the internet using an internet service provider).
The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The units involved in the embodiments of the present disclosure may be implemented by means of software, or may be implemented by means of hardware. The name of the unit does not in any way constitute a limitation of the unit itself, for example the first acquisition unit may also be described as "unit acquiring at least two internet protocol addresses".
The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: a Field Programmable Gate Array (FPGA), an Application Specific Integrated Circuit (ASIC), an Application Specific Standard Product (ASSP), a system on a chip (SOC), a Complex Programmable Logic Device (CPLD), and the like.
In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable superposition of the above. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable overlay of the foregoing.
According to one or more embodiments of the present disclosure, there is provided an image processing method including:
acquiring a frame of image from a video;
when the resolution of the image is larger than the preset resolution, denoising the image to obtain a denoised image;
when the resolution of the image is smaller than or equal to the preset resolution, performing super-resolution reconstruction on the image to obtain a high-resolution image corresponding to the image;
and performing color restoration on the high-resolution image or the denoising image to obtain a final image.
Further, when the resolution of the image is greater than the preset resolution, performing noise reduction processing on the image to obtain a denoised image, including:
when the resolution of the image is larger than the preset resolution, determining the power spectrum density of Gaussian noise of the image;
determining the blurring degree of the image according to the power spectral density of the Gaussian noise;
and carrying out self-adaptive denoising on the image according to the blurring degree to obtain a denoised image.
Further, when the resolution of the image is less than or equal to a preset resolution, performing super-resolution reconstruction on the image to obtain a high-resolution image corresponding to the image, including:
Acquiring m frames of images in front of the image and n frames of images behind the image;
inputting the image, the m-frame image and the n-frame image into a super-resolution reconstruction network;
and in the super-resolution reconstruction network, performing high-resolution reconstruction according to the image, the m-frame image and the n-frame image to obtain a high-resolution image corresponding to the image.
Further, in the super-resolution reconstruction network, performing high-resolution reconstruction according to the image, the m-frame image, and the n-frame image to obtain a high-resolution image corresponding to the image, including:
in the super-resolution reconstruction network, performing high-resolution reconstruction according to the image, the m-frame image and the n-frame image to obtain an initial high-resolution image;
and in the super-resolution reconstruction network, determining the smoothness among pixels in the initial high-resolution image until the smoothness meets the smoothness constraint condition among the pixels, ending the high-resolution reconstruction process, and obtaining a final high-resolution image.
Further, the performing color restoration on the high resolution image or the denoising image to obtain a final image includes:
Determining the original brightness of each pixel point in the high-resolution image or the denoising image;
determining average brightness and maximum brightness of the high-resolution image or the denoising image according to the original brightness of each pixel point in the high-resolution image or the denoising image;
determining the final brightness of each pixel point according to the original brightness, the average brightness and the maximum brightness of each pixel point;
and obtaining a final image according to the final brightness of each pixel point.
Further, the determining the average brightness of the high resolution image or the denoising image according to the original brightness of each pixel point includes:
by the formulaDetermining an average brightness of the high resolution image or the denoised image; wherein L (x, y) is the original brightness of the pixel point (x, y), -/->For the average luminance, σ is a constant, m×n is the width and height of the high resolution image or the denoising image, exp () is an exponential function based on a natural constant e, and log () is a logarithmic operation.
Further, the determining the final brightness of each pixel according to the original brightness, the average brightness and the maximum brightness of each pixel includes:
By the formulaDetermining the final brightness of each pixel point; wherein L is max For the maximum luminance, L is the final luminance, and L (x, y) is the original luminance of the pixel point (x, y).
Further, the method further comprises:
obtaining a pressed format of the video;
and when the pressed format is interlace scanning, de-interlacing the video in an anti-interlacing mode.
Further, the method further comprises:
and when the coding format of the video is the old coding format, transcoding the video to obtain the video with the latest coding format.
According to one or more embodiments of the present disclosure, there is provided an image processing apparatus including:
the image acquisition module is used for acquiring a frame of image from the video;
the noise reduction processing module is used for carrying out noise reduction processing on the image to obtain a denoising image when the resolution of the image is larger than the preset resolution;
the super-resolution reconstruction module is used for performing super-resolution reconstruction on the image when the resolution of the image is smaller than or equal to the preset resolution, so as to obtain a high-resolution image corresponding to the image;
and the color restoration module is used for carrying out color restoration on the high-resolution image or the denoising image to obtain a final image.
Further, the noise reduction processing module includes: when the resolution of the image is larger than the preset resolution, determining the power spectrum density of Gaussian noise of the image; determining the blurring degree of the image according to the power spectral density of the Gaussian noise; and carrying out self-adaptive denoising on the image according to the blurring degree to obtain a denoised image.
Further, the super-resolution reconstruction module includes:
an image frame acquisition unit configured to acquire an m-frame image located before the image and an n-frame image located after the image;
the image frame input unit is used for inputting the image, the m-frame image and the n-frame image into a super-resolution reconstruction network;
and the super-resolution reconstruction unit is used for carrying out high-resolution reconstruction according to the image, the m-frame image and the n-frame image in the super-resolution reconstruction network to obtain a high-resolution image corresponding to the image.
Further, the super-resolution reconstruction unit is specifically configured to: in the super-resolution reconstruction network, performing high-resolution reconstruction according to the image, the m-frame image and the n-frame image to obtain an initial high-resolution image; and in the super-resolution reconstruction network, determining the smoothness among pixels in the initial high-resolution image until the smoothness meets the smoothness constraint condition among the pixels, ending the high-resolution reconstruction process, and obtaining a final high-resolution image.
Further, the color repair module includes:
an original brightness determining unit, configured to determine an original brightness of each pixel point in the high resolution image or the denoising image;
an average brightness determining unit, configured to determine an average brightness and a maximum brightness of the high-resolution image or the denoising image according to an original brightness of each pixel point in the high-resolution image or the denoising image;
a final brightness determining unit, configured to determine a final brightness of each pixel point according to the original brightness, the average brightness, and the maximum brightness of each pixel point;
and the image reconstruction unit is used for obtaining a final image according to the final brightness of each pixel point.
Further, the average brightness determining unit is specifically configured to: by the formulaDetermining an average brightness of the high resolution image or the denoised image; wherein L (x, y) is the original brightness of the pixel point (x, y), -/->For the average luminance, σ is a constant, m×n is the width and height of the high resolution image or the denoising image, exp () is an exponential function based on a natural constant e, and log () is a logarithmic operation.
Further, the image reconstruction unit is specifically configured to: by the formula Determining the final brightness of each pixel point; wherein L is max For the maximum brightness, L is the final brightness, L (x, y) is the pixel point(x, y) raw luminance.
Further, the device further comprises:
the de-interlacing processing module is used for acquiring the pressed format of the video;
and when the pressed format is interlace scanning, de-interlacing the video in an anti-interlacing mode.
Further, the device further comprises:
and the transcoding module is used for transcoding the video when the encoding format of the video is the old encoding format, so as to obtain the video with the latest encoding format.
According to one or more embodiments of the present disclosure, there is provided an electronic device including:
a memory for storing non-transitory computer readable instructions; and
and a processor configured to execute the computer readable instructions, such that the processor performs the image processing method described above.
According to one or more embodiments of the present disclosure, there is provided a computer-readable storage medium storing non-transitory computer-readable instructions that, when executed by a computer, cause the computer to perform the above-described image processing method.
The foregoing description is only of the preferred embodiments of the present disclosure and description of the principles of the technology being employed. It will be appreciated by those skilled in the art that the scope of the disclosure referred to in this disclosure is not limited to the specific arrangements of the technical features described above, but also encompasses other arrangements of the technical features described above, or their equivalents, in any manner without departing from the spirit of the disclosure. Such as those described above, are mutually substituted with the technical features having similar functions disclosed in the present disclosure (but not limited thereto).
Moreover, although operations are depicted in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order. In certain circumstances, multitasking and parallel processing may be advantageous. Likewise, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of the present disclosure. Certain features that are described in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcompositions.
Although the subject matter has been described in language specific to structural features or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are example forms of implementing the claims.
Claims (10)
1. An image processing method, comprising:
acquiring a frame of image from a video;
when the resolution of the image is larger than the preset resolution, carrying out noise reduction on the image to obtain a denoising image;
when the resolution of the image is smaller than or equal to the preset resolution, performing super-resolution reconstruction on the image to obtain a high-resolution image corresponding to the image;
performing color restoration on the high-resolution image or the denoising image to obtain a final image;
when the resolution of the image is smaller than or equal to a preset resolution, performing super-resolution reconstruction on the image to obtain a high-resolution image corresponding to the image, wherein the method comprises the following steps:
acquiring m frames of images in front of the image and n frames of images behind the image;
inputting the image, the m-frame image and the n-frame image into a super-resolution reconstruction network;
In the super-resolution reconstruction network, performing high-resolution reconstruction according to the image, the m-frame image and the n-frame image to obtain an initial high-resolution image;
and in the super-resolution reconstruction network, determining the smoothness among pixels in the initial high-resolution image until the smoothness meets the smoothness constraint condition among the pixels, ending the high-resolution reconstruction process, and obtaining a final high-resolution image.
2. The method according to claim 1, wherein when the resolution of the image is greater than the preset resolution, performing noise reduction processing on the image to obtain a denoised image, includes:
when the resolution of the image is larger than the preset resolution, determining the power spectrum density of Gaussian noise of the image;
determining the blurring degree of the image according to the power spectral density of the Gaussian noise;
and carrying out self-adaptive denoising on the image according to the blurring degree to obtain a denoised image.
3. The method of claim 1, wherein color repairing the high resolution image or the de-noised image to obtain a final image comprises:
determining the original brightness of each pixel point in the high-resolution image or the denoising image;
Determining average brightness and maximum brightness of the high-resolution image or the denoising image according to the original brightness of each pixel point in the high-resolution image or the denoising image;
determining the final brightness of each pixel point according to the original brightness, the average brightness and the maximum brightness of each pixel point;
and obtaining a final image according to the final brightness of each pixel point.
4. A method according to claim 3, wherein determining the average luminance of the high resolution image or the denoised image from the original luminance of each pixel comprises:
by the formulaDetermining an average brightness of the high resolution image or the denoised image; wherein L (x, y) is the original brightness of the pixel point (x, y), -/->For the average luminance, σ is a constant, m×n is the width and height of the high resolution image or the denoising image, exp () is an exponential function based on a natural constant e, and log () is a logarithmic operation.
5. The method of claim 4, wherein determining the final luminance of each pixel based on the original luminance, the average luminance, and the maximum luminance of each pixel comprises:
By the formulaDetermining the final brightness of each pixel point; wherein L is max For the maximum luminance, L is the final luminance, and L (x, y) is the original luminance of the pixel point (x, y).
6. The method according to any one of claims 1-5, further comprising:
obtaining a pressed format of the video;
and when the pressed format is interlace scanning, de-interlacing the video in an anti-interlacing mode.
7. The method according to any one of claims 1-5, further comprising:
and when the coding format of the video is the old coding format, transcoding the video to obtain the video with the latest coding format.
8. An image processing apparatus, comprising:
the image acquisition module is used for acquiring a frame of image from the video;
the noise reduction processing module is used for carrying out noise reduction processing on the image to obtain a denoising image when the resolution of the image is larger than a preset resolution;
the super-resolution reconstruction module is used for performing super-resolution reconstruction on the image when the resolution of the image is smaller than or equal to the preset resolution, so as to obtain a high-resolution image corresponding to the image;
The color restoration module is used for carrying out color restoration on the high-resolution image or the denoising image to obtain a final image;
wherein, super resolution rebuild module includes: the device comprises an image frame acquisition unit, an image frame input unit and a super-resolution reconstruction unit;
the image frame acquisition unit is used for acquiring m frames of images in front of the image and n frames of images behind the image;
the image frame input unit is used for inputting the image, the m-frame image and the n-frame image into a super-resolution reconstruction network;
the super-resolution reconstruction unit is used for: in the super-resolution reconstruction network, performing high-resolution reconstruction according to the image, the m-frame image and the n-frame image to obtain an initial high-resolution image; and in the super-resolution reconstruction network, determining the smoothness among pixels in the initial high-resolution image until the smoothness meets the smoothness constraint condition among the pixels, ending the high-resolution reconstruction process, and obtaining a final high-resolution image.
9. An electronic device, comprising:
a memory for storing non-transitory computer readable instructions; and
a processor for executing the computer readable instructions such that the processor, when executed, implements the image processing method according to any of claims 1-7.
10. A computer readable storage medium storing non-transitory computer readable instructions which, when executed by a computer, cause the computer to perform the image processing method of any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010572461.1A CN111738951B (en) | 2020-06-22 | 2020-06-22 | Image processing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010572461.1A CN111738951B (en) | 2020-06-22 | 2020-06-22 | Image processing method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111738951A CN111738951A (en) | 2020-10-02 |
| CN111738951B true CN111738951B (en) | 2024-03-15 |
Family
ID=72650195
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010572461.1A Active CN111738951B (en) | 2020-06-22 | 2020-06-22 | Image processing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111738951B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112950465A (en) * | 2021-01-26 | 2021-06-11 | 北京小米移动软件有限公司 | Video super-resolution processing method, video super-resolution processing device and storage medium |
| CN114972021A (en) * | 2022-04-13 | 2022-08-30 | 北京字节跳动网络技术有限公司 | Image processing method and device, electronic equipment and storage medium |
| CN116055778B (en) * | 2022-05-30 | 2023-11-21 | 荣耀终端有限公司 | Video data processing method, electronic device and readable storage medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104520897A (en) * | 2012-08-06 | 2015-04-15 | 皇家飞利浦有限公司 | Image noise reduction and/or image resolution improvement |
| CN107274347A (en) * | 2017-07-11 | 2017-10-20 | 福建帝视信息科技有限公司 | A kind of video super-resolution method for reconstructing based on depth residual error network |
| CN108961186A (en) * | 2018-06-29 | 2018-12-07 | 赵岩 | A kind of old film reparation recasting method based on deep learning |
-
2020
- 2020-06-22 CN CN202010572461.1A patent/CN111738951B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104520897A (en) * | 2012-08-06 | 2015-04-15 | 皇家飞利浦有限公司 | Image noise reduction and/or image resolution improvement |
| CN107274347A (en) * | 2017-07-11 | 2017-10-20 | 福建帝视信息科技有限公司 | A kind of video super-resolution method for reconstructing based on depth residual error network |
| CN108961186A (en) * | 2018-06-29 | 2018-12-07 | 赵岩 | A kind of old film reparation recasting method based on deep learning |
Non-Patent Citations (1)
| Title |
|---|
| 基于色调映射的快速低照度图像增强;陈迎春;《计算机工程与应用》;第56卷(第09期);第234-239页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111738951A (en) | 2020-10-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111580765B (en) | Screen projection method, screen projection device, storage medium, screen projection equipment and screen projection equipment | |
| CN112419151B (en) | Image degradation processing method and device, storage medium and electronic equipment | |
| CN111738951B (en) | Image processing method and device | |
| US20210274224A1 (en) | Selectively enhancing compressed digital content | |
| RU2740034C1 (en) | Linear encoder for image / video processing | |
| US20200404309A1 (en) | Video watermark adding method and apparatus, and electronic device and storage medium | |
| CN107431770B (en) | Method, system, device and medium for adaptive linear luma domain video pipeline architecture | |
| JP7359521B2 (en) | Image processing method and device | |
| US11538136B2 (en) | System and method to process images of a video stream | |
| CN110889809B (en) | Image processing method and device, electronic equipment and storage medium | |
| CN113962859B (en) | Panorama generation method, device, equipment and medium | |
| CN113784175A (en) | HDR video conversion method, device, equipment and computer storage medium | |
| CN111696039B (en) | Image processing method and device, storage medium and electronic equipment | |
| US10674163B2 (en) | Color space compression | |
| CN112686810A (en) | Image processing method and device | |
| CN111738950B (en) | Image processing method and device | |
| CN112261417B (en) | Video pushing method and system, equipment and readable storage medium | |
| CN114467298A (en) | Image signal conversion processing method and device and terminal equipment | |
| CN115278189A (en) | Image tone mapping method and apparatus, computer readable medium and electronic device | |
| CN111526366B (en) | Image processing method, image processing apparatus, image capturing device, and storage medium | |
| CN113613024A (en) | Video preprocessing method and device | |
| CN117692679B (en) | Quality enhancement video transmission method, device, electronic equipment and readable medium | |
| CN114266696B (en) | Image processing method, apparatus, electronic device, and computer-readable storage medium | |
| CN112911186B (en) | Image storage method and device, electronic equipment and storage medium | |
| CN117097913A (en) | Video coding method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |