WO2020078102A1

WO2020078102A1 - Image enhancement method and apparatus, and computer-readable storage medium

Info

Publication number: WO2020078102A1
Application number: PCT/CN2019/102020
Authority: WO
Inventors: 陈宇聪; 闻兴; 郑云飞; 陈敏; 王晓楠; 蔡砚刚; 黄跃; 于冰
Original assignee: 北京达佳互联信息技术有限公司
Priority date: 2018-10-17
Filing date: 2019-08-22
Publication date: 2020-04-23
Also published as: CN109544490A; CN109544490B

Abstract

The present application relates to an image enhancement method and apparatus, and a computer-readable storage medium. The image enhancement method comprises: extracting an edge image from an original image to obtain a first edge image; performing adaptive feature processing on the first edge image to obtain a second edge image; and linearly superposing the original image and the second edge image to obtain an edge-enhanced image. In the image enhancement method, adaptive feature processing is performed on the first edge image to obtain the second edge image. Pixel points and noise of the first edge image are adaptively distinguished, thereby avoiding incorrect processing of noise pixel points of the first edge image, and further having a better anti-noise effect; moreover, the problem of power consumption caused by using a super-resolution algorithm based on deep learning and machine learning is solved.

Description

Image enhancement method, device and computer readable storage medium

Cross-reference of related applications

This application requires the priority of the Chinese patent application filed on October 17, 2018 with the Chinese Patent Office, the application number is 201811207500.7, and the application name is "image enhancement method, device, and computer-readable storage medium", the entire contents of which are incorporated by reference In this application.

Technical field

This application belongs to the field of computer software applications, especially image enhancement methods, devices, and computer-readable storage media.

Background technique

Image enhancement is to use a series of techniques to improve the quality and visual effects of the image, highlight the features of interest in the image, and obtain valuable information in the image, thereby transforming the image into a more suitable for human or machine analysis and processing The form makes the processed image better for some specific applications. Image enhancement theory is widely used in the fields of biomedicine, industrial production, public safety, and aerospace.

In the related art, the image enhancement method uses a super-resolution technology, that is, up-converting the input image into a high-resolution output image. Among them, the super-resolution algorithm based on deep learning and machine learning trains the super-resolution model through a large number of low-resolution images and high-resolution image samples to achieve super-resolution image enhancement effects. The image enhancement method can also achieve the subjective image enhancement effect of equivalent high resolution through the traditional filtering algorithm without changing the objective resolution of the image. In the process of image enhancement, the noise and blur of the image need to be considered at the same time to achieve a clear and natural high-resolution visual effect.

The inventor realized that the super-resolution algorithm based on deep learning and machine learning has a large calculation amount, is not suitable for use on mobile platforms such as mobile phones, and has a large power consumption problem. At the same time, the inventor found that none of the above image enhancement methods considered the noise of the pixels of the edge image, and erroneously processed the noise pixels of the edge image, causing image glitches.

Summary of the invention

In order to overcome the problems in the related art, the present application discloses an image enhancement method and device, which performs adaptive feature processing on the first edge image extracted from the original image to obtain a second edge image to achieve better The anti-noise image enhancement effect also solves the power consumption problem.

According to a first aspect of the embodiments of the present application, an image enhancement method is provided, including:

Extract the edge image from the original image to get the first edge image;

Performing adaptive feature processing on the first edge image to obtain a second edge image;

Linearly superimposing the original image and the second edge image to obtain an edge enhanced image.

According to a second aspect of the embodiments of the present application, an image enhancement device is provided, including:

Edge extraction unit: configured to extract the edge image from the original image to obtain the first edge image;

Adaptive feature processing unit: configured to perform adaptive feature processing on the first edge image to obtain a second edge image;

Adder unit: linearly superimpose the original image and the second edge image to obtain an edge-enhanced image.

According to a third aspect of the embodiments of the present application, an image enhancement device is provided, including:

processor;

Memory for storing processor executable instructions;

Wherein, the processor is configured to perform any one of the image enhancement methods described above.

According to a fourth aspect of the embodiments of the present application, a computer-readable storage medium is provided. The computer-readable storage medium stores computer instructions, and when the computer instructions are executed, the above image enhancement method is implemented.

The technical solutions provided by the embodiments of the present application may include the following beneficial effects:

In this image enhancement method, the first edge image is subjected to adaptive feature processing to obtain a second edge image, and the pixels and noise of the first edge image are adaptively distinguished to avoid noise on the first edge image Pixels are processed in error, so as to have better anti-noise effect, and at the same time, solve the power consumption problem caused by the use of super-resolution algorithms of deep learning and machine learning.

BRIEF DESCRIPTION

Fig. 1 is a flowchart of an image enhancement method according to an exemplary embodiment;

Fig. 2 is a flowchart of an image enhancement method according to an exemplary embodiment;

Fig. 3 is a flowchart of an image enhancement method according to an exemplary embodiment;

Fig. 4 is a schematic diagram of an image enhancement device according to an exemplary embodiment;

Fig. 5 is a block diagram of a device for performing an image enhancement method according to an exemplary embodiment;

Fig. 6 is a block diagram of a device for performing an image enhancement method according to an exemplary embodiment.

detailed description

Exemplary embodiments will be described in detail here, examples of which are shown in the drawings. Fig. 1 is a flowchart of an image enhancement method according to an exemplary embodiment, and specifically includes the following steps:

In step S101, an edge image is extracted from the original image to obtain a first edge image.

In step S102, adaptive feature processing is performed on the first edge image to obtain a second edge image.

In step S103, the original image and the second edge image are linearly superimposed to obtain an edge-enhanced image.

In an embodiment of the present application, first, an edge image is extracted from the original image to obtain a first edge image. Then, adaptive feature processing is performed on the first edge image to obtain a second edge image. Wherein, the adaptive feature processing is to perform adaptive feature compensation on the time domain feature and the frequency domain feature of the first edge image. Finally, the original image and the second edge image are linearly superimposed to obtain an edge-enhanced image.

According to an embodiment of the present application, adaptive feature processing is performed on the first edge image to obtain a second edge image. Adaptively distinguish the pixels and noises of the first edge image, avoid erroneous processing of the noise pixels of the first edge image, have better anti-noise effect, and at the same time, solve 1. The power consumption problem caused by the super-resolution algorithm of machine learning.

Fig. 2 is a flowchart of an image enhancement method according to an exemplary embodiment, and specifically includes the following steps:

In step S201, an edge image is extracted from the original image to obtain a first edge image.

In step S202, the neighborhood noise estimate of each pixel of the first edge image is calculated separately.

In step S203, the neighborhood noise estimates of each pixel of the first edge image are superimposed to obtain a second edge image.

In step S204, the original image and the second edge image are linearly superimposed to obtain an edge-enhanced image.

In step S205, the edge-enhanced image is output.

In an embodiment of the present application, first, an edge image is extracted from the original image to obtain a first edge image. Then, the neighborhood noise estimates of each pixel of the first edge image are calculated separately. Secondly, the neighborhood noise estimates of each pixel of the first edge image are superimposed to obtain the second edge image. Again, the original image and the second edge image are linearly superimposed to obtain an edge-enhanced image. Finally, the edge-enhanced image is output.

According to an embodiment of the present application, the neighborhood noise estimate of each pixel of the first edge image is calculated separately. The neighborhood noise estimates of each pixel of the first edge image are superimposed to obtain the second edge image. The adaptive feature processing of the edge image requires less computation, is suitable for use on mobile platforms such as mobile phones, and has lower power consumption.

In an optional embodiment, the formula for superimposing the neighborhood noise estimate of each pixel of the first edge image to obtain the second edge image is:

among them,

Is the second edge image, P is the first edge image, k, a, b are parameter constants, and η is the neighborhood noise estimate. The formula for calculating the neighborhood noise estimate of each pixel of the first edge image separately is:

Where x is the pixel set of the first edge image, x ₀ is a central pixel in the first edge image, x _ij is the pixel around x ₀ , and i is the pixel x _{ij is} in the first The abscissa of an edge image, j is the ordinate of the pixel x _ij in the first edge image.

The neighborhood relationship is the position relationship of adjacent pixels in the image. Each pixel of the first edge image can be used as a central pixel. The separately calculating the neighborhood noise estimate of each pixel of the first edge image is to calculate the absolute value of the error between the pixel of the center pixel of the first edge image and the pixel of the neighborhood position And, the difference between the error sum of the pixel of the center pixel of the first edge image and the pixel of the pixel at the neighborhood position.

In an optional embodiment, the neighborhood of the neighborhood noise estimate includes a neighborhood relationship of at least one of the following neighborhood relationships: 4 neighborhoods, D neighborhoods, and 8 neighborhoods.

In an optional embodiment, the coordinate of any pixel in the first edge image is (h, k). There are four adjacent pixels at the 4 neighborhood positions of the pixels, usually represented by N ₄ (p), and their coordinates are: (h + 1, k), (h-1, k), (h, k + 1) and (h, k-1). There position D at the neighborhood of the four pixel points corresponding to the vertex pixels, usually expressed in N _D (p), which coordinates are: (h + 1, k + 1), (h + 1, k -1), (h-1, k + 1) and (h-1, k-1). The pixel point at the 8-neighborhood position of the pixel point, usually represented by N ₈ (p), is the four adjacent pixel points at the 4-neighborhood position of the pixel point and the D of the pixel point at the four vertices positions corresponding to the neighborhood pixels, and, i.e. _{_{N 8 (p) = N 4}} (p) + N D (p).

In an alternative embodiment, an edge detection operator is used to extract the edge image from the original image. The edge detection operator includes at least one of the following detection operators: a Laplacian Gaussian operator, a Roberts operator, and a Sobel operator. Among them, the Laplacian Gaussian operator (LoG) first performs Gaussian convolution filtering on the original image to reduce noise, and then uses the Laplacian operator for edge detection. Roberts operator is a kind of operator that uses local difference operator to find the edge of image. The difference between the pixels of two adjacent pixels in the diagonal direction is used to approximate the gradient amplitude to detect the image edge of the original image. The Roberts operator detects vertical edges better than diagonal edges and has high positioning accuracy, but it is sensitive to noise and cannot suppress the effects of noise. The Sobel operator (sobel) weights the difference in gray values of the pixels at the four upper, lower, left, and right neighborhoods of each pixel in the original image. The weighted difference reaches the extreme at the image edge of the original image. Value for edge detection.

Fig. 3 is a flowchart of an image enhancement method according to an exemplary embodiment, and specifically includes the following steps:

In step S301, the low-resolution image is up-sampled to obtain an up-sampled high-resolution image.

In step S302, extract an edge image from the up-sampled high-resolution image to obtain a third edge image.

In step S303, the neighborhood noise estimate of each pixel of the third edge image is calculated separately.

In step S304, the neighborhood noise estimates of each pixel of the third edge image are superimposed to obtain a fourth edge image.

In step S305, the up-sampled high-resolution image and the fourth edge image are linearly superimposed to obtain an edge-enhanced image.

In step S306, the edge-enhanced image is output.

Image resolution refers to the amount of information stored in an image, which is how many pixels per inch of image, the image resolution determines the quality of the image output, the image resolution and the image size (height and width) values together determine the file Size, and the larger the value, the more storage space the image file occupies;

A low-resolution image refers to an image with a relatively small value of image resolution. The low-resolution image has a low resolution and contains insufficient pixels, which will appear rough and take up less storage space;

A high-resolution image refers to an image with a relatively large value of image resolution. The high-resolution image has a high resolution, contains many pixels, the image is clear, and the storage space occupied is relatively large.

In an embodiment of the present application, the low-resolution image is up-sampled to obtain an up-sampled high-resolution image. Extract an edge image from the up-sampled high-resolution image to obtain a third edge image. The neighborhood noise estimates of each pixel of the third edge image are calculated separately. The neighborhood noise estimates of each pixel of the third edge image are superimposed to obtain a fourth edge image. Linearly superimposing the up-sampled high-resolution image and the fourth edge image to obtain an edge-enhanced image. The edge-enhanced image is output.

According to an embodiment of the present application, the low-resolution image is up-sampled to obtain an up-sampled high-resolution image. The super-resolution image enhancement effect can be achieved without establishing a super-resolution model, which greatly reduces the algorithm calculation amount of the image enhancement method, so that the image enhancement method has less power consumption.

In an optional embodiment, an edge detection operator is used to extract an edge image from the up-sampled high-resolution image. The edge detection operator includes at least one of the following detection operators: a Laplacian Gaussian operator, a Roberts operator, and a Sobel operator.

In an optional embodiment, the method for upsampling the low-resolution image includes at least one of the following sampling methods: bilinear interpolation, deconvolution, and depooling. Among them, bilinear interpolation is also called bilinear interpolation. Mathematically, bilinear interpolation is a linear interpolation extension of an interpolation function with two variables. Its core idea is to perform linear interpolation in two directions. Convolution is to convolve a neighborhood of an image to obtain the neighborhood features of the image. Deconvolution is the inverse process of convolution. It is an algorithm-based process used to reverse the effect of convolution on recorded data. One layer in the neural network is the pooling layer, which uses pooling to extract features and reduce image data. Anti-pooling is the reverse operation of pooling. Because the pooling process only retains the main information of the original image and discards part of the information, the de-pooling cannot restore all the original image data through the pooling result.

Fig. 4 is a schematic diagram of an image enhancement device according to an exemplary embodiment. As shown in FIG. 4, the device 40 includes an edge extraction unit 401, an adaptive feature processing unit 402, an adder unit 403, an upsampling unit 404, and an image output unit 405.

Edge extraction unit 401: configured to extract an edge image from the original image to obtain a first edge image.

Adaptive feature processing unit 402: configured to perform adaptive feature processing on the first edge image to obtain a second edge image.

Adder unit 403: linearly superimpose the original image and the second edge image to obtain an edge-enhanced image.

Up-sampling unit 404: configured to up-sample the low-resolution image to obtain an up-sampled high-resolution image.

Image output unit 405: configured to output the edge-enhanced image.

In an optional embodiment, if the original image is a low-resolution image, the edge extraction unit 401 is configured to extract an edge image from the up-sampled high-resolution image to obtain a third edge image.

In an optional embodiment, if the original image is a low-resolution image, the adaptive feature processing unit 402 is configured to perform adaptive feature processing on the third edge image to obtain a fourth edge image .

In an optional embodiment, if the original image is a low-resolution image, the adder unit 403 is configured to linearly superimpose the up-sampled high-resolution image and the fourth edge image to obtain Edge enhanced image.

Fig. 5 is a block diagram of a device 1200 for performing an image enhancement method according to an exemplary embodiment. For example, the interactive device 1200 may be a mobile phone, a computer, a digital broadcasting terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, and so on.

5, the device 1200 may include one or more of the following components: processing component 1202, memory 1204, power supply component 1206, multimedia component 1208, audio component 1210, input / output (I / O) interface 1212, sensor component 1214,和通信组 1216。 And communication components 1216.

The processing component 1202 generally controls the overall operations of the device 1200, such as operations associated with display, telephone calls, data communications, camera operations, and recording operations. The processing component 1202 may include one or more processors 1220 to execute instructions to complete all or part of the steps in the above method. In addition, the processing component 1202 may include one or more modules to facilitate interaction between the processing component 1202 and other components. For example, the processing component 1202 may include a multimedia module to facilitate interaction between the multimedia component 1208 and the processing component 1202.

The memory 1204 is configured to store various types of data to support operation at the device 1200. Examples of these data include instructions for any application or method operating on the device 1200, contact data, phonebook data, messages, pictures, videos, and so on. The memory 1204 may be implemented by any type of volatile or nonvolatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read only memory (EEPROM), erasable and removable Programmable read only memory (EPROM), programmable read only memory (PROM), read only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

The power supply component 1206 provides power to various components of the device 1200. The power supply component 1206 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 1200.

The multimedia component 1208 includes a screen between the device 1200 and the user that provides an output interface. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, swipe, and gestures on the touch panel. The touch sensor may not only sense the boundary of the touch or sliding action, but also detect the duration and pressure related to the touch or sliding operation. In some embodiments, the multimedia component 1208 includes a front camera and / or a rear camera. When the device 1200 is in an operation mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

The audio component 1210 is configured to output and / or input audio signals. For example, the audio component 1210 includes a microphone (MIC). When the device 1200 is in an operation mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal may be further stored in the memory 1204 or sent via the communication component 1216. In some embodiments, the audio component 1210 further includes a speaker for outputting audio signals.

The I / O interface 1212 provides an interface between the processing component 1202 and a peripheral interface module. The peripheral interface module may be a keyboard, a click wheel, or a button. These buttons may include, but are not limited to: home button, volume button, start button, and lock button.

The sensor assembly 1214 includes one or more sensors for providing the device 1200 with status assessments in various aspects. For example, the sensor component 1214 can detect the on / off state of the device 1200, and the relative positioning of the components, for example, the component is the display and keypad of the device 1200, and the sensor component 1214 can also detect the position change of the device 1200 or a component of the device 1200 The presence or absence of user contact with the device 1200, the orientation or acceleration / deceleration of the device 1200, and the temperature change of the device 1200. The sensor assembly 1214 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 1214 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 1214 may further include an acceleration sensor, a gyro sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

The communication component 1216 is configured to facilitate wired or wireless communication between the device 1200 and other devices. The device 1200 may access a wireless network based on a communication standard, such as WiFi, an operator network (such as 2G, 3G, 4G, or 5G), or a combination thereof. In an exemplary embodiment, the communication component 1216 receives a broadcast signal or broadcast related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 1216 further includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

In an exemplary embodiment, the apparatus 1200 may be one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable A gate array (FPGA), controller, microcontroller, microprocessor or other electronic components are implemented to perform the above method.

In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, for example, a memory 1204 including instructions, which can be executed by the processor 1220 of the device 1200 to complete the above method. For example, the non-transitory computer-readable storage medium may be ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, optical data storage device, or the like.

Fig. 6 is a block diagram of a device 1300 for performing an image enhancement method according to an exemplary embodiment. For example, the device 1300 may be provided as a server. 6, the device 1300 includes a processing component 1322, which further includes one or more processors, and memory resources represented by the memory 1332, for storing instructions executable by the processing component 1322, such as application programs. The application programs stored in the memory 1332 may include one or more modules each corresponding to a set of instructions. In addition, the processing component 1322 is configured to execute instructions to execute the above-mentioned information list display method.

The device 1300 may also include a power component 1326 configured to perform power management of the device 1300, a wired or wireless network interface 1350 configured to connect the device 1300 to the network, and an input output (I / O) interface 1358. The device 1300 can operate based on an operating system stored in the memory 1332, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM or the like.

It should be understood that the present application is not limited to the precise structure that has been described above and shown in the drawings, and various modifications and changes can be made without departing from the scope thereof. The scope of this application is limited only by the appended claims.

Claims

An image enhancement method, including:

Extract an edge image from the original image to obtain a first edge image; perform adaptive feature processing on the first edge image to obtain a second edge image;

Linearly superimposing the original image and the second edge image to obtain an edge enhanced image.
The image enhancement method according to claim 1, wherein the adaptive feature processing is performed on the first edge image to obtain a second edge image, comprising: separately calculating the neighborhood of each pixel of the first edge image Noise estimate; and

The neighborhood noise estimates of each pixel of the first edge image are superimposed to obtain a second edge image.
The image enhancement method according to claim 2, wherein the neighborhood of the neighborhood noise estimate includes a neighborhood relationship of at least one of the following neighborhood relationships: 4 neighborhoods, D neighborhoods, and 8 neighborhoods.
According to the image enhancement method of claim 2, calculating the neighborhood noise estimate of each pixel of the first edge image includes:

Calculate the sum of the absolute values of the errors of the pixels of the center pixel of the first edge image and the pixels of the neighboring position, and the pixels of the center pixel of the first edge image The difference between the sum of the errors of the pixels.
The image enhancement method according to claim 1, using an edge detection operator to extract an edge image from the original image.
The image enhancement method according to claim 5, wherein the edge detection operator includes at least one of the following detection operators: a Laplacian Gaussian operator, a Roberts operator, and a Sobel operator.
The image enhancement method according to claim 1, if the original image is a low-resolution image, before extracting the edge image from the original image to obtain the first edge image, further comprising: performing an operation on the low-resolution image Up-sampling to get up-sampled high-resolution images.
The image enhancement method according to claim 7, wherein the up-sampling method for up-sampling the low-resolution image includes a sampling method of at least one of the following sampling methods: bilinear interpolation, deconvolution, and Anti-pooling.
The image enhancement method according to claim 1, further comprising: outputting the edge-enhanced image.
An image enhancement device, including:

Edge extraction unit: configured to extract the edge image from the original image to obtain the first edge image;

Adaptive feature processing unit: configured to perform adaptive feature processing on the first edge image to obtain a second edge image;

Adder unit: linearly superimpose the original image and the second edge image to obtain an edge-enhanced image.
The image enhancement device according to claim 10, further comprising:

Up-sampling unit: configured to up-sample the low-resolution image to obtain an up-sampled high-resolution image;

Image output unit: configured to output the edge-enhanced image.
According to the image enhancement device of claim 10, if the original image is a low-resolution image, the edge extraction unit is configured to extract an edge image from the upsampled high-resolution image to obtain a third edge image.
The image enhancement device according to claim 10, if the original image is a low-resolution image, the adaptive feature processing unit is configured to perform adaptive feature processing on the third edge image to obtain a fourth edge image.
The image enhancement device according to claim 10, if the original image is a low-resolution image, the adder unit is configured to linearly superimpose the up-sampled high-resolution image and the fourth edge image, Get an edge-enhanced image.
An image enhancement device, including:

A memory for storing a computer program, and intermediate data and result data generated by executing the candidate sequence of the computer program;

The processor is configured to execute the program in the memory and implement the steps of the image enhancement method according to any one of claims 1 to 9.
A computer-readable storage medium carrying one or more computer instruction programs thereon, when the computer instruction programs are executed by one or more processors, the one or more processors execute any of claims 1 to 9 One of the methods.