CN117689584A

CN117689584A - Image processing method, electronic device and storage medium

Info

Publication number: CN117689584A
Application number: CN202211091103.4A
Authority: CN
Inventors: 李乙锋; 杨淦富
Original assignee: Huawei Technologies Co Ltd
Current assignee: Huawei Technologies Co Ltd
Priority date: 2022-09-07
Filing date: 2022-09-07
Publication date: 2024-03-12

Abstract

The application provides an image processing method, electronic equipment and storage medium, wherein the method comprises the following steps: acquiring an image to be processed; decomposing the image to be processed to obtain a plurality of sub-images; and carrying out image quality restoration on the plurality of sub-images to obtain a restored image to be processed. The method provided by the application is beneficial to improving the image quality.

Description

Image processing method, electronic device and storage medium

Technical Field

The present disclosure relates to the field of image processing, and in particular, to an image processing method, an electronic device, and a storage medium.

Background

For some old movies, the image quality of these old movies is lower due to the current photographing device and photographing technology, for example, the image quality of these old movies is often accompanied by "speckles" of stars and points, and these "speckles" are called noise in the field of image processing. However, since the presence of these noises affects the viewing effect of people, an image processing method is needed to remove these noises so as to improve the image quality.

Disclosure of Invention

The application provides an image processing method, electronic equipment and a storage medium, which are beneficial to reducing noise and improving image quality.

In a first aspect, the present application provides an image processing method, including:

acquiring an image to be processed;

decomposing the image to be processed to obtain a plurality of sub-images;

and carrying out image quality restoration on the plurality of sub-images to obtain a restored image to be processed.

In the application, the sub-image obtained by decomposing the image is subjected to image quality restoration, so that noise in the image can be reduced, and the image quality of the image can be improved.

In one possible implementation manner, the performing image quality restoration on the plurality of sub-images includes:

inputting the plurality of sub-images into a preset image library to be grouped to obtain a plurality of sub-image sets, wherein the preset image library comprises a plurality of preset sub-image types, each sub-image set comprises one or more sub-images, and each sub-image set corresponds to one preset sub-image type;

calculating means and variances of the plurality of sub-image sets;

and performing image quality restoration on the plurality of sub-images based on the mean and the variance of the plurality of sub-image sets.

In one possible implementation manner, the type of the preset image library is determined by the type of the image to be processed.

In the method, the external image library is determined according to the type of the image to be processed, so that the mean value and the variance of the sub-image set can be calculated more accurately, and the image quality of the image can be improved effectively.

In one possible implementation manner, the method further includes:

and carrying out image quality restoration on the restored image to be processed again so as to realize iterative restoration.

In the application, the image quality of the image can be further improved by performing iterative restoration on the first image.

In one possible implementation manner, the number of iterative repairs is preset; or the number of iterative repairs is determined according to user requirements.

In the method, the user operation can be simplified by presetting the iteration times; the iteration times are determined according to the user requirements, so that the iteration on demand can be realized, and the flexibility is improved.

In one possible implementation manner, before performing image quality restoration on the restored image to be processed again to implement iterative restoration, the method further includes:

and judging the image quality of the restored image to be processed.

In the method, the image quality of the restored image is automatically judged, and whether iteration is carried out or not is determined according to the image quality, so that the user operation is simplified, and the image quality of the image can be improved.

In one possible implementation manner, the image to be processed is each frame of image extracted in sequence in the video playing process.

In the application, the image quality restoration of each frame of image in the video played in real time can be realized by carrying out the image quality restoration of the video in the video playing process.

In one possible implementation manner, the image to be processed is all images extracted from the still video.

In the application, by repairing all the images in the static video, the image quality can be repaired before video playing, and the image quality is not required to be repaired in the video playing process, so that the fluency of video playing can be improved.

In a second aspect, the present application provides an image processing apparatus including:

the acquisition module is used for acquiring the image to be processed;

the decomposition module is used for decomposing the image to be processed to obtain a plurality of sub-images;

and the image processing module is used for carrying out image quality restoration on the plurality of sub-images to obtain a restored image to be processed.

In one possible implementation manner, the image processing module is specifically configured to input the plurality of sub-images into a preset image library to be grouped to obtain a plurality of sub-image sets, where the preset image library includes a plurality of preset sub-image types, each sub-image set includes one or more sub-images, and each sub-image set corresponds to one preset sub-image type;

Calculating means and variances of the plurality of sub-image sets;

In one possible implementation manner, the image processing module is further configured to perform image quality restoration again on the restored image to be processed, so as to implement iterative restoration.

In one possible implementation manner, the image processing apparatus further includes:

and the judging module is used for judging the image quality of the repaired image to be processed.

In a third aspect, the present application provides an electronic device, including: a processor and a memory for storing a computer program; the processor is configured to execute the computer program to implement the image processing method according to the first aspect.

In a fourth aspect, the present application provides a computer readable storage medium having a computer program stored therein, which when run on a computer causes the computer to implement the image processing method as described in the first aspect.

In a fifth aspect, the present application provides a computer program which, when run on a processor of an electronic device, causes the electronic device to perform the image processing method of the first aspect.

In one possible design, the program in the fifth aspect may be stored in whole or in part on a storage medium packaged with the processor, or in part or in whole on a memory not packaged with the processor.

Drawings

Fig. 1 is a schematic structural diagram of an electronic device according to an embodiment of the present application;

FIG. 2 is a flowchart illustrating an embodiment of an image processing method provided herein;

fig. 3 is a schematic view of a sub-image provided in an embodiment of the present application;

fig. 4 is a schematic diagram of a repair effect provided in an embodiment of the present application;

FIG. 5 is a flowchart illustrating another embodiment of an image processing method provided in the present application;

fig. 6 is a schematic structural diagram of an image processing apparatus according to an embodiment of the present application.

Detailed Description

In the embodiment of the present application, unless otherwise specified, the character "/" indicates that the front-rear association object is one or a relationship. For example, A/B may represent A or B. "and/or" describes an association relationship of an association object, meaning that three relationships may exist. For example, a and/or B may represent: a exists alone, A and B exist together, and B exists alone.

It should be noted that the terms "first," "second," and the like in the embodiments of the present application are used for distinguishing between description and not necessarily for indicating or implying a relative importance or number of features or characteristics that are indicated, nor does it imply a sequential order.

In the embodiments of the present application, "at least one" means one or more, and "a plurality" means two or more. Furthermore, "at least one item(s)" below, or the like, refers to any combination of these items, and may include any combination of single item(s) or plural items(s). For example, at least one (one) of A, B or C may represent: a, B, C, a and B, a and C, B and C, or A, B and C. Wherein each of A, B, C may itself be an element or a collection comprising one or more elements.

In this application embodiments, "exemplary," "in some embodiments," "in another embodiment," etc. are used to indicate an example, instance, or illustration. Any embodiment or design described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments or designs. Rather, the term use of an example is intended to present concepts in a concrete fashion.

"of", "corresponding" and "corresponding" in the embodiments of the present application may be sometimes used in combination, and it should be noted that the meaning to be expressed is consistent when the distinction is not emphasized. In the embodiments of the present application, communications and transmissions may sometimes be mixed, and it should be noted that, when the distinction is not emphasized, the meaning expressed is consistent. For example, a transmission may include sending and/or receiving, either nouns or verbs.

The equal to that relates to in this application embodiment can be with being greater than even using, is applicable to the technical scheme that adopts when being greater than, also can be with being less than even using, is applicable to the technical scheme that adopts when being less than. It should be noted that when the number is equal to or greater than the sum, the number cannot be smaller than the sum; when the value is equal to or smaller than that used together, the value is not larger than that used together.

Based on the above problems, an embodiment of the present application provides an image processing method, which is applied to an electronic device. The electronic device may be a terminal device having a display screen. The electronic device may be a fixed terminal, such as a notebook computer, desktop computer, etc., or a mobile terminal, which may also be referred to as a User Equipment (UE), access terminal, subscriber unit, subscriber station, mobile station, remote terminal, mobile device, user terminal, wireless communication device, user agent, or User Equipment. The mobile terminal may be a Station (ST) in a WLAN, may be a cellular telephone, a cordless telephone, a session initiation protocol (Session Initiation Protocol, SIP) phone, a wireless local loop (Wireless Local Loop, WLL) station, a personal digital assistant (Personal Digital Assistant, PDA) device, a handheld device with wireless communication capabilities, a computing device or other processing device connected to a wireless modem, an in-vehicle device, a car networking terminal, a computer, a laptop computer, a handheld communication device, a handheld computing device, a satellite radio, a wireless modem card, a television Set Top Box (STB), a customer premise equipment (customer premise equipment, CPE) and/or other devices for communicating over a wireless system as well as next generation communication systems, such as a mobile terminal in a 5G network or a mobile terminal in a future evolved public land mobile network (Public Land Mobile Network, PLMN) network, etc. The electronic device may also be a wearable device. The wearable device can also be called as a wearable intelligent device, and is a generic name for intelligently designing daily wearing and developing wearable devices by applying a wearable technology, such as a smart watch, a smart bracelet and the like.

Fig. 1 is a schematic diagram schematically illustrating a structure of an electronic device 100.

The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (universal serial bus, USB) interface 130, a charge management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, keys 190, a motor 191, an indicator 192, a camera 193, a display 194, and a subscriber identity module (subscriber identification module, SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyro sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, and the like.

It should be understood that the illustrated structure of the embodiment of the present invention does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, electronic device 100 may include more or fewer components than shown, or certain components may be combined, or certain components may be split, or different arrangements of components. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

The processor 110 may include one or more processing units, such as: the processor 110 may include an application processor (application processor, AP), a modem processor, a graphics processor (graphics processing unit, GPU), an image signal processor (image signal processor, ISP), a controller, a video codec, a digital signal processor (digital signal processor, DSP), a baseband processor, and/or a neural network processor (neural-network processing unit, NPU), etc. Wherein the different processing units may be separate devices or may be integrated in one or more processors.

The controller can generate operation control signals according to the instruction operation codes and the time sequence signals to finish the control of instruction fetching and instruction execution.

A memory may also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may hold instructions or data that the processor 110 has just used or recycled. If the processor 110 needs to reuse the instruction or data, it can be called directly from the memory. Repeated accesses are avoided and the latency of the processor 110 is reduced, thereby improving the efficiency of the system.

In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an integrated circuit (inter-integrated circuit, I2C) interface, an integrated circuit built-in audio (inter-integrated circuit sound, I2S) interface, a pulse code modulation (pulse code modulation, PCM) interface, a universal asynchronous receiver transmitter (universal asynchronous receiver/transmitter, UART) interface, a mobile industry processor interface (mobile industry processor interface, MIPI), a general-purpose input/output (GPIO) interface, a subscriber identity module (subscriber identity module, SIM) interface, and/or a universal serial bus (universal serial bus, USB) interface, among others.

The I2C interface is a bi-directional synchronous serial bus comprising a serial data line (SDA) and a serial clock line (derail clock line, SCL). In some embodiments, the processor 110 may contain multiple sets of I2C buses. The processor 110 may be coupled to the touch sensor 180K, charger, flash, camera 193, etc., respectively, through different I2C bus interfaces. For example: the processor 110 may be coupled to the touch sensor 180K through an I2C interface, such that the processor 110 communicates with the touch sensor 180K through an I2C bus interface to implement a touch function of the electronic device 100.

The I2S interface may be used for audio communication. In some embodiments, the processor 110 may contain multiple sets of I2S buses. The processor 110 may be coupled to the audio module 170 via an I2S bus to enable communication between the processor 110 and the audio module 170. In some embodiments, the audio module 170 may transmit an audio signal to the wireless communication module 160 through the I2S interface, to implement a function of answering a call through the bluetooth headset.

PCM interfaces may also be used for audio communication to sample, quantize and encode analog signals. In some embodiments, the audio module 170 and the wireless communication module 160 may be coupled through a PCM bus interface. In some embodiments, the audio module 170 may also transmit audio signals to the wireless communication module 160 through the PCM interface to implement a function of answering a call through the bluetooth headset. Both the I2S interface and the PCM interface may be used for audio communication.

The UART interface is a universal serial data bus for asynchronous communications. The bus may be a bi-directional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, a UART interface is typically used to connect the processor 110 with the wireless communication module 160. For example: the processor 110 communicates with a bluetooth module in the wireless communication module 160 through a UART interface to implement a bluetooth function. In some embodiments, the audio module 170 may transmit an audio signal to the wireless communication module 160 through a UART interface, to implement a function of playing music through a bluetooth headset.

The MIPI interface may be used to connect the processor 110 to peripheral devices such as a display 194, a camera 193, and the like. The MIPI interfaces include camera serial interfaces (camera serial interface, CSI), display serial interfaces (display serial interface, DSI), and the like. In some embodiments, processor 110 and camera 193 communicate through a CSI interface to implement the photographing functions of electronic device 100. The processor 110 and the display 194 communicate via a DSI interface to implement the display functionality of the electronic device 100.

The GPIO interface may be configured by software. The GPIO interface may be configured as a control signal or as a data signal. In some embodiments, a GPIO interface may be used to connect the processor 110 with the camera 193, the display 194, the wireless communication module 160, the audio module 170, the sensor module 180, and the like. The GPIO interface may also be configured as an I2C interface, an I2S interface, a UART interface, an MIPI interface, etc.

The USB interface 130 is an interface conforming to the USB standard specification, and may specifically be a Mini USB interface, a Micro USB interface, a USB Type C interface, or the like. The USB interface 130 may be used to connect a charger to charge the electronic device 100, and may also be used to transfer data between the electronic device 100 and a peripheral device. And can also be used for connecting with a headset, and playing audio through the headset. The interface may also be used to connect other terminal devices, such as AR devices, etc.

It should be understood that the interfacing relationship between the modules illustrated in the embodiments of the present invention is only illustrative, and is not meant to limit the structure of the electronic device 100. In other embodiments of the present application, the electronic device 100 may also use different interfacing manners, or a combination of multiple interfacing manners in the foregoing embodiments.

The charge management module 140 is configured to receive a charge input from a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charge management module 140 may receive a charging input of a wired charger through the USB interface 130. In some wireless charging embodiments, the charge management module 140 may receive wireless charging input through a wireless charging coil of the electronic device 100. The charging management module 140 may also supply power to the terminal device through the power management module 141 while charging the battery 142.

The power management module 141 is used for connecting the battery 142, and the charge management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and/or the charge management module 140 to power the processor 110, the internal memory 121, the display 194, the camera 193, the wireless communication module 160, and the like. The power management module 141 may also be configured to monitor battery capacity, battery cycle number, battery health (leakage, impedance) and other parameters. In other embodiments, the power management module 141 may also be provided in the processor 110. In other embodiments, the power management module 141 and the charge management module 140 may be disposed in the same device.

The wireless communication function of the electronic device 100 may be implemented by the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, a modem processor, a baseband processor, and the like.

The antennas 1 and 2 are used for transmitting and receiving electromagnetic wave signals. Each antenna in the electronic device 100 may be used to cover a single or multiple communication bands. Different antennas may also be multiplexed to improve the utilization of the antennas. For example: the antenna 1 may be multiplexed into a diversity antenna of a wireless local area network. In other embodiments, the antenna may be used in conjunction with a tuning switch.

The mobile communication module 150 may provide a solution for wireless communication including 2G/3G/4G/5G, etc., applied to the electronic device 100. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (low noise amplifier, LNA), etc. The mobile communication module 150 may receive electromagnetic waves from the antenna 1, perform processes such as filtering, amplifying, and the like on the received electromagnetic waves, and transmit the processed electromagnetic waves to the modem processor for demodulation. The mobile communication module 150 can amplify the signal modulated by the modem processor, and convert the signal into electromagnetic waves through the antenna 1 to radiate. In some embodiments, at least some of the functional modules of the mobile communication module 150 may be disposed in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 may be provided in the same device as at least some of the modules of the processor 110.

The modem processor may include a modulator and a demodulator. The modulator is used for modulating the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used for demodulating the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low frequency baseband signal to the baseband processor for processing. The low frequency baseband signal is processed by the baseband processor and then transferred to the application processor. The application processor outputs sound signals through an audio device (not limited to the speaker 170A, the receiver 170B, etc.), or displays images or video through the display screen 194. In some embodiments, the modem processor may be a stand-alone device. In other embodiments, the modem processor may be provided in the same device as the mobile communication module 150 or other functional module, independent of the processor 110.

The wireless communication module 160 may provide solutions for wireless communication including wireless local area network (wireless local area networks, WLAN) (e.g., wireless fidelity (wireless fidelity, wi-Fi) network), bluetooth (BT), global navigation satellite system (global navigation satellite system, GNSS), frequency modulation (frequency modulation, FM), near field wireless communication technology (near field communication, NFC), infrared technology (IR), etc., as applied to the electronic device 100. The wireless communication module 160 may be one or more devices that integrate at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the electromagnetic wave signals, filters the electromagnetic wave signals, and transmits the processed signals to the processor 110. The wireless communication module 160 may also receive a signal to be transmitted from the processor 110, frequency modulate it, amplify it, and convert it to electromagnetic waves for radiation via the antenna 2.

In some embodiments, antenna 1 and mobile communication module 150 of electronic device 100 are coupled, and antenna 2 and wireless communication module 160 are coupled, such that electronic device 100 may communicate with a network and other devices through wireless communication techniques. The wireless communication techniques may include the Global System for Mobile communications (global system for mobile communications, GSM), general packet radio service (general packet radio service, GPRS), code division multiple access (code division multiple access, CDMA), wideband code division multiple access (wideband code division multiple access, WCDMA), time division code division multiple access (time-division code division multiple access, TD-SCDMA), long term evolution (long term evolution, LTE), BT, GNSS, WLAN, NFC, FM, and/or IR techniques, among others. The GNSS may include a global satellite positioning system (global positioning system, GPS), a global navigation satellite system (global navigation satellite system, GLONASS), a beidou satellite navigation system (beidou navigation satellite system, BDS), a quasi zenith satellite system (quasi-zenith satellite system, QZSS) and/or a satellite based augmentation system (satellite based augmentation systems, SBAS).

The electronic device 100 implements display functions through a GPU, a display screen 194, an application processor, and the like. The GPU is a microprocessor for image processing, and is connected to the display 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

The display screen 194 is used to display images, videos, and the like. The display 194 includes a display panel. The display panel may employ a liquid crystal display (liquid crystal display, LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED) or an active-matrix organic light-emitting diode (matrix organic light emitting diode), a flexible light-emitting diode (flex), a mini, a Micro led, a Micro-OLED, a quantum dot light-emitting diode (quantum dot light emitting diodes, QLED), or the like. In some embodiments, the electronic device 100 may include 1 or N display screens 194, N being a positive integer greater than 1.

The electronic device 100 may implement photographing functions through an ISP, a camera 193, a video codec, a GPU, a display screen 194, an application processor, and the like.

The ISP is used to process data fed back by the camera 193. For example, when photographing, the shutter is opened, light is transmitted to the camera photosensitive element through the lens, the optical signal is converted into an electric signal, and the camera photosensitive element transmits the electric signal to the ISP for processing and is converted into an image visible to naked eyes. ISP can also optimize the noise, brightness and skin color of the image. The ISP can also optimize parameters such as exposure, color temperature and the like of a shooting scene. In some embodiments, the ISP may be provided in the camera 193.

The camera 193 is used to capture still images or video. The object generates an optical image through the lens and projects the optical image onto the photosensitive element. The photosensitive element may be a charge coupled device (charge coupled device, CCD) or a Complementary Metal Oxide Semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, which is then transferred to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV, or the like format. In some embodiments, electronic device 100 may include 1 or N cameras 193, N being a positive integer greater than 1.

The digital signal processor is used for processing digital signals, and can process other digital signals besides digital image signals. For example, when the electronic device 100 selects a frequency bin, the digital signal processor is used to fourier transform the frequency bin energy, or the like.

Video codecs are used to compress or decompress digital video. The electronic device 100 may support one or more video codecs. In this way, the electronic device 100 may play or record video in a variety of encoding formats, such as: dynamic picture experts group (moving picture experts group, MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

The NPU is a neural-network (NN) computing processor, and can rapidly process input information by referencing a biological neural network structure, for example, referencing a transmission mode between human brain neurons, and can also continuously perform self-learning. Applications such as intelligent awareness of the electronic device 100 may be implemented through the NPU, for example: image recognition, face recognition, speech recognition, text understanding, etc.

The external memory interface 120 may be used to connect an external memory card, such as a Micro SD card, to enable expansion of the memory capabilities of the electronic device 100. The external memory card communicates with the processor 110 through an external memory interface 120 to implement data storage functions. For example, files such as music, video, etc. are stored in an external memory card.

The internal memory 121 may be used to store computer executable program code including instructions. The internal memory 121 may include a storage program area and a storage data area. The storage program area may store an application program (such as a sound playing function, an image playing function, etc.) required for at least one function of the operating system, etc. The storage data area may store data created during use of the electronic device 100 (e.g., audio data, phonebook, etc.), and so on. In addition, the internal memory 121 may include a high-speed random access memory, and may further include a nonvolatile memory such as at least one magnetic disk storage device, a flash memory device, a universal flash memory (universal flash storage, UFS), and the like. The processor 110 performs various functional applications of the electronic device 100 and data processing by executing instructions stored in the internal memory 121 and/or instructions stored in a memory provided in the processor.

The electronic device 100 may implement audio functions through an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, an application processor, and the like. Such as music playing, recording, etc.

The audio module 170 is used to convert digital audio information into an analog audio signal output and also to convert an analog audio input into a digital audio signal. The audio module 170 may also be used to encode and decode audio signals. In some embodiments, the audio module 170 may be disposed in the processor 110, or a portion of the functional modules of the audio module 170 may be disposed in the processor 110.

The speaker 170A, also referred to as a "horn," is used to convert audio electrical signals into sound signals. The electronic device 100 may listen to music, or to hands-free conversations, through the speaker 170A.

A receiver 170B, also referred to as a "earpiece", is used to convert the audio electrical signal into a sound signal. When electronic device 100 is answering a telephone call or voice message, voice may be received by placing receiver 170B in close proximity to the human ear.

Microphone 170C, also referred to as a "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or transmitting voice information, the user can sound near the microphone 170C through the mouth, inputting a sound signal to the microphone 170C. The electronic device 100 may be provided with at least one microphone 170C. In other embodiments, the electronic device 100 may be provided with two microphones 170C, and may implement a noise reduction function in addition to collecting sound signals. In other embodiments, the electronic device 100 may also be provided with three, four, or more microphones 170C to enable collection of sound signals, noise reduction, identification of sound sources, directional recording functions, etc.

The earphone interface 170D is used to connect a wired earphone. The earphone interface 170D may be a USB interface 130 or a 3.5mm open mobile terminal platform (open mobile terminal platform, OMTP) standard interface, a american cellular telecommunications industry association (cellular telecommunications industry association of the USA, CTIA) standard interface.

The pressure sensor 180A is used to sense a pressure signal, and may convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A may be disposed on the display screen 194. The pressure sensor 180A is of various types, such as a resistive pressure sensor, an inductive pressure sensor, a capacitive pressure sensor, and the like. The capacitive pressure sensor may be a capacitive pressure sensor comprising at least two parallel plates with conductive material. The capacitance between the electrodes changes when a force is applied to the pressure sensor 180A. The electronic device 100 determines the strength of the pressure from the change in capacitance. When a touch operation is applied to the display screen 194, the electronic apparatus 100 detects the touch operation intensity according to the pressure sensor 180A. The electronic device 100 may also calculate the location of the touch based on the detection signal of the pressure sensor 180A. In some embodiments, touch operations that act on the same touch location, but at different touch operation strengths, may correspond to different operation instructions. For example: and executing an instruction for checking the short message when the touch operation with the touch operation intensity smaller than the first pressure threshold acts on the short message application icon. And executing an instruction for newly creating the short message when the touch operation with the touch operation intensity being greater than or equal to the first pressure threshold acts on the short message application icon.

The gyro sensor 180B may be used to determine a motion gesture of the electronic device 100. In some embodiments, the angular velocity of electronic device 100 about three axes (i.e., x, y, and z axes) may be determined by gyro sensor 180B. The gyro sensor 180B may be used for photographing anti-shake. For example, when the shutter is pressed, the gyro sensor 180B detects the shake angle of the electronic device 100, calculates the distance to be compensated by the lens module according to the angle, and makes the lens counteract the shake of the electronic device 100 through the reverse motion, so as to realize anti-shake. The gyro sensor 180B may also be used for navigating, somatosensory game scenes.

The air pressure sensor 180C is used to measure air pressure. In some embodiments, electronic device 100 calculates altitude from barometric pressure values measured by barometric pressure sensor 180C, aiding in positioning and navigation.

The magnetic sensor 180D includes a hall sensor. The electronic device 100 may detect the opening and closing of the flip cover using the magnetic sensor 180D. In some embodiments, when the electronic device 100 is a flip machine, the electronic device 100 may detect the opening and closing of the flip according to the magnetic sensor 180D. And then according to the detected opening and closing state of the leather sheath or the opening and closing state of the flip, the characteristics of automatic unlocking of the flip and the like are set.

The acceleration sensor 180E may detect the magnitude of acceleration of the electronic device 100 in various directions (typically three axes). The magnitude and direction of gravity may be detected when the electronic device 100 is stationary. The method can also be used for identifying the gesture of the terminal equipment, and is applied to the applications such as horizontal and vertical screen switching, pedometers and the like.

A distance sensor 180F for measuring a distance. The electronic device 100 may measure the distance by infrared or laser. In some embodiments, the electronic device 100 may range using the distance sensor 180F to achieve quick focus.

The proximity light sensor 180G may include, for example, a Light Emitting Diode (LED) and a light detector, such as a photodiode. The light emitting diode may be an infrared light emitting diode. The electronic device 100 emits infrared light outward through the light emitting diode. The electronic device 100 detects infrared reflected light from nearby objects using a photodiode. When sufficient reflected light is detected, it may be determined that there is an object in the vicinity of the electronic device 100. When insufficient reflected light is detected, the electronic device 100 may determine that there is no object in the vicinity of the electronic device 100. The electronic device 100 can detect that the user holds the electronic device 100 close to the ear by using the proximity light sensor 180G, so as to automatically extinguish the screen for the purpose of saving power. The proximity light sensor 180G may also be used in holster mode, pocket mode to automatically unlock and lock the screen.

The ambient light sensor 180L is used to sense ambient light level. The electronic device 100 may adaptively adjust the brightness of the display 194 based on the perceived ambient light level. The ambient light sensor 180L may also be used to automatically adjust white balance when taking a photograph. Ambient light sensor 180L may also cooperate with proximity light sensor 180G to detect whether electronic device 100 is in a pocket to prevent false touches.

The fingerprint sensor 180H is used to collect a fingerprint. The electronic device 100 may utilize the collected fingerprint feature to unlock the fingerprint, access the application lock, photograph the fingerprint, answer the incoming call, etc.

The temperature sensor 180J is for detecting temperature. In some embodiments, the electronic device 100 performs a temperature processing strategy using the temperature detected by the temperature sensor 180J. For example, when the temperature reported by temperature sensor 180J exceeds a threshold, electronic device 100 performs a reduction in the performance of a processor located in the vicinity of temperature sensor 180J in order to reduce power consumption to implement thermal protection. In other embodiments, when the temperature is below another threshold, the electronic device 100 heats the battery 142 to avoid the low temperature causing the electronic device 100 to be abnormally shut down. In other embodiments, when the temperature is below a further threshold, the electronic device 100 performs boosting of the output voltage of the battery 142 to avoid abnormal shutdown caused by low temperatures.

The touch sensor 180K, also referred to as a "touch device". The touch sensor 180K may be disposed on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, which is also called a "touch screen". The touch sensor 180K is for detecting a touch operation acting thereon or thereabout. The touch sensor may communicate the detected touch operation to the application processor to determine the touch event type. Visual output related to touch operations may be provided through the display 194. In other embodiments, the touch sensor 180K may also be disposed on the surface of the electronic device 100 at a different location than the display 194.

The bone conduction sensor 180M may acquire a vibration signal. In some embodiments, bone conduction sensor 180M may acquire a vibration signal of a human vocal tract vibrating bone pieces. The bone conduction sensor 180M may also contact the pulse of the human body to receive the blood pressure pulsation signal. In some embodiments, bone conduction sensor 180M may also be provided in a headset, in combination with an osteoinductive headset. The audio module 170 may analyze the voice signal based on the vibration signal of the sound portion vibration bone block obtained by the bone conduction sensor 180M, so as to implement a voice function. The application processor may analyze the heart rate information based on the blood pressure beat signal acquired by the bone conduction sensor 180M, so as to implement a heart rate detection function.

The keys 190 include a power-on key, a volume key, etc. The keys 190 may be mechanical keys. Or may be a touch key. The electronic device 100 may receive key inputs, generating key signal inputs related to user settings and function controls of the electronic device 100.

The motor 191 may generate a vibration cue. The motor 191 may be used for incoming call vibration alerting as well as for touch vibration feedback. For example, touch operations acting on different applications (e.g., photographing, audio playing, etc.) may correspond to different vibration feedback effects. The motor 191 may also correspond to different vibration feedback effects by touching different areas of the display screen 194. Different application scenarios (such as time reminding, receiving information, alarm clock, game, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect may also support customization.

The indicator 192 may be an indicator light, may be used to indicate a state of charge, a change in charge, a message indicating a missed call, a notification, etc.

The SIM card interface 195 is used to connect a SIM card. The SIM card may be inserted into the SIM card interface 195, or removed from the SIM card interface 195 to enable contact and separation with the electronic device 100. The electronic device 100 may support 1 or N SIM card interfaces, N being a positive integer greater than 1. The SIM card interface 195 may support Nano SIM cards, micro SIM cards, and the like. The same SIM card interface 195 may be used to insert multiple cards simultaneously. The types of the plurality of cards may be the same or different. The SIM card interface 195 may also be compatible with different types of SIM cards. The SIM card interface 195 may also be compatible with external memory cards. The electronic device 100 interacts with the network through the SIM card to realize functions such as communication and data communication. In some embodiments, the electronic device 100 employs esims, i.e.: an embedded SIM card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100.

Referring now to fig. 2 to 5, a video is taken as an example to describe an image processing method provided in an embodiment of the present application. It will be appreciated that although the embodiments of the present application are illustrated with respect to video, they are not limiting of the embodiments of the present application, and in some embodiments, the present application may be applied to individual images as well.

Fig. 2 is a schematic flow chart of an embodiment of an image processing method provided in the present application, where the embodiment shown in fig. 2 may be applied to video in real-time playing, and specifically includes the following steps:

step 201, a video to be processed is acquired.

Step 202, extracting a first image in the video to be processed in the playing process of the video to be processed.

Specifically, the video to be processed may be composed of a plurality of frames of images, and it is understood that the continuous playing of the plurality of frames of images may form a video. Therefore, during the playing process of the video to be processed, a currently played frame of image may be extracted, and for convenience of explanation, the currently played frame of image will be hereinafter referred to as a "first image".

In step 203, the first image is decomposed to obtain a plurality of sub-images.

Specifically, when the first image is extracted, the first image may be decomposed, whereby a plurality of sub-images may be obtained. Where each sub-image may include a plurality of pixels, each sub-image may include, for example, 8×8=64 pixels. It will be appreciated that the above sub-image containing 64 pixels is only exemplary, and not limiting to the embodiments of the present application, and in some embodiments, the sub-image may also include other numbers of pixels, for example, 4*4 =16, 16×16=256, and so on.

It can be understood that the greater the number of pixels of the sub-image, the greater the calculation amount and the higher the calculation accuracy; the smaller the data of the pixels of the sub-image is, the smaller the calculation amount is, and the lower the calculation accuracy is. Therefore, the decomposition of the sub-images can also be determined according to the requirements.

The acquisition of sub-images is now exemplarily described in connection with fig. 3. Referring to fig. 3, the first image is an image including n×m pixels, and when the first image is decomposed, a plurality of sub-images may be obtained, where each sub-image includes a plurality of pixels, for example, each sub-image may include 8×8 pixels.

And 204, performing image quality restoration on the plurality of sub-images in the first image to obtain a restored first image.

Specifically, after a plurality of sub-images in the first image are acquired, the plurality of sub-images in the first image may be subjected to image quality restoration, whereby a restored first image may be obtained.

The image quality restoration method for the plurality of sub-images in the first image may be as follows:

first, an external image library is acquired, which may include a plurality of preset sub-image types, wherein each preset sub-image type has its pixel characteristics.

Then, the plurality of sub-images in the first image may be grouped, thereby obtaining a plurality of sub-image sets, where each sub-image set may include a plurality of sub-images, and the plurality of sub-images in each sub-image set have the same or similar pixel characteristics, for example, pixel values between pixels of the plurality of sub-images in each sub-image are the same or similar. In a specific implementation, a plurality of sub-images in a first image may be input into a preset gaussian mixture model, and the plurality of sub-images in the first image may be grouped by using a preset sub-image type in the external image library as a reference image, thereby obtaining a plurality of sub-image sets. It will be appreciated that the gaussian mixture model described above may be obtained by pre-training, and is merely illustrative, and not limiting of the embodiments of the present application, and in some embodiments, may be other types of computational models.

The grouping of sub-images will now be exemplified by a gaussian mixture model. In order to better group sub-images, the matching probability of any sub-image and the preset sub-image type in the external image library can be calculated, and the formula for calculating the matching probability of any sub-image and the preset sub-image type in the external image library is shown as follows:

π _k ＝p(k|z _i )；

Wherein pi _k Is z _i Probability, z, belonging to the kth preset sub-picture type _i Is the i-th sub-image.

When the ith sub-image z is obtained _i After the probability of each preset sub-image type, the preset sub-image type corresponding to the maximum probability can be selected, for example, the maximum probability can be:

π _max ＝max(π _k )；

the ith sub-image z may then be used _i Grouping to pi _max And the corresponding preset sub-image type. After grouping all the sub-images into corresponding preset sub-image types, a plurality of sub-image sets can be obtained, wherein each sub-image set corresponds to one preset sub-image type, and one or more sub-images can be included in each sub-image set.

When multiple sub-image sets are obtained, the mean μ of each sub-image set can be calculated _k Sum of variances sigma _k . Wherein mu _k Sigma, the mean value of the kth sub-image set _k Is the variance of the kth sub-image set.

Then, the sub-image can be restored based on the mean value and the variance of the sub-image set where any sub-image is located, so that the restored sub-image can be obtained through calculation.

The calculation formula for repairing the sub-image is as follows:

z′ _i ＝(σ _k +σ ² ×I) ^-1 (σ _k ×y+σ ² ×I×μ _k )；

Z _i ' is a restored sub-image, sigma ² For noise variance, y is the first image and I is the sub-image vector. Taking 8 x 8 sub-images as an example, I is a 64-dimensional vector.

After repairing all the sub-images in the first image in the above manner, all the repaired sub-images can be synthesized, so that the repaired first image can be obtained. The calculation mode for synthesizing all the repaired sub-images can be shown by the following formula:

wherein x is the first image after repair, alpha is a constant coefficient, A is a noise matrix, L is the total number of sub-images in the first image, and P is a decomposition matrix for decomposing the first image to obtain sub-images. It will be appreciated that each decomposition matrix corresponds to a sub-image, that is, any one sub-image has its corresponding decomposition matrix P, and therefore the decomposition matrices are different from sub-image to sub-image.

In some optional embodiments, in order to improve the repair accuracy of the first image, when the external image library is selected, a corresponding external image library may also be selected according to the type of the first image, and different types of images may correspond to different types of external image libraries, so that the first image may be repaired more specifically, and thus the repair accuracy may be improved. By way of example, the types of the first image may include a person, an animal, a plant, etc., and then an external person image library may be selected when the type of the first image is a person, or an external animal image library may be selected when the type of the first image is an animal, and an external plant image library may be selected when the type of the first image is a plant. It will be appreciated that the above-described types of figures, animals, and plants are merely exemplary, and are not limiting of embodiments of the present application, and that other types may be included in some embodiments.

In some optional embodiments, after the repaired first image is obtained, iterative repair may be performed on the repaired first image, for example, step 203 and step 204 may be further performed on the repaired first image, so that the repaired first image is further repaired, and thus the image quality of the first image may be further improved. The number of iterations may be preset. It can be appreciated that the fewer the number of iterations, the faster the first image is repaired, that is, the higher the efficiency of the repair, but the lower the accuracy of the repair; the more iterations, the slower the speed of the repair of the first image, that is, the lower the efficiency of the repair, but the higher the accuracy of the repair. Thus, in some embodiments, the number of iterations may also be as desired. For example, if higher repair accuracy is required, a higher number of iterations may be set; if a faster repair speed is required, a lower number of iterations may be set.

In some alternative embodiments, the calculation manner of synthesizing the sub-images may be updated during iterative repair, and, by way of example, the calculation manner of synthesizing all the repaired sub-images may be shown by the following formula:

Where β is a constant coefficient at the S-th iterative repair, illustratively, the β value at the first iterative repair may be 1, the β value at the second iterative repair may be 4, the β value at the third iterative repair may be 8, the β value at the fourth iterative repair may be 16, the β value at the fifth iterative repair may be 32, and so on. It is to be understood that the above-mentioned beta values are merely exemplary, and are not limiting to the embodiments of the present application, and in some embodiments, the above-mentioned beta values may be other values.

In some optional embodiments, the image quality of the repaired first image may be further determined, and if the image quality of the repaired first image is higher, that is, the requirement of precision is satisfied, iterative repair may not be performed; if the image quality of the repaired first image is lower, that is, the requirement of precision is not met, iterative repair can be further performed until the requirement of precision is met.

Fig. 4 is a schematic diagram of a restoration effect of the first image.

It can be understood that after the first image processing is completed, the next frame of image can be processed, and the next frame of image can be processed by referring to the processing mode of the first image until all the frame of images in the video to be processed are processed, so that the processing of the whole video to be processed can be completed, and the image quality of the video to be processed can be improved and the viewing experience of the user can be improved in the process of viewing the video to be processed in real time.

In the embodiment of the application, each frame of image is repaired in the real-time playing process of the video, so that the image quality of the whole playing video can be improved, and the viewing experience of a user can be improved.

Fig. 5 is a schematic flow chart of another embodiment of the image processing method provided in the present application, and the embodiment shown in fig. 5 may be applied to still video, that is, video that is not played in real time, and specifically includes the following steps:

step 501, a video to be processed is acquired.

Step 502, all images in the video to be processed are extracted.

And step 503, decomposing all the images in the video to be processed to obtain sub-images corresponding to each image.

Specifically, the method of decomposing all the images in the step 503 may refer to the method of decomposing the first image in the above embodiment, which is not described herein.

And step 504, repairing the sub-image corresponding to each image to obtain each repaired image.

Specifically, after obtaining the sub-image corresponding to each image, the sub-image corresponding to each image may be repaired. The restoration of the sub-image corresponding to each image may be to sequentially restore the sub-image corresponding to each image, or may also restore the sub-image corresponding to each image at the same time, which is not limited in particular in the embodiment of the present application.

After repairing the sub-image corresponding to each image, each repaired image can be obtained, and then the repaired image is covered with the original image, so that a repaired video can be obtained. At this time, if the user views the repaired video, it is not necessary to repair each frame of image during video playing, so that the smooth feeling of the picture can be improved, and the picture quality can be improved.

In the embodiment of the application, through repairing the image of the static video, the smooth feeling of the picture can be ensured, and the image quality can be improved.

Fig. 6 is a schematic structural diagram of an embodiment of an image processing apparatus of the present application, and as shown in fig. 6, the image processing apparatus 60 may include: an acquisition module 61, a decomposition module 62, and an image processing module 63; wherein,

an acquisition module 61 for acquiring an image to be processed;

the decomposition module 62 is configured to decompose the image to be processed to obtain a plurality of sub-images;

and the image processing module 63 is configured to repair the image quality of the plurality of sub-images to obtain a repaired image to be processed.

In one possible implementation manner, the image processing module 63 is specifically configured to input the plurality of sub-images into a preset image library to be grouped to obtain a plurality of sub-image sets, where the preset image library includes a plurality of preset sub-image types, each sub-image set includes one or more sub-images, and each sub-image set corresponds to one preset sub-image type;

Calculating means and variances of the plurality of sub-image sets;

In one possible implementation manner, the image processing module 63 is further configured to perform image quality restoration on the restored image to be processed again, so as to implement iterative restoration.

In one possible implementation manner, the image processing apparatus 60 further includes:

The image processing apparatus 60 provided in the embodiment shown in fig. 6 may be used to implement the technical solution of the method embodiment shown in the present application, and the implementation principle and technical effects may be further referred to in the related description of the method embodiment.

It should be understood that the above division of the respective modules of the image processing apparatus 60 shown in fig. 6 is merely a division of a logic function, and may be integrated into one physical entity in whole or in part or may be physically separated. And these modules may all be implemented in software in the form of calls by the processing element; or can be realized in hardware; it is also possible that part of the modules are implemented in the form of software called by the processing element and part of the modules are implemented in the form of hardware. For example, the detection module may be a separately established processing element or may be implemented integrated in a certain chip of the electronic device. The implementation of the other modules is similar. In addition, all or part of the modules can be integrated together or can be independently implemented. In implementation, each step of the above method or each module above may be implemented by an integrated logic circuit of hardware in a processor element or an instruction in a software form.

For example, the modules above may be one or more integrated circuits configured to implement the methods above, such as: one or more specific integrated circuits (Application Specific Integrated Circuit; hereinafter ASIC), or one or more microprocessors (Digital Signal Processor; hereinafter DSP), or one or more field programmable gate arrays (Field Programmable Gate Array; hereinafter FPGA), etc. For another example, the modules may be integrated together and implemented in the form of a System-On-a-Chip (SOC).

In the above embodiments, the processor may include, for example, a CPU, a DSP, a microcontroller, or a digital signal processor, and may further include a GPU, an embedded Neural Network Processor (NPU) and an image signal processor (Image Signal Processing; ISP), where the processor may further include a necessary hardware accelerator or a logic processing hardware circuit, such as an ASIC, or one or more integrated circuits for controlling the execution of the program in the technical solution of the present application, and so on. Further, the processor may have a function of operating one or more software programs, which may be stored in a storage medium.

Embodiments of the present application also provide a computer-readable storage medium having a computer program stored therein, which when run on a computer, causes the computer to perform the methods provided by the embodiments shown in the present application.

Embodiments of the present application also provide a computer program product comprising a computer program which, when run on a computer, causes the computer to perform the methods provided by the embodiments shown in the present application.

In the embodiments of the present application, "at least one" means one or more, and "a plurality" means two or more. "and/or", describes an association relation of association objects, and indicates that there may be three kinds of relations, for example, a and/or B, and may indicate that a alone exists, a and B together, and B alone exists. Wherein A, B may be singular or plural. The character "/" generally indicates that the context-dependent object is an "or" relationship. "at least one of the following" and the like means any combination of these items, including any combination of single or plural items. For example, at least one of a, b and c may represent: a, b, c, a and b, a and c, b and c or a and b and c, wherein a, b and c can be single or multiple.

Those of ordinary skill in the art will appreciate that the various elements and algorithm steps described in the embodiments disclosed herein can be implemented as a combination of electronic hardware, computer software, and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

It will be clear to those skilled in the art that, for convenience and brevity of description, specific working procedures of the above-described systems, apparatuses and units may refer to corresponding procedures in the foregoing method embodiments, and are not repeated herein.

In several embodiments provided herein, any of the functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a computer-readable storage medium. Based on such understanding, the technical solution of the present application may be embodied essentially or in a part contributing to the prior art or in a part of the technical solution, in the form of a software product stored in a storage medium, including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in the embodiments of the present application. And the aforementioned storage medium includes: a usb disk, a removable hard disk, a Read-Only Memory (hereinafter referred to as ROM), a random access Memory (Random Access Memory) and various media capable of storing program codes such as a magnetic disk or an optical disk.

The foregoing is merely specific embodiments of the present application, and any person skilled in the art may easily conceive of changes or substitutions within the technical scope of the present application, which should be covered by the protection scope of the present application. The protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. An image processing method, the method comprising:

acquiring an image to be processed;

decomposing the image to be processed to obtain a plurality of sub-images;

2. The method of claim 1, wherein performing image quality restoration on the plurality of sub-images comprises:

calculating means and variances of the plurality of sub-image sets;

3. The method according to claim 2, characterized in that the type of the preset image library is determined by the type of the image to be processed.

4. A method according to any one of claims 1-3, characterized in that the method further comprises:

5. The method of claim 4, wherein the number of iterative repairs is preset; or the number of iterative repairs is determined according to user requirements.

6. The method of claim 4, wherein before performing image quality restoration again on the restored image to be processed to implement iterative restoration, the method further comprises:

and judging the image quality of the restored image to be processed.

7. The method according to any one of claims 1-6, wherein the image to be processed is each frame of image extracted sequentially during video playback.

8. The method according to any one of claims 1-6, wherein the image to be processed is all images extracted from a still video.

9. An electronic device, comprising: a processor and a memory for storing a computer program; the processor is configured to execute the computer program to implement the image processing method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program which, when run on a computer, implements the image processing method according to any one of claims 1-8.