CN116996778A

CN116996778A - Image restoration method and device, storage medium and electronic equipment

Info

Publication number: CN116996778A
Application number: CN202210414052.8A
Authority: CN
Inventors: 黄华进
Original assignee: Anker Innovations Co Ltd
Current assignee: Anker Innovations Co Ltd
Priority date: 2022-04-18
Filing date: 2022-04-18
Publication date: 2023-11-03

Abstract

The application discloses an image restoration method, an image restoration device, a storage medium and electronic equipment, wherein the method comprises the following steps: acquiring an audio signal output by audio playing equipment, acquiring an initial image by adopting image pickup equipment, acquiring instantaneous frequency and instantaneous volume corresponding to acquisition time of the initial image in the audio signal, and if the instantaneous frequency exists in a resonance frequency range, acquiring resonance amplitude and resonance vibration direction corresponding to the instantaneous volume in a repair correspondence table corresponding to the audio playing equipment and the image pickup equipment, and performing repair processing on the initial image based on the resonance amplitude and the resonance vibration direction to obtain a repair image. According to the application, the repair correspondence table is adopted, and the audio frequency and the volume when the image is acquired are combined, so that the image blurred due to the influence of audio resonance is repaired, and the image repair effect and the image repair efficiency are improved.

Description

Image restoration method and device, storage medium and electronic equipment

Technical Field

The present application relates to the field of computer technologies, and in particular, to an image restoration method, an image restoration device, a storage medium, and an electronic device.

Background

When the image pickup device is positioned in the cavity of the audio playing device, the image pickup device is adopted to acquire images when the audio playing device outputs audio, so that the images acquired by the image pickup device can be affected by the resonance of the audio output by the audio playing device, the phenomenon of image blurring occurs, and a method for repairing the blurred images affected by the audio is needed.

Disclosure of Invention

The embodiment of the application provides an image restoration method, an image restoration device, a storage medium and electronic equipment, which can restore an image blurred due to the influence of audio resonance by adopting a restoration correspondence table and combining audio frequency and volume when an image is acquired, thereby improving the image restoration effect and the image restoration efficiency. The technical scheme is as follows:

in a first aspect, an embodiment of the present application provides an image restoration method, where the method includes:

acquiring an audio signal output by audio playing equipment, and acquiring an initial image by adopting camera equipment;

acquiring the instantaneous frequency and the instantaneous volume corresponding to the acquisition time in the audio signal according to the acquisition time of the initial image;

if the instantaneous frequency exists in the resonance frequency range, obtaining a resonance amplitude and a resonance vibration direction corresponding to the instantaneous volume;

And repairing the initial image based on the resonance amplitude and the resonance vibration direction to obtain a repaired image.

According to the embodiment of the application, the repairing correspondence table is adopted, and the images blurred due to the influence of audio resonance are repaired by combining the audio frequency and the volume when the images are acquired, so that the image repairing effect and the image repairing efficiency are improved.

Optionally, before the acquiring the audio signal output by the audio playing device and acquiring the initial image by using the image capturing device, the method further includes:

acquiring a resonance frequency range in which an acquisition picture of an image pickup apparatus vibrates, and acquiring a resonance audio frequency in the resonance frequency range in which the acquisition picture has a maximum vibration amplitude;

the step of obtaining the resonance amplitude and the resonance vibration direction corresponding to the instantaneous volume comprises the following steps:

and based on the resonance audio frequency, obtaining a resonance amplitude and a resonance vibration direction corresponding to the instantaneous volume.

Based on the embodiment, the resonance frequency range is acquired, the repair corresponding table is acquired, the image in the resonance frequency range can be repaired, the power consumption required by the image repair device is reduced, the amplitude and the vibration direction are directly searched in the repair corresponding table, and the image repair efficiency is improved.

Optionally, the acquiring a resonance frequency range in which the acquired picture of the image capturing apparatus vibrates, and acquiring a resonance audio frequency in the resonance frequency range in which the acquired picture has a maximum vibration amplitude, includes:

outputting at least one single-frequency audio by adopting an audio playing device;

acquiring a single-frequency audio set which enables an acquisition picture of the image pickup device to vibrate from the at least one single-frequency audio, and acquiring a resonance frequency range based on the frequency of each single-frequency audio in the single-frequency audio set;

and acquiring a resonance audio frequency with the maximum vibration amplitude of the acquired picture in the resonance frequency range.

Based on the embodiment, the vibration resonance frequency range of the acquired picture is acquired in at least one single-frequency audio, and the accuracy of the resonance frequency range is improved, so that the accuracy of repairing the corresponding table is improved.

Optionally, the obtaining, based on the resonant audio frequency, a resonant amplitude and a resonant vibration direction corresponding to the instantaneous volume includes:

based on the resonance audio frequency, acquiring a repair correspondence table corresponding to the image pickup device and the audio playing device, wherein the repair correspondence table is a correspondence table of instantaneous volume, resonance amplitude and resonance vibration direction in the resonance frequency range;

And acquiring resonance amplitude and resonance vibration direction corresponding to the instantaneous volume in the repair correspondence table.

Based on the embodiment, the resonance amplitude and the resonance vibration direction corresponding to the instantaneous volume are searched in the repair corresponding table, so that the image repair effect and the image repair efficiency are improved.

Optionally, the acquiring, based on the resonant audio frequency, a repair correspondence table corresponding to the image capturing device and the audio playing device includes:

obtaining resonance amplitudes and resonance vibration directions corresponding to all sound volumes in the sound volume set under the resonance audio frequency;

and generating a repair corresponding table corresponding to the image pickup device and the audio playing device based on the all sound volumes and the resonance amplitudes and resonance vibration directions corresponding to the all sound volumes.

Based on the embodiment, the repairing correspondence table is generated according to the resonance amplitudes and the resonance vibration directions corresponding to all the sound volumes, so that the accuracy of the repairing correspondence table is further improved, and the image repairing effect and the image repairing efficiency are improved.

Optionally, the acquiring the resonant amplitude and the resonant vibration direction corresponding to all the volumes in the volume set at the resonant audio frequency includes:

Acquiring target volume in the volume set, and outputting audio meeting the resonant audio frequency and the target volume by adopting audio playing equipment;

acquiring a target image by adopting the camera equipment, and acquiring a spectrum polar coordinate image of the target image;

and performing linear detection calculation on the frequency spectrum polar coordinate image to obtain a target resonance amplitude and a target resonance vibration direction corresponding to the target volume.

Based on the embodiment, the linear detection calculation is carried out on the frequency spectrum polar coordinate image of the target image to obtain the target resonance vibration direction, then the target resonance amplitude is calculated, and the accuracy of the repair correspondence table is improved, so that the image repair effect and the image repair efficiency are improved.

Optionally, the acquiring the target image by using the image capturing device, acquiring a spectrum polar coordinate image of the target image includes:

acquiring a target image by adopting the image pickup equipment, and performing filtering processing on the target image;

and carrying out Fourier transform processing and polar coordinate transformation processing on the target image after the filtering processing in sequence to obtain a frequency spectrum polar coordinate image of the target image.

Based on the embodiment, the target image is subjected to filtering processing, so that the noise influence in the target image is removed, and the accuracy of the target resonance amplitude and the target resonance vibration direction is improved, so that the accuracy of repairing the corresponding table is improved.

Optionally, the repairing processing is performed on the initial image based on the resonance amplitude and the resonance vibration direction to obtain a repaired image, including:

performing Fourier transform processing on the initial image to obtain first image data;

repairing the first image processing by adopting a vibration fuzzy model based on the resonance amplitude and the resonance vibration direction to obtain second image data;

and carrying out inverse Fourier transform processing on the second image data to obtain a repair image.

Based on the embodiment, the resonance amplitude and the resonance vibration direction are obtained, the repairing image can be obtained by substituting the Fourier transform of the initial image into the vibration blurred image, and the image repairing efficiency is improved.

In a second aspect, an embodiment of the present application provides an image restoration apparatus, including: audio playback apparatus, image pickup apparatus, and processor; wherein, the liquid crystal display device comprises a liquid crystal display device,

one end of the processor is connected with the audio playing component, and the other end of the processor is connected with the camera shooting component;

the audio playing component outputs an audio signal and sends the audio signal to the processor;

the camera shooting assembly collects an initial image and sends the initial image to the processor;

The processor acquires the instantaneous frequency and the instantaneous volume corresponding to the acquisition time of the initial image from the audio signal;

the processor acquires a resonance amplitude and a resonance vibration direction corresponding to the instantaneous volume if the instantaneous frequency exists in a resonance frequency range;

and the processor performs restoration processing on the initial image based on the resonance amplitude and the resonance vibration direction to obtain a restoration image.

In a third aspect, embodiments of the present application provide a computer storage medium storing a plurality of instructions adapted to be loaded by a processor and to perform the above-described method steps.

In a fourth aspect, an embodiment of the present application provides an electronic device, which may include: a processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and to perform the above-mentioned method steps.

In one or more embodiments of the present application, an audio signal output by an audio playing device is acquired, an image capturing device is used to acquire an initial image, an instantaneous frequency and an instantaneous volume corresponding to an acquisition time of the initial image are acquired in the audio signal, if the instantaneous frequency exists in a resonance frequency range, a resonance amplitude and a resonance vibration direction corresponding to the instantaneous volume are acquired in a repair correspondence table corresponding to the audio playing device and the image capturing device, and repair processing is performed on the initial image based on the resonance amplitude and the resonance vibration direction, so as to obtain a repair image. The repair correspondence table is adopted, and the audio frequency and the volume when the image is acquired are combined, so that the image blurred due to the influence of audio resonance is repaired, and the image repair effect and the image repair efficiency are improved.

Drawings

In order to more clearly illustrate the embodiments of the application or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described, it being obvious that the drawings in the following description are only some embodiments of the application, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

FIG. 1 is an exemplary schematic diagram of an image restoration provided by an embodiment of the present application;

FIG. 2 is a schematic flow chart of an image restoration method according to an embodiment of the present application;

FIG. 3 is a schematic flow chart of an image restoration method according to an embodiment of the present application;

FIG. 4 is a graph of a Bessel function provided by an embodiment of the present application;

fig. 5 is a schematic structural diagram of an image restoration device according to an embodiment of the present application;

fig. 6 is a schematic structural diagram of an image restoration device according to an embodiment of the present application;

fig. 7 is a schematic structural diagram of an image restoration device according to an embodiment of the present application;

FIG. 8 is a schematic diagram of a resonant data acquisition module according to an embodiment of the present application;

Fig. 9 is a schematic structural diagram of an electronic device according to an embodiment of the present application.

Detailed Description

The following description of the embodiments of the present application will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present application, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application.

The image restoration method provided by the embodiment of the application can be realized by depending on a computer program and can be operated on an image restoration device based on a von neumann system. The computer program may be integrated in the application or may run as a stand-alone tool class application. The image restoration device in the embodiment of the application can comprise an audio playing component and a camera shooting component, and can also be connected with and control the camera shooting equipment and the audio playing equipment, and it can be understood that the camera shooting component can be the camera shooting equipment, can acquire the image of the current scene in real time, and the audio playing component can be the audio playing equipment and can output audio signals. As shown in fig. 1, an exemplary schematic diagram of image restoration is provided for an embodiment of the present application, where when an audio playing device outputs audio, if a natural frequency of a camera system of the image capturing device is in a frequency range of the audio, resonance is caused due to energy conduction, and accordingly, an image acquired by the image capturing device is dithered to obtain a blurred image, and an image restoration device may use a vibration blur model to perform restoration processing on an initial image acquired by the image capturing device to obtain a restored image. For an optical system, when the input object is a point light source, the light field distribution of the output image is called a point spread function (Point Spread Function, PSF), which is also called a point spread function, and in mathematics, a point light source can be represented by a point pulse function, and the light field distribution of the output image is called an impulse response, so the point spread function is also called an impulse response function of the optical system. The static normal image, that is, the restored image expected to be acquired after the restoration of the image restoration device, may be set as f (x, y), where x and y are the abscissa of the image, f (x, y) may be the gray value of the pixel point on the coordinate (x, y), and if the PSF is h (x, y), the vibration blurred image obtained after the static normal image is affected by resonance, that is, the initial image acquired by the image capturing device is g (x, y), and g (x, y) is the convolution of the restored image f (x, y) and the PSF, where the formula is as follows:

g(x,y)＝f(x,y)*h(x,y)

The resonance caused by the audio frequency is high-frequency vibration, so that the combination of a plurality of uniform linear motions can be understood in a small time range, the gray value of any point in the image acquired by the image pickup device is in direct proportion to the integral of the light intensity radiated to the point in the imaging process in the exposure time, and the initial image can be superposition of motion transient, and the formula is as follows:

wherein t is _e X is exposure time of the image pickup apparatus _m Is the displacement of x in dt time, y _m Is the displacement of y in dt time.

Let F (u, v) be the Fourier transform of F (x, y) and G (u, v) be the Fourier transform of G (x, y), and based on the linearity and displacement properties of the Fourier transform, the frequency domain relationship between the original image and the restored image is obtained as follows:

the image pickup apparatus is resonance due to audio played by the audio playing apparatus, the phase displacement is s (t), and the formula is as follows:

wherein D is the offset of the resonance equilibrium position relative to the static position, A is the resonance amplitude, f is the resonance vibration frequency,the initial phase of vibration corresponding to the acquisition time of the initial image. Assuming that the resonance vibration direction is θ, let h=ucosθ+vsin θ, the following formula can be obtained:

due to the high frequency vibration, it is possible to obtain: g (u, v) =f (u, v) e ^-2πjDh J ₀ (2pi hA), then taking the mode of two sides of the equation, and recovering h to obtain the following formula, namely a vibration blur model for repairing in the image repairing device:

|G(u,v)|＝|F(u,v)|·|J ₀ (2πA(ucosθ+vcosθ))|

wherein J is ₀ As a zero-order Bessel (Bessel) function of the first class. It can be understood that the image restoration device only needs to acquire the resonance amplitude and the resonance vibration direction corresponding to the initial image acquisition time, then perform fourier transformation on the initial image and bring the initial image into the vibration fuzzy model to obtain fourier transformation of the restored image, and then perform inverse fourier transformation to obtain the restored image.

The image restoration method provided by the application is described in detail below with reference to specific examples.

Referring to fig. 2, a flowchart of an image restoration method is provided in an embodiment of the present application. As shown in fig. 2, the method of the embodiment of the present application may include the following steps S101 to S104.

S101, acquiring an audio signal output by an audio playing device, and acquiring an initial image by adopting an image capturing device.

Specifically, the image capturing device is located in a cavity of the audio playing device, the image repairing device can output an audio signal by adopting the audio playing device, and acquire an initial image by adopting the image capturing device to acquire the audio signal output by the audio playing device.

S102, acquiring the instantaneous frequency and the instantaneous volume corresponding to the acquisition time in the audio signal according to the acquisition time of the initial image.

Specifically, the image restoration device may acquire the acquisition time of the initial image, acquire a time point corresponding to the acquisition time of the initial image in the audio signal, and acquire the instantaneous frequency and the instantaneous volume of the audio signal at the time point. It can be understood that the audio signal can be subjected to fourier transformation to obtain an audio function corresponding to the audio signal, and the amplitude corresponding to the acquisition time in the audio function is obtained, namely the instantaneous volume corresponding to the acquisition time.

S103, if the instantaneous frequency exists in the resonance frequency range, the resonance amplitude and the resonance vibration direction corresponding to the instantaneous volume are obtained.

Specifically, it is understood that not all the images acquired by the image capturing apparatus at all times are affected by the audio playing apparatus, and only when the audio played by the audio playing apparatus is within the resonance frequency range, the images acquired by the image capturing apparatus are affected by vibration. If the instantaneous frequency exists in the resonance frequency range, the initial image acquired by the image pickup device is a blurred image affected by vibration, repair processing is needed, the image repair device can calculate random resonance amplitude and resonance vibration direction under the instantaneous volume according to the resonance frequency range, and the resonance amplitude and resonance vibration direction corresponding to the instantaneous volume can be searched in a repair corresponding table corresponding to the audio playing device and the image pickup device. The repair correspondence table is a correspondence table of instantaneous volume, resonance amplitude and resonance vibration direction in a resonance frequency range, and it can be understood that if one audio playing device and one image capturing device are set as one set of combined device, the resonance frequency ranges corresponding to different combined devices are different, and the corresponding repair correspondence tables are also different.

S104, repairing the initial image based on the resonance amplitude and the resonance vibration direction to obtain a repaired image.

Specifically, the resonance amplitude and the resonance vibration direction are obtained, the image restoration device can perform Fourier transform on the initial image, the initial image after the Fourier transform is brought into the vibration fuzzy model, the Fourier transform of the restoration image is obtained, and then the inverse Fourier transform is performed to obtain the restoration image.

In the embodiment of the application, an audio signal output by audio playing equipment is acquired, an initial image is acquired by adopting image pickup equipment, instantaneous frequency and instantaneous volume corresponding to the acquisition time of the initial image are acquired in the audio signal, if the instantaneous frequency exists in a resonance frequency range, resonance amplitude and resonance vibration direction corresponding to the instantaneous volume are acquired in a repair correspondence table corresponding to the audio playing equipment and the image pickup equipment, and repair processing is carried out on the initial image based on the resonance amplitude and the resonance vibration direction, so as to obtain a repair image. The repair correspondence table is adopted, and the audio frequency and the volume when the image is acquired are combined, so that the image blurred due to the influence of audio resonance is repaired, and the image repair effect and the image repair efficiency are improved.

Referring to fig. 3, a flowchart of an image restoration method is provided in an embodiment of the present application. As shown in fig. 3, the method of the embodiment of the present application may include the following steps S201 to S207.

S201, a resonance frequency range in which the acquisition screen of the image pickup apparatus vibrates is acquired, and a resonance audio frequency in which the acquisition screen has the maximum vibration amplitude is acquired in the resonance frequency range.

Specifically, images acquired by the image capturing device at all times are not affected by the audio playing device, and only when audio played by the audio playing device is in the resonance frequency range, the images acquired by the image capturing device are affected by vibration. It can be understood that if one audio playing device and one image capturing device are set as one set of combined device, the resonance frequency ranges corresponding to the different combined devices are different, and the corresponding repair correspondence tables are also different, so that the image repairing device needs to acquire the resonance frequency ranges and the repair correspondence tables corresponding to the image capturing device and the audio playing device.

The image restoration device may output at least one single-frequency audio by using an audio playing device, sequentially output the single-frequency audio and acquire a picture by using an image capturing device, acquire the single-frequency audio which causes the acquired picture of the image capturing device to vibrate in the at least one single-frequency audio, store the single-frequency audio in a single-frequency audio set, and then obtain a resonance frequency range according to the frequency of each single-frequency audio in the single-frequency audio set, for example, the highest frequency is 30.5Hz and the lowest frequency is 31.2Hz in the frequencies of each single-frequency audio in the single-frequency audio set, and then the resonance frequency range is 30.5Hz to 31.2Hz. The image restoration device may then acquire a frequency in the resonance frequency range such that the acquired picture of the image pickup apparatus has the maximum vibration amplitude, and confirm the frequency as the resonance audio frequency.

Alternatively, the image restoration device may acquire an initial frequency range, for example, 20Hz to 20kHz, acquire at least one first frequency in the initial frequency range with a first interval frequency as a step, and the frequency difference between adjacent first frequencies is the first interval frequency, for example, the first interval frequency is 10Hz, then the first frequency may be 20Hz, 30Hz, 40Hz … …, and the like, sequentially play all single-frequency audio meeting the first frequency, acquire the first frequency range based on the first frequency that causes the acquired image of the image capturing device to vibrate, for example, when the first frequency is 50Hz and 60Hz, then the first frequency range may be 40Hz to 70Hz.

At least one second frequency is obtained in the first frequency range by taking the second interval frequency as a step length, the frequency difference between every two adjacent second frequencies is the second interval frequency, the second interval frequency can be smaller than the first interval frequency and can be one tenth of the first interval frequency, for example, the second interval frequency is 1Hz, the second frequencies can be 40Hz, 41Hz, the first and second frequencies, 69Hz and 70Hz, all single-frequency audios meeting the second frequencies are sequentially played, the second frequency range is obtained based on the second frequency enabling the acquired images of the image pickup device to vibrate, for example, when the second frequency is 55Hz, the acquired images vibrate, and the second frequency range can be 54Hz to 56Hz.

At least one third frequency is obtained in the second frequency range by taking the third interval frequency as a step length, the frequency difference between every two adjacent third frequencies is the third interval frequency, the third interval frequency can be smaller than the second interval frequency and can be one tenth of the second interval frequency, for example, the third interval frequency is 0.1Hz, then the third frequency can be 54.1Hz, 54.2Hz,..2 Hz, 55.8Hz and 55.9Hz, all single-frequency audios meeting the third frequency are sequentially played, and a resonance frequency range is obtained based on the third frequency which enables the acquired picture of the image pickup device to vibrate, for example, when the third frequency is 54.5 Hz-55.5 Hz, then the resonance frequency range can be 54.5 Hz-55.5 Hz. If the third frequency is 55Hz, the maximum vibration amplitude exists in the vibration picture, and the resonance audio frequency is 55Hz.

S202, acquiring an audio signal output by an audio playing device, and acquiring an initial image by adopting an image capturing device.

S203, acquiring the instantaneous frequency and the instantaneous volume corresponding to the acquisition time in the audio signal according to the acquisition time of the initial image.

S204, if the instantaneous frequency exists in the resonance frequency range, a repair correspondence table corresponding to the imaging equipment and the audio playing equipment is obtained based on the resonance audio frequency, and the resonance amplitude and the resonance vibration direction corresponding to the instantaneous volume are obtained in the repair correspondence table.

Specifically, not all images acquired by the image capturing device at all times are affected by the audio playing device, and only when the audio played by the audio playing device is in the resonance frequency range, the images acquired by the image capturing device are affected by vibration. If the instantaneous frequency exists in the resonance frequency range, the initial image acquired by the image pickup device is a blurred image affected by vibration, repair processing is needed, the image repair device can calculate random resonance amplitude and resonance vibration direction under the instantaneous volume according to the resonance frequency range, and the resonance amplitude and resonance vibration direction corresponding to the instantaneous volume can be searched in a repair corresponding table corresponding to the audio playing device and the image pickup device. The repair correspondence table is a correspondence table of instantaneous volume, resonance amplitude and resonance vibration direction in a resonance frequency range, and it can be understood that if one audio playing device and one image capturing device are set as one set of combined device, the resonance frequency ranges corresponding to different combined devices are different, and the corresponding repair correspondence tables are also different. If the instantaneous frequency exists in the resonance frequency range, the initial image acquired by the image pickup device is a blurred image affected by vibration, and repair processing is needed, and the image repair device can search the resonance amplitude and the resonance vibration direction corresponding to the instantaneous volume in a repair corresponding table corresponding to the audio playing device and the image pickup device.

Optionally, the image restoration device may acquire resonance amplitudes and resonance vibration directions corresponding to all volumes in the volume set at the resonance audio frequency, and then generate a restoration correspondence table corresponding to the image capturing apparatus and the audio playing apparatus based on all volumes and the resonance amplitudes and resonance vibration directions corresponding to all volumes.

The volume set is a set of all volumes that can be played by the audio playing device, for example, the volume range that can be played by the audio playing device is 0-100 db, and the volume set contains 0-100 db. The image restoration device can sequentially output the audios meeting all the audios in the volume set by adopting the audio playing equipment, the frequencies of the audios are resonance audio frequencies, and then resonance amplitudes and resonance vibration directions corresponding to all the audios are obtained. The image restoration device can generate restoration corresponding tables corresponding to the image pickup equipment and the audio playing equipment according to all the sound volumes and the resonance amplitudes and resonance vibration directions corresponding to all the sound volumes, wherein the restoration corresponding tables are corresponding relation tables of the instantaneous sound volumes, the resonance amplitudes and the resonance vibration directions in the resonance frequency range. When the image restoration device needs to restore the image, the resonance amplitude and the resonance vibration direction corresponding to the volume can be searched in the restoration corresponding table according to the volume.

Optionally, the image restoration device may acquire a target volume in the volume set, and output audio meeting the resonant audio frequency and the target volume with the audio playing device. Then, a target image is acquired by using the image pickup device, and then, a spectrum polar coordinate image of the target image is calculated, it is understood that the target image acquired by the image pickup device has a series of parallel dark fringes on a spectrum, and the positions of the dark fringes correspond to J ₀ Zero point of function, J ₀ The zero positions of the functions are related to the resonance amplitude and the resonance vibration direction, and the formula is as follows:

wherein A is resonance amplitude, θ is resonance vibration direction, X _i (i=0, 1,2, 3.) J ₀ And (3) a function zero point. Referring to fig. 4, a graph of a bessel function is provided according to an embodiment of the present application, as shown in fig. 4, the modulus of the bessel function is zero, and the value X greater than 0 is J ₀ Zero point of function, exampleFor example X ₀ 、X ₁ 、X ₂ 、X ₃ Etc. Points with the same frequency in the spectrum polar coordinate image form a straight line, so that the position corresponding to the parallel dark stripes in the spectrum sister coordinate image is theta. The image restoration device can perform straight line detection calculation on the frequency spectrum polar coordinate image to obtain a target resonance vibration direction corresponding to the target volume, and then calculate to obtain the target resonance amplitude.

Optionally, after the target image is acquired by using the image capturing device, the image restoration device may perform filtering processing on the target image, so as to eliminate noise existing in the target image and improve accuracy of restoration of the correspondence table. And then carrying out Fourier transform processing on the target image after the filter processing, and carrying out polar coordinate transform processing on the target image after the Fourier transform processing to obtain a frequency spectrum polar coordinate image of the target image.

S205, repairing the initial image based on the resonance amplitude and the resonance vibration direction to obtain a repaired image.

Specifically, the resonance amplitude and the resonance vibration direction are obtained, the image restoration device may perform fourier transform processing on the initial image G (x, y) to obtain first image data G (u, v), then, according to the resonance amplitude and the resonance vibration direction, bring the first image data into the vibration blur pattern to perform restoration processing to obtain second image data F (u, v), and then perform fourier transform processing on the second image data to obtain a restored image F (x, y).

In the embodiment of the application, single-frequency audio is sequentially output, the resonance frequency range which enables the acquired picture of the image pickup device to vibrate and the resonance audio frequency which enables the picture to vibrate maximally are obtained, then the resonance amplitudes and the resonance vibration directions corresponding to all sound volumes are obtained under the resonance audio frequency, a repair correspondence table is generated, the image repair device can conveniently search the resonance amplitudes and the resonance vibration directions, and the image repair efficiency is improved. And filtering processing is performed before the target image is acquired, so that the accuracy of repairing the corresponding table is improved. Acquiring an audio signal output by audio playing equipment, acquiring an initial image by adopting image pickup equipment, acquiring instantaneous frequency and instantaneous volume corresponding to the acquisition time of the initial image in the audio signal, judging whether the instantaneous frequency exists in a resonance frequency range, screening out an image needing to be repaired, reducing the power consumption required by an image repairing device, acquiring resonance amplitude and resonance vibration direction corresponding to the instantaneous volume in a repairing corresponding table corresponding to the audio playing equipment and the image pickup equipment if the instantaneous frequency exists in the resonance frequency range, and repairing the initial image based on the resonance amplitude and the resonance vibration direction to obtain a repaired image. The repair correspondence table is adopted, and the audio frequency and the volume when the image is acquired are combined, so that the image blurred due to the influence of audio resonance is repaired, and the image repair effect and the image repair efficiency are improved.

Referring to fig. 5, a schematic structural diagram of an image restoration device is provided in an embodiment of the present application. The image restoration device comprises an audio playing component, a camera shooting component and a processor, wherein the processor comprises a resonance frequency acquisition component, a correspondence table generation component and a restoration processing component. The camera shooting assembly is positioned in a cavity of the audio playing assembly, the audio playing assembly is connected with the resonant frequency acquisition assembly, the camera shooting assembly is connected with the resonant frequency acquisition assembly, the corresponding table generation assembly is connected with the repairing assembly, the audio playing assembly is connected with the repairing assembly, the camera shooting assembly price is connected with the repairing assembly, the corresponding table generation assembly is connected with the audio playing assembly, and the corresponding table generation assembly is connected with the camera shooting assembly.

The resonance frequency acquisition component acquires at least one single-frequency audio, sends the at least one single-frequency audio to the audio playing component, and instructs the audio playing component to sequentially output the single-frequency audio. The resonance frequency acquisition component sends a picture acquisition instruction to the camera component, only the camera component acquires pictures, and the camera component sends the acquired pictures to the resonance frequency acquisition component. The resonance frequency acquisition component acquires single-frequency audio which enables the acquired picture of the camera component to vibrate in the at least one single-frequency audio, the single-frequency audio is stored in a single-frequency audio set, then a resonance frequency range is obtained according to the frequency of each single-frequency audio in the single-frequency audio set, and then the resonance frequency acquisition component can acquire the frequency which enables the acquired picture of the camera component to have the maximum vibration amplitude in the resonance frequency range and confirm the frequency as the resonance audio frequency. The resonant frequency acquisition component sends the resonant frequency range and the resonant audio frequency to the correspondence table generating component.

The volume set is a set of all volumes that can be played by the audio playing device, for example, the volume range that can be played by the audio playing device is 0-100 db, and the volume set contains 0-100 db. The corresponding table generating component sends an audio playing instruction to the audio playing component, the audio playing component can sequentially output audio meeting all the volumes in the volume set according to the audio playing instruction, the frequencies of the audio are resonance audio frequencies, and then the corresponding table generating component can acquire resonance amplitudes and resonance vibration directions corresponding to all the volumes.

Optionally, the correspondence table generating component may obtain a target volume in the volume set, send an audio playing instruction containing the target volume to the audio playing component, and output audio meeting the resonant audio frequency and the target volume by the audio playing component. The mapping table generating component sends a target image acquisition instruction to the camera component, and the camera component acquires a target image based on the target image acquisition instruction and sends the target image to the mapping table generating component. The corresponding table generating component can perform filtering processing on the target image, eliminate noise in the target image, improve accuracy of repairing the corresponding table, perform Fourier transform processing on the target image after the filtering processing, perform polar coordinate transform processing on the target image after the Fourier transform processing to obtain a frequency spectrum polar coordinate image of the target image, and then perform linear detection calculation on the frequency spectrum polar coordinate image to obtain a target resonance vibration direction corresponding to target volume, and then calculate to obtain target resonance amplitude.

The corresponding table generating component can generate a repairing corresponding table corresponding to the camera shooting component and the audio playing component according to all the sound volumes and the resonance amplitude and resonance vibration direction corresponding to all the sound volumes, wherein the repairing corresponding table is a corresponding relation table of the instantaneous sound volumes, the resonance amplitude and the resonance vibration direction in the resonance frequency range. When the restoration processing component needs to restore the image, the resonance amplitude and the resonance vibration direction corresponding to the volume can be searched in the restoration corresponding table according to the volume. The correspondence table generating component sends the resonance frequency range, the resonance audio frequency, and the repair correspondence table to the repair processing component.

The restoration processing component can acquire the audio signal output by the audio playing component, send an image acquisition instruction to the camera component, acquire an initial image based on the image acquisition instruction, and send the initial image and the acquisition time of the initial image to the restoration processing component. The restoration processing component acquires a time point corresponding to the acquisition time of the initial image in the audio signal, and acquires the instantaneous frequency and the instantaneous volume of the audio signal at the time point. It can be understood that the audio signal can be subjected to fourier transformation to obtain an audio function corresponding to the audio signal, and the amplitude corresponding to the acquisition time in the audio function is obtained, namely the instantaneous volume corresponding to the acquisition time.

If the instantaneous frequency exists in the resonance frequency range, the initial image acquired by the image pickup assembly is a blurred image affected by vibration, and repair processing is needed, and the repair processing assembly can search the resonance amplitude and the resonance vibration direction corresponding to the instantaneous volume in the repair corresponding table. The resonance amplitude and the resonance vibration direction are obtained, the restoration processing component can perform fourier transform processing on the initial image G (x, y) to obtain first image data G (u, v), then the first image data is brought into a vibration blur pattern according to the resonance amplitude and the resonance vibration direction to perform restoration processing to obtain second image data F (u, v), and then fourier transform processing is performed on the second image data to obtain a restoration image F (x, y).

The following describes in detail the image restoration device provided in the embodiment of the present application with reference to fig. 6 to 8. It should be noted that, the image restoration apparatus shown in fig. 6-8 is used to perform the method of the embodiment shown in fig. 2 and 4, and for convenience of explanation, only the portion relevant to the embodiment of the present application is shown, and specific technical details are not disclosed, please refer to the embodiment shown in fig. 2 and 4 of the present application.

Referring to fig. 6, a schematic structural diagram of an image restoration device according to an exemplary embodiment of the present application is shown. The image restoration device may be implemented as all or part of the device by software, hardware, or a combination of both. The apparatus 1 comprises an image acquisition module 11, an instantaneous data acquisition module 12, a resonance data acquisition module 13 and a repair processing module 14.

An image acquisition module 11, configured to acquire an audio signal output by an audio playing device, and acquire an initial image by using a camera device;

an instantaneous data acquisition module 12, configured to acquire an instantaneous frequency and an instantaneous volume corresponding to the acquisition time in the audio signal according to the acquisition time of the initial image;

a resonance data acquisition module 13, configured to acquire a resonance amplitude and a resonance vibration direction corresponding to the instantaneous volume if the instantaneous frequency exists in a resonance frequency range;

And the restoration processing module 14 is used for carrying out restoration processing on the initial image based on the resonance amplitude and the resonance vibration direction to obtain a restoration image.

In this embodiment, an audio signal output by an audio playing device is acquired, an image capturing device is used to acquire an initial image, an instantaneous frequency and an instantaneous volume corresponding to the acquisition time of the initial image are acquired in the audio signal, if the instantaneous frequency exists in a resonance frequency range, a resonance amplitude and a resonance vibration direction corresponding to the instantaneous volume are acquired in a repair correspondence table corresponding to the audio playing device and the image capturing device, and repair processing is performed on the initial image based on the resonance amplitude and the resonance vibration direction, so as to obtain a repair image. The repair correspondence table is adopted, and the audio frequency and the volume when the image is acquired are combined, so that the image blurred due to the influence of audio resonance is repaired, and the image repair effect and the image repair efficiency are improved.

Referring to fig. 7, a schematic structural diagram of an image restoration device according to an exemplary embodiment of the present application is shown. The image restoration device may be implemented as all or part of the device by software, hardware, or a combination of both. The device 1 comprises an image acquisition module 11, an instantaneous data acquisition module 12, a resonance data acquisition module 13, a repair processing module 14 and a resonance frequency acquisition module 15

A resonance frequency acquisition module 15 for acquiring a resonance frequency range in which an acquisition screen of an image pickup apparatus is vibrated, and acquiring a resonance audio frequency in which the acquisition screen is vibrated to a maximum vibration amplitude;

optionally, the resonant frequency obtaining module 15 is specifically configured to output at least one single-frequency audio with an audio playing device;

Optionally, the resonance data acquisition module 13 is specifically configured to acquire a resonance amplitude and a resonance vibration direction corresponding to the instantaneous volume based on the resonance audio frequency.

Specifically, please refer to fig. 8, a schematic structural diagram of a resonance data acquisition module is provided in an embodiment of the present application. As shown in fig. 8, the resonance data acquisition module 13 may include:

a correspondence table obtaining unit 131, configured to obtain a repair correspondence table corresponding to the image capturing apparatus and the audio playing apparatus based on the resonance audio frequency, where the repair correspondence table is a correspondence table of instantaneous volume, resonance amplitude, and resonance vibration direction in the resonance frequency range;

and a resonance data acquisition unit 132, configured to acquire a resonance amplitude and a resonance vibration direction corresponding to the instantaneous volume in the repair correspondence table.

Optionally, the correspondence table obtaining unit 131 is specifically configured to obtain, at the resonant audio frequency, resonant amplitudes and resonant vibration directions corresponding to all volumes in the volume set;

Optionally, the correspondence table obtaining unit 131 is specifically configured to obtain a target volume in the volume set, and output, by using an audio playing device, audio that meets the resonant audio frequency and the target volume;

Optionally, the correspondence table acquiring unit 131 is specifically configured to acquire a target image by using the image capturing apparatus, and perform filtering processing on the target image;

Optionally, the restoration processing module 14 is specifically configured to perform fourier transform processing on the initial image to obtain first image data;

In this embodiment, single-frequency audio is sequentially output, a resonance frequency range in which the acquired picture of the image capturing apparatus vibrates and a resonance audio frequency in which the picture vibrates maximally are obtained, and then resonance amplitudes and resonance vibration directions corresponding to all volumes are obtained at the resonance audio frequency, so that a repair correspondence table is generated, which is convenient for the image repair device to search the resonance amplitudes and resonance vibration directions, and the efficiency of image repair is improved. And filtering processing is performed before the target image is acquired, so that the accuracy of repairing the corresponding table is improved. Acquiring an audio signal output by audio playing equipment, acquiring an initial image by adopting image pickup equipment, acquiring instantaneous frequency and instantaneous volume corresponding to the acquisition time of the initial image in the audio signal, judging whether the instantaneous frequency exists in a resonance frequency range, screening out an image needing to be repaired, reducing the power consumption required by an image repairing device, acquiring resonance amplitude and resonance vibration direction corresponding to the instantaneous volume in a repairing corresponding table corresponding to the audio playing equipment and the image pickup equipment if the instantaneous frequency exists in the resonance frequency range, and repairing the initial image based on the resonance amplitude and the resonance vibration direction to obtain a repaired image. The repair correspondence table is adopted, and the audio frequency and the volume when the image is acquired are combined, so that the image blurred due to the influence of audio resonance is repaired, and the image repair effect and the image repair efficiency are improved.

It should be noted that, in the image restoration apparatus provided in the foregoing embodiment, only the division of the foregoing functional modules is used as an example when the image restoration method is executed, and in practical application, the foregoing functional allocation may be performed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules, so as to complete all or part of the functions described above. In addition, the image restoration device and the image restoration method provided in the foregoing embodiments belong to the same concept, which embody the detailed implementation process in the method embodiment, and are not described herein again.

The foregoing embodiment numbers of the present application are merely for the purpose of description, and do not represent the advantages or disadvantages of the embodiments.

The embodiment of the present application further provides a computer storage medium, where the computer storage medium may store a plurality of instructions, where the instructions are adapted to be loaded by a processor and execute the image restoration method according to the embodiment shown in fig. 2 to fig. 4, and the specific execution process may refer to the specific description of the embodiment shown in fig. 2 to fig. 4, which is not repeated herein.

The present application further provides a computer program product, where at least one instruction is stored, where the at least one instruction is loaded by the processor and executed by the processor to implement the image restoration method according to the embodiment shown in fig. 2 to fig. 4, and the specific implementation process may refer to the specific description of the embodiment shown in fig. 2 to fig. 4, which is not repeated herein.

Referring to fig. 9, a block diagram of an electronic device according to an exemplary embodiment of the present application is shown. The electronic device of the present application may include one or more of the following components: processor 110, memory 120, input device 130, output device 140, and bus 150. The processor 110, the memory 120, the input device 130, and the output device 140 may be connected by a bus 150.

Processor 110 may include one or more processing cores. The processor 110 connects various parts within the overall electronic device using various interfaces and lines, performs various functions of the terminal 100 and processes data by executing or executing instructions, programs, code sets, or instruction sets stored in the memory 120, and invoking data stored in the memory 120. Alternatively, the processor 110 may be implemented in hardware in at least one of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), programmable logic array (Programmab Leogic Array, PLA). The processor 110 may integrate one or a combination of several of a central processing unit (Central Processing Unit, CPU), an image processor (Graphics Processing Unit, GPU), and a modem, etc. The CPU mainly processes an operating system, a user page, an application program and the like; the GPU is used for being responsible for rendering and drawing of display content; the modem is used to handle wireless communications. It will be appreciated that the modem may not be integrated into the processor 110 and may be implemented solely by a single communication chip.

The Memory 120 may include a random access Memory (Random Access Memory, RAM) or a Read-Only Memory (ROM). Optionally, the memory 120 includes a Non-transitory computer readable medium (Non-Transitory Computer-Readable Storage Medium). Memory 120 may be used to store instructions, programs, code, sets of codes, or sets of instructions. The memory 120 may include a stored program area and a stored data area, wherein the stored program area may store instructions for implementing an operating system, which may be an Android (Android) system, including an Android system-based deep development system, an IOS system developed by apple corporation, including an IOS system-based deep development system, or other systems, instructions for implementing at least one function (such as a touch function, a sound playing function, an image playing function, etc.), instructions for implementing the various method embodiments described above, and the like.

Memory 120 may be divided into an operating system space in which the operating system runs and a user space in which native and third party applications run. In order to ensure that different third party application programs can achieve better operation effects, the operating system allocates corresponding system resources for the different third party application programs. However, the requirements of different application scenarios in the same third party application program on system resources are different, for example, under the local resource loading scenario, the third party application program has higher requirement on the disk reading speed; in the animation rendering scene, the third party application program has higher requirements on the GPU performance. The operating system and the third party application program are mutually independent, and the operating system often cannot timely sense the current application scene of the third party application program, so that the operating system cannot perform targeted system resource adaptation according to the specific application scene of the third party application program.

In order to enable the operating system to distinguish specific application scenes of the third-party application program, data communication between the third-party application program and the operating system needs to be communicated, so that the operating system can acquire current scene information of the third-party application program at any time, and targeted system resource adaptation is performed based on the current scene.

The input device 130 is configured to receive input instructions or data, and the input device 130 includes, but is not limited to, a keyboard, a mouse, a camera, a microphone, or a touch device. The output device 140 is used to output instructions or data, and the output device 140 includes, but is not limited to, a display device, a speaker, and the like. In one example, the input device 130 and the output device 140 may be combined, and the input device 130 and the output device 140 are touch display screens.

The touch display screen may be designed as a full screen, a curved screen, or a contoured screen. The touch display screen may also be designed as a combination of a full screen and a curved screen, and the combination of a special-shaped screen and a curved screen, which is not limited in the embodiment of the present application.

In addition, those skilled in the art will appreciate that the configuration of the electronic device shown in the above-described figures does not constitute a limitation of the electronic device, and the electronic device may include more or less components than illustrated, or may combine certain components, or may have a different arrangement of components. For example, the electronic device further includes components such as a radio frequency circuit, an input unit, a sensor, an audio circuit, a wireless fidelity (Wireless Fidelity, wiFi) module, a power supply, and a bluetooth module, which are not described herein.

In the electronic device shown in fig. 9, the processor 110 may be configured to invoke the image restoration application program stored in the memory 120, and specifically perform the following operations:

In one embodiment, before executing the acquisition of the audio signal output by the audio playing device, the processor 110 further executes the following operations, before acquiring the initial image with the image capturing device:

when the resonant amplitude and the resonant vibration direction corresponding to the instantaneous volume are acquired, the following operations are specifically executed:

In one embodiment, the processor 110, when executing the acquisition of the resonance frequency range in which the acquisition screen of the image capturing apparatus vibrates, acquires the resonance audio frequency in the resonance frequency range in which the acquisition screen has the maximum vibration amplitude, specifically executes the following operations:

In one embodiment, the processor 110, when executing the acquisition of the resonance amplitude and the resonance vibration direction corresponding to the instantaneous volume based on the resonance audio frequency, specifically executes the following operations:

In one embodiment, the processor 110, when executing the retrieval of the repair correspondence table corresponding to the image capturing apparatus and the audio playing apparatus based on the resonance audio frequency, specifically executes the following operations:

In one embodiment, the processor 110, when executing the acquisition of the resonant amplitudes and resonant vibration directions corresponding to all volumes in the volume set at the resonant audio frequency, specifically performs the following operations:

In one embodiment, the processor 110, when executing the capturing of the target image using the image capturing apparatus, specifically performs the following operations:

In one embodiment, the processor 110 performs the following operations when performing the repair process on the initial image based on the resonance amplitude and the resonance vibration direction to obtain a repair image:

Those skilled in the art will appreciate that implementing all or part of the above-described methods in accordance with the embodiments may be accomplished by way of a computer program stored on a computer readable storage medium, which when executed may comprise the steps of the embodiments of the methods described above. The storage medium may be a magnetic disk, an optical disk, a read-only memory, a random access memory, or the like.

The foregoing disclosure is illustrative of the present application and is not to be construed as limiting the scope of the application, which is defined by the appended claims.

Claims

1. A method of image restoration, the method comprising:

2. The method of claim 1, wherein the acquiring the audio signal output by the audio playback device, before the capturing the initial image with the image capturing device, further comprises:

3. The method according to claim 2, wherein the acquiring a resonance frequency range in which an acquisition screen of an image pickup apparatus is vibrated, acquiring a resonance audio frequency in the resonance frequency range in which the acquisition screen is vibrated at a maximum vibration amplitude, includes:

4. The method according to claim 2, wherein the acquiring the resonance amplitude and the resonance vibration direction corresponding to the instantaneous volume based on the resonance audio frequency includes:

5. The method according to claim 4, wherein the acquiring the repair correspondence table corresponding to the image capturing apparatus and the audio playback apparatus based on the resonance audio frequency includes:

6. The method of claim 5, wherein the acquiring resonant amplitudes and resonant vibration directions for all volumes in the volume set at the resonant audio frequency comprises:

7. The method of claim 6, wherein the acquiring a target image with the imaging device, acquiring a spectral polar image of the target image, comprises:

8. The method of claim 1, wherein the repairing the initial image based on the resonance amplitude and the resonance vibration direction to obtain a repaired image comprises:

9. An image restoration device, the device comprising: the device comprises an audio playing component, a camera component and a processor; wherein, the liquid crystal display device comprises a liquid crystal display device,

10. A computer storage medium storing a plurality of instructions adapted to be loaded by a processor and to perform the method steps of any one of claims 1 to 8.

11. An electronic device, comprising: a processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and to perform the method steps of any of claims 1-8.