CN115496671A

CN115496671A - Image noise reduction device and method, processing chip and electronic equipment

Info

Publication number: CN115496671A
Application number: CN202110683120.6A
Authority: CN
Inventors: 李彦良
Original assignee: Zeku Technology Shanghai Corp Ltd
Current assignee: Zeku Technology Shanghai Corp Ltd
Priority date: 2021-06-18
Filing date: 2021-06-18
Publication date: 2022-12-20

Abstract

The embodiment of the application discloses an image noise reduction device and method, a processing chip and electronic equipment. The image noise reduction device includes: the device comprises an image processing module, a filtering processing module and a noise reduction processing module; the image processing module is used for matching a first feature point data window in an image block of a reference frame in an image block of a current frame to output image affine transformation information and parallelly detecting feature points of the image block of the current frame to obtain a second feature point data window, and the reference frame is positioned in front of the current frame in time sequence; the filtering processing module is used for carrying out time-space domain smooth filtering processing on the affine transformation information of the image so as to output motion adjustment information; and the noise reduction processing module is used for performing rotation alignment processing on the reference frame by adopting the motion adjustment information to obtain a first image frame, and performing noise reduction filtering processing on the current frame and/or the first image frame to output a second image frame, so that the stability and the accuracy of image noise reduction processing are improved.

Description

Image noise reduction device and method, processing chip and electronic equipment

Technical Field

The application relates to the technical field of computers, in particular to an image noise reduction device and method, a processing chip and electronic equipment.

Background

Currently, in the video denoising technology, it is generally necessary to perform affine image transformation (affine transformation) on image blocks of image frames in a video stream to obtain affine image transformation information, so as to perform operations such as rotation alignment processing on the image frames by using the affine image transformation information. However, in the actual video denoising process, the image affine transformation information is directly used to perform the rotation alignment process on the image frame, so that the rotation alignment effect of the image frame is poor, and the stability and accuracy of the image denoising process are affected.

Disclosure of Invention

The embodiment of the application provides an image noise reduction device and method, a processing chip and electronic equipment, and aims to output motion adjustment information by performing time-space domain smoothing filtering on affine transformation information of an image so as to obtain more accurate and reasonable motion adjustment information, improve the accuracy of rotation alignment of a reference frame and improve the stability and accuracy of image noise reduction processing.

In a first aspect, an embodiment of the present application provides an image noise reduction apparatus, including: the device comprises an image processing module, a filtering processing module and a noise reduction processing module; wherein, the first and the second end of the pipe are connected with each other,

the image processing module is used for matching a first feature point data window in an image block of a reference frame in an image block of a current frame to output image affine transformation information, and performing feature point detection on the image block of the current frame in parallel to obtain a second feature point data window, wherein the reference frame is positioned in front of the current frame in time sequence;

the filtering processing module is used for performing time-space domain smoothing filtering processing on the image affine transformation information to output motion adjustment information;

the noise reduction processing module is configured to perform rotation alignment processing on the reference frame by using the motion adjustment information to obtain a first image frame, and perform noise reduction filtering processing on the current frame and/or the first image frame to output a second image frame.

It can be seen that, first, the image processing module performs matching of a first feature point data window in an image block of a reference frame in an image block of a current frame and feature point detection of the image block of the current frame in parallel; thirdly, the filter processing module carries out time-space domain smooth filter processing on the affine transformation information of the image so as to output motion adjustment information; and finally, the noise reduction processing model adopts the motion adjustment information to carry out rotation alignment processing on the reference frame to obtain a first image frame, and carries out noise reduction filtering processing on the current frame and/or the first image frame to output a second image frame. Because the time-space domain smooth filtering processing is carried out on the affine transformation information of the image to obtain the motion adjustment information, more accurate and reasonable motion adjustment information is obtained, the accuracy of the rotational alignment of the reference frame is improved, the stability and the accuracy of the noise reduction filtering processing of the current frame and/or the rotationally aligned reference frame are improved, the time domain noise reduction effect presented by the second image frame is better, and the stability and the accuracy of the noise reduction processing of the image are improved.

In a second aspect, an embodiment of the present application provides an image denoising method, including:

acquiring a first characteristic point data window and a current frame in an image block of a reference frame, wherein the reference frame is positioned in front of the current frame in time sequence;

matching the first feature point data window in the image block of the current frame to obtain image affine transformation information, and performing feature point detection on the image block of the current frame in parallel to obtain a second feature point data window;

carrying out time-space domain smoothing filtering processing on the image affine transformation information to obtain motion adjustment information;

performing rotation alignment processing on the reference frame by using the motion adjustment information to obtain a first image frame;

and carrying out noise reduction filtering processing on the current frame and/or the first image frame to obtain a second image frame.

It can be seen that, first, a first feature point data window (i.e. feature point detection does not need to be performed) and a current frame in an image block of a reference frame are directly obtained; secondly, matching of a first feature point data window of the reference frame in an image block of the current frame and feature point detection of the image block of the current frame are executed in parallel; thirdly, performing time-space domain smoothing filtering processing on the affine transformation information of the image to obtain motion adjustment information; and finally, the reference frame is rotationally aligned by adopting the motion adjustment information to obtain a first image frame, and the current frame and/or the first image frame are subjected to noise reduction filtering processing to output a second image frame. Because the motion adjustment information is obtained by performing time-space domain smoothing filtering processing on the image affine transformation information, more accurate and reasonable motion adjustment information is obtained, the accuracy of the rotational alignment of the reference frame is improved, the stability and the accuracy of noise reduction filtering processing on the current frame and/or the rotationally aligned reference frame are improved, the time domain noise reduction effect presented by the second image frame is better, and the stability and the accuracy of the noise reduction processing on the image are improved.

In a third aspect, an embodiment of the present application provides a processing chip, including the image noise reduction apparatus in the first aspect of the embodiment of the present application.

Therefore, the image noise reduction device in the processing chip outputs the motion adjustment information by performing time-space domain smoothing filtering on the image affine transformation information, so that more accurate and reasonable motion adjustment information is obtained, the accuracy of the rotational alignment of the reference frame is improved, the stability and the accuracy of noise reduction filtering processing on the current frame and/or the rotationally aligned reference frame are improved, the time-domain noise reduction effect presented by the second image frame is better, and the stability and the accuracy of the image noise reduction processing are improved.

In a fourth aspect, an embodiment of the present application provides an electronic device, which includes a memory, a communication interface, and the image noise reduction apparatus in the first aspect of the embodiment of the present application.

It can be seen that, the image noise reduction device in the electronic device outputs the motion adjustment information by performing the time-space domain smoothing filtering processing on the image affine transformation information, so as to obtain more accurate and reasonable motion adjustment information, further improve the accuracy of the rotational alignment of the reference frame, improve the stability and accuracy of the noise reduction filtering processing on the current frame and/or the rotationally aligned reference frame, make the time-domain noise reduction effect presented by the second image frame better, and improve the stability and accuracy of the image noise reduction processing.

Drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below. It is to be expressly understood that the drawings described below are only illustrative of some embodiments of the invention. It is also possible for a person skilled in the art to derive other figures from these figures without inventive effort.

Fig. 1 is a schematic structural diagram of an image noise reduction apparatus provided in an embodiment of the present application;

FIG. 2 is a schematic flow chart of image denoising provided in an embodiment of the present application;

fig. 3 is a schematic structural diagram of feature point detection performed on an image block of a current frame according to an embodiment of the present application;

fig. 4 is a schematic structural diagram of performing search matching on a current frame and a reference frame according to an embodiment of the present disclosure;

fig. 5 is a schematic structural diagram of relative displacement from a feature point data window of a current frame to a target feature point data window according to an embodiment of the present application;

FIG. 6 is a schematic structural diagram of another image noise reduction apparatus provided in an embodiment of the present application;

FIG. 7 is a schematic structural diagram of exposure time of image blocks in line blocks of another image frame according to an embodiment of the present application;

fig. 8 is a schematic structural diagram of an image block in a row block of a current frame according to an embodiment of the present disclosure;

fig. 9 is a schematic structural diagram of an image block in a row block of a current frame and an image block in a row block of a reference frame according to an embodiment of the present application;

FIG. 10 is a schematic flow chart illustrating still another image denoising method provided in an embodiment of the present application;

fig. 11 is a schematic flowchart of an image denoising method according to an embodiment of the present application;

FIG. 12 is a schematic structural diagram of a processing chip according to an embodiment of the present disclosure;

fig. 13 is a schematic structural diagram of an electronic device according to an embodiment of the present application.

Detailed Description

In order to better understand the technical solutions of the present application, the following description is given for clarity and completeness in conjunction with the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only some embodiments of the present application, but not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the description of the embodiments of the present application without inventive step, are within the scope of the present application.

The terms "first," "second," and the like in the description and claims of the present application and in the foregoing drawings are used for distinguishing between different objects and not for describing a particular sequential order. Furthermore, the terms "include" and "have," as well as any variations thereof, are intended to cover a non-exclusive inclusion. For example, a process, method, software, product, or apparatus that comprises a list of steps or elements is not limited to those listed but may include other steps or elements not listed or inherent to such process, method, product, or apparatus.

In the embodiments of the present application, the term "connected" should be interpreted broadly unless explicitly stated or limited otherwise. For example, "connected" may be fixed, coupled, or detachable; may be directly connected, indirectly connected through an intermediary, or in spaced contact; but may be physically or electrically connected, etc.

Reference in the embodiments of the present application to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the present application. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. It is explicitly and implicitly understood by one skilled in the art that the embodiments described herein may be combined with other embodiments.

Before describing the technical solutions of the embodiments of the present application, the following description will be made on related concepts that may be related to the present application.

(1) Video noise reduction

Since an image capturing apparatus, such as a complementary metal-oxide-semiconductor (CMOS) sensor, a charge-coupled device (CCD) sensor, etc., will be affected by noise during data acquisition, so that random noise exists in a video stream, it is necessary to remove the noise by using an image noise reduction technology.

The video denoising method can be divided into a space domain, a frequency domain, a wavelet domain, a time-space domain and the like according to different processing domains. Wherein the noise reduction methods between different processing domains overlap, or one noise reduction method involves multiple processing domains. For example, a noise reduction method using frequency domain filtering may be used in a noise reduction method using time domain or time-space domain, that is, after all or a part of image frames in a video stream are transformed into a frequency domain by fourier transform, noise reduction processing is performed using time domain filtering or time-space domain filtering.

The modes of spatial filtering, frequency domain filtering, wavelet domain filtering and the like in video denoising are consistent with the modes in image denoising, and only the modes are expanded to process multi-frame images, and meanwhile, more redundant information in video signals is utilized for optimization, so that a better denoising effect is achieved. The spatial filtering is to directly perform algebraic operation of pixel values on each frame of image in a video stream, and only consider the correlation of the image frame on a spatial domain. The frequency domain filtering is to transform the image frames in the video stream into the frequency domain through fourier transform, and then attenuate the frequencies representing noise to retain the original information in the video stream to the maximum extent. The wavelet domain filtering is to convert the image frames in the video stream into a time-frequency domain and then perform denoising. The time domain filtering is to perform noise reduction filtering by considering the correlation of image frames in a video stream in a time dimension, and has the advantages of simple operation, high efficiency and no introduction of a spatial blurring phenomenon.

(2) Three-dimensional Digital Noise Reduction (3-dimensional Digital Noise Reduction)

The 3DNR is a noise reduction method combining spatial filtering and temporal filtering, and the general idea is to detect motion vectors among multiple frames of images in a video stream through motion estimation, perform alignment processing according to the motion vectors, and finally perform fusion noise reduction filtering on the multiple frames of images after the alignment processing, thereby outputting filtered images.

Currently, in the video noise reduction technology, it is generally required to perform affine image transformation (affine transformation) on image blocks of image frames in a video stream to obtain affine image transformation information, so as to perform operations such as alignment processing on the image frames by using the affine image transformation information. However, in the actual video denoising process, the image frame is aligned directly by using the affine transformation information of the image, which results in poor alignment effect of the image frame, and thus affects the stability and accuracy of the image denoising process.

In conjunction with the above description, the image noise reduction apparatus according to the embodiment of the present application will be described with reference to fig. 1. Fig. 1 is a schematic structural diagram of an image noise reduction device according to an embodiment of the present disclosure. The image noise reduction apparatus 10 includes an image processing module 110, a filter processing module 120, and a noise reduction processing module 130. The image processing module 110 is respectively connected to the filtering processing module 120 and the denoising processing module 130, and the filtering processing module 120 is connected to the denoising processing module 130. Wherein the content of the first and second substances,

the image processing module 110 is configured to perform matching (also referred to as search matching) on a first feature point data window in an image block of a reference frame in an image block of a current frame to output affine transformation information of an image, and perform feature point detection on the image block of the current frame in parallel to obtain a second feature point data window, where the reference frame is an image frame located before the current frame in time sequence.

And a filtering processing module 120, configured to perform time-space domain smoothing filtering on the affine transformation information of the image to output motion adjustment information.

A noise reduction processing module 130, configured to perform rotation alignment processing on the reference frame by using the motion adjustment information to obtain a first image frame, and perform noise reduction filtering processing on the current frame and/or the first image frame to output a second image frame.

It should be noted that, in order to improve the stability and accuracy of the image noise reduction process, the image noise reduction apparatus 10 according to the embodiment of the present application includes hardware and/or software modules for performing the respective functions. Those skilled in the art will readily appreciate that the functions of the various modules described in connection with the embodiments provided herein may be implemented in hardware or a combination of hardware and computer software. Whether a module is executed as hardware or computer software driven hardware depends upon the particular application and design constraints imposed on the solution. Those skilled in the art may call a specific program or algorithm through a processor (such as a CPU, etc.) to implement the functions of the modules described in the embodiments of the present application, but such implementation should not be considered as beyond the scope of the present application. In addition, in the embodiment of the present application, the image noise reduction apparatus 10 may be divided into functional modules, for example, each functional module may be divided according to the function, or two or more functions may be integrated into one functional module. The integrated functional module can be realized in a hardware form or a software form. Meanwhile, the division of the functional modules in the embodiment of the present application is schematic, and is only a logic function division, and another division manner may be provided in actual implementation, which is not specifically limited.

In the image denoising method shown in fig. 2, a reference frame and a current frame to be denoised are obtained from a video stream, and the current frame is uploaded to a bus for storage, wherein the reference frame is a previous frame or a next frame of the current frame; carrying out smooth filtering on the current frame, and carrying out feature point detection on the current frame after smooth filtering to obtain a feature point data window; searching and matching the characteristic point data window of the current frame to a reference frame to obtain a motion vector, namely a motion matching result, and calculating motion adjustment information through the motion matching result, wherein the motion adjustment information is used for performing rotary alignment processing on the reference frame; reading a current frame and a reference frame from a bus, and performing rotation alignment processing on the reference frame through the motion adjustment information; and performing 2-frame fusion noise reduction processing on the current frame and the reference frame after the rotation alignment to obtain a second image frame, thereby realizing time domain noise reduction on the current frame.

For the feature point detection of the current frame, image blocking processing may be performed on the current frame to obtain N × N image blocks with equal size and non-overlapping, where the value of N may be 2, 4, 8, or 16, and feature point detection may be performed on each image block to obtain a plurality of feature point data windows. For example, referring to fig. 3, a current frame is divided into 4 × 4 image blocks, and then feature point detection is performed on each image block to obtain a plurality of feature point data windows.

For searching and matching the feature point data window of the current frame to the reference frame to obtain the optimal motion vector, a data window similar to the feature point data window of the current frame may be searched and matched in the reference frame to obtain a target feature point data window in the reference frame, as shown in fig. 4, an area 410 in fig. 4 represents a search and matching range, and then the relative displacement from the feature point data window of the current frame to the target feature point data window is calculated to obtain the motion vector, as shown in fig. 5.

In the feature point detection process shown in fig. 3, feature point detection is usually performed on 4 image blocks in the first row block of the current frame (i.e., 1/4 frame of the current frame) after being partitioned, and then feature point detection is performed on 4 image blocks in the second row block of the current frame, and so on. Therefore, in the feature point detection process, a processing delay of 1/N frame, that is, a processing delay of 1/4 frame in fig. 3, is usually introduced. Meanwhile, in the search matching process shown in fig. 4, the feature point data window of 1/4 frame of the current frame (i.e. the first row block of the current frame) is usually searched and matched in 1/4 frame of the reference frame (i.e. the first row block of the reference frame), and so on. Therefore, 1/N frame processing delay is also introduced in the process of searching for matching.

Because the current frame and the reference frame after the rotation alignment still need to be completely read from the bus in the fusion noise reduction processing of the current frame and the reference frame after the rotation alignment, a processing delay of 1 frame is also introduced in the fusion noise reduction processing. It can be seen that there is usually a large processing delay in the actual video denoising processing step.

In addition, in the process of performing rotation alignment processing on the reference frame, since usually the affine transformation information of the image is directly used as motion adjustment information, that is, the affine transformation information of the image is directly used to perform rotation alignment processing on the reference frame, the rotation alignment effect of the reference frame is poor, and the stability and accuracy of the fusion noise reduction processing performed on the current frame and the reference frame after rotation alignment are further affected, so that the time domain noise reduction effect presented by the second image frame is poor.

In combination with the above description, in order to improve the accuracy of image frame rotation alignment and improve the stability and accuracy of image denoising processing, the following specifically describes the technical solution of the embodiment of the present application.

Specifically, the current frame may be used to represent a frame of image to be denoised currently in the video stream.

In particular, the reference frame may temporally precede the current frame. For example, the reference frame may be a previous frame image of the current frame.

Specifically, matching or search matching may be understood as determining a search range in an image block of a current frame through a search matching algorithm, finding a most approximate (or matched) data window in the search range through a first feature point data window, and finally obtaining motion adjustment information through a Motion Vector (MV) from the first feature point data window to the most approximate data window. The search matching algorithm may include a full search matching algorithm, a three-step search algorithm, a diamond search algorithm, a four-step search algorithm, a continuous filtering algorithm, a multi-layer filtering algorithm, a partial distortion removal algorithm, and the like.

Specifically, the feature point detection may include hash corner detection (Harris corner detection). Wherein a corner in an image frame may have the following characteristics: an intersection between the contours; for the same scene, the scene still has stable property even if the visual angle changes; the pixel points in the area near the corner point have large changes in the gradient direction or the gradient amplitude.

It should be noted that, the hash corner detection may be performed by moving on the image block of the current frame through a local window, and determining whether a pixel value in the local window changes greatly. Whether the pixel value has a large change can be described by an image gradient. The larger the image gradient, the larger the change in the pixel value (or the larger the change rate of the gray scale value) in the local window. Therefore, if there is a large variation in the gray values (over the image gradient) within the local window, there are corner points in the region where the local window is located.

It can be understood that, if the hash corner detection is adopted to perform the feature point detection on the image block of the current frame to obtain the feature point data window, the feature point data window may be used to represent a corner in the image block, or the feature point data window may be used to represent an image area and the like composed of the corner in the image block and pixel points around the corner.

Specifically, the image blocks of the reference frame may be used to represent at least one image block with equal size and non-overlapping in the reference frame.

It can be understood that, in the embodiment of the present application, image blocking processing may be performed on an obtained reference frame to obtain N × N image blocks that are equal in size and are not overlapped with each other, where a value of N may be 2, 4, 8, or 16, and feature point detection may be performed on each image block to obtain a plurality of feature point data windows, and finally, the plurality of feature point data windows are stored, so that it is beneficial to directly obtain the feature point data windows of the reference frame in subsequent search matching without performing feature point detection, and further, processing delay in a video denoising process is reduced.

Specifically, the image block of the current frame may be used to represent at least one image block in the current frame, which has an equal size and does not overlap with each other.

It can be understood that, in the embodiment of the present application, image blocking processing may be performed on an acquired current frame to obtain N × N image blocks that are equal in size and do not overlap with each other, where a value of N may be 2, 4, 8, or 16.

In combination with the above description, compared with the distributed processing in which the feature point detection is performed on the current frame first and then the search matching is performed on the feature point data window of the current frame to the reference frame in fig. 2, in the embodiment of the present application, the image processing module 110 directly obtains the first feature point data window of the reference frame (i.e., the feature point detection is not required to be performed) and the current frame, and performs the search matching of the first feature point data window in the image block of the current frame and the feature point detection of the image block of the current frame in parallel, so that the parallel processing of the search matching and the feature point detection is realized by directly obtaining the first feature point data window and the current frame, and further, the processing delay in the video denoising process is reduced by the parallel processing, and the processing efficiency is improved. Meanwhile, the second characteristic point data window of the current frame is directly uploaded to avoid uploading and storing the rest redundant data in the current frame so as to reduce the bandwidth of data uploading.

In addition, compared with the case that the image affine transformation information is directly used as the motion adjustment information in fig. 2, that is, the reference frame is directly subjected to the rotation alignment processing by using the image affine transformation information, in the embodiment of the present application, the filter processing module 120 performs the time-space domain smoothing filter processing on the image affine transformation information to output the motion adjustment information, so as to obtain more accurate and reasonable motion adjustment information, further improve the accuracy of the rotation alignment of the reference frame, and improve the stability and accuracy of the noise reduction filter processing performed on the current frame and/or the reference frame after the rotation alignment, so that the time domain noise reduction effect presented by the second image frame is better, and the stability and accuracy of the noise reduction processing of the image are improved.

In conjunction with the above description, the following embodiments of the present application will specifically describe each module in the image noise reduction apparatus 10 with reference to fig. 6.

Specifically, the filtering processing module 120 may specifically include: a time-space domain smoothing filter module 1201 and a motion adjustment module 1202; wherein the content of the first and second substances,

the time-space domain smoothing filtering module 1201 may be configured to perform, based on an exposure characteristic of the rolling shutter, spatial domain smoothing filtering and/or time domain smoothing filtering on image affine transformation information corresponding to the image block of the current frame to output smoothing filtering information.

A motion adjustment module 1202 may be configured to determine motion adjustment information based on the smoothing filter information.

The determining of the motion adjustment information based on the smooth filtering information may include using the smooth filtering information as the motion adjustment information, or performing linear weighting on the smooth filtering information to obtain the motion adjustment information.

The spatial smoothing filtering may be understood as performing smoothing filtering between affine transformation information of images corresponding to previous and/or next image blocks in the same image frame (current frame), or performing smoothing filtering between affine transformation information of images corresponding to previous and/or next image blocks in the same line of blocks of the same image frame.

The time-domain smoothing filtering may be performed by performing smoothing filtering between affine transformation information of images corresponding to image blocks in different image frames (between a current frame and a reference frame), or performing smoothing filtering between affine transformation information of images corresponding to image blocks in different line blocks of the same image frame.

It should be noted that, compared to the one-time exposure of the global shutter (global shutter), the exposure characteristic of the rolling shutter (rolling shutter) satisfies one of the following conditions: serial exposure, top to bottom exposure, left to right exposure. It can be seen that the exposure characteristics of a rolling shutter can form image blocks at different exposure times. Based on this, the embodiment of the present application, based on the exposure characteristic of the rolling shutter, each row block of the current frame (reference frame) (as shown in fig. 3, after the current frame (reference frame) is subjected to the image blocking process, the current frame includes 4 row blocks from top to bottom, each row block includes 4 image blocks from left to right) is represented as a row block at a different exposure time, and each image block from left to right in the same row block is represented as an image block at a different exposure time.

For example, as shown in fig. 7, the image frame includes 4 line blocks from top to bottom, each line block including 4 image blocks from left to right. Based on the exposure characteristics of the rolling shutter, the 4 row blocks are sequentially exposed from top to bottom, and the 4 image blocks in each row block are sequentially exposed from left to right.

In addition, in the embodiment of the application, searching and matching can be performed in each image block of the current frame (or each row block of the current frame) through the directly obtained feature point data window of the reference frame, so that image affine transformation information corresponding to each image block of the current frame is obtained. As can be seen from the exposure characteristic of the rolling shutter (rollingshutter), since each image block of the current frame has a different exposure time, the affine transformation information of the images corresponding to the different image blocks is obtained at the different exposure times.

In summary, the embodiment of the present application performs spatial domain smoothing filtering and/or temporal domain smoothing filtering on the image affine transformation information corresponding to the image block of the current frame based on the exposure characteristic of the rolling shutter, so as to obtain more accurate and reasonable motion adjustment information, thereby facilitating improvement of accuracy of image frame rotation alignment.

With reference to the above description, three cases of performing spatial smoothing filtering and/or temporal smoothing filtering on image affine transformation information corresponding to an image block of a current frame based on exposure characteristics of a rolling shutter will be described below.

The first situation is as follows:

specifically, the time-space domain smoothing filtering module 1201 may be configured to: and performing smoothing filtering on first image affine transformation information corresponding to a first image block in a first row of blocks of the current frame and second image affine transformation information corresponding to image blocks located at the previous position and/or the next position of the first image block in the first row of blocks to output first smoothing filtering information in the smoothing filtering information.

Wherein, the image block of the current frame is positioned in the different row blocks of the current frame.

Wherein the second image affine transformation information may include at least one of: the image affine transformation information comprises image affine transformation information corresponding to an image block located at the front position of the first image block in the first row block, image affine transformation information corresponding to an image block located at the rear position of the first image block in the first row block, and image affine transformation information corresponding to image blocks located at the front position and the rear position of the first image block in the first row block.

The first row block may be one of the row blocks of the current frame, the first image block may be at least one of the first row blocks, and the image affine transformation information may include first image affine transformation information and second image affine transformation information.

The motion adjustment module 1202 may be configured to use the first smoothing filter information in the smoothing filter information as the motion adjustment information.

It should be noted that, in case one, the time-space domain smoothing filtering module 1201 may perform smoothing filtering on the image affine transformation information corresponding to at least one image block in a certain row of blocks of the current frame and the image affine transformation information corresponding to the image block located at the previous position and/or the next position of the at least one image block in the row of blocks, and so on, thereby implementing performing spatial domain smoothing filtering on the image affine transformation information corresponding to the image block of the current frame to obtain first smoothing filtering information, and then using the first smoothing filtering information as motion adjustment information through the motion adjustment module 1202, which is beneficial to ensuring that the obtained motion adjustment information is more accurate and reasonable.

For example, as shown in fig. 8, if the first image block is the second image block (i.e., the image block 801) in the first row block of the current frame, the image block 802 is the image block located at the position before the image block 801, and the image block 803 (or the image block 804) is the image block located at the position after the image block 801, the spatio-temporal smoothing filtering module 1201 performs smoothing filtering on the image affine transformation information corresponding to the image block 801, the image affine transformation information corresponding to the image block 802, and the image affine transformation information corresponding to the image block 803 (or the image block 804), in a manner of performing spatial smoothing filtering on the image affine transformation information.

For another example, as shown in fig. 8, if the first image block is the first image block (i.e., the image block 802) in the first row block of the current frame and the image block 801 (or the image block 803 or the image block 804) is the image block located at the position subsequent to the image block 801, the spatio-temporal smoothing filter module 1201 performs smoothing filtering on the image affine transformation information corresponding to the image block 802 and the image affine transformation information corresponding to the image block 801 (or the image block 803 or the image block 804) so that the image affine transformation information is subjected to spatial smoothing filtering.

For another example, as shown in fig. 8, if the first image block is the last image block in the first row block of the current frame (that is, the image block 804), and the image block 803 (or the image block 802 or the image block 801) is an image block located at a position before the image block 804, the spatio-temporal smoothing filtering module 1201 performs smoothing filtering on the image affine transformation information corresponding to the image block 804 and the image affine transformation information corresponding to the image block 803 (or the image block 802 or the image block 801) so as to perform spatial smoothing filtering on the image affine transformation information.

The second situation:

specifically, the time-space domain smoothing filtering module 1201 may be configured to: and performing smoothing filtering on first image affine transformation information corresponding to a first image block in a first row of blocks of the current frame and third image affine transformation information corresponding to a second image block in a previous row of blocks of the first row to output second smoothing filtering information in the smoothing filtering information.

The first row block may be one row block in the row blocks of the current frame, the first image block may be at least one image block in the first row block, the image affine transformation information may include first image affine transformation information and second image affine transformation information, the second image block may be at least one image block in the image blocks of the current frame or at least one image block in the image blocks of the reference frame, and the image affine transformation information may further include third image affine transformation information.

The motion adjustment module 1202 is configured to use the second smoothing information in the smoothing filtering information as the motion adjustment information.

It should be noted that, in case two, the time-space domain smoothing filtering module 1201 may perform smoothing filtering on the image affine transformation information corresponding to at least one image block in a certain row of blocks of the current frame and the image affine transformation information corresponding to at least one image block in a previous row of blocks, and so on, so as to perform time-domain smoothing filtering on the image affine transformation information corresponding to the image block of the current frame to obtain second smoothing filtering information, and then use the second smoothing filtering information as motion adjustment information through the motion adjustment module 1202, which is favorable for ensuring that the obtained motion adjustment information is more accurate and reasonable.

Further, if the first row block is the first row block in the row blocks of the current frame, the previous row block of the first row block is the last row block in the row blocks of the reference frame.

Wherein the image blocks of the reference frame are located in line blocks of different reference frames.

It should be noted that the previous line block of the first line block may be a line block of the current frame, or may be a line block of the reference frame. When the first line block is the first line block in the line blocks of the current frame, the reference frame is the previous image frame of the current frame, and the last line block in the line blocks of the reference frame is closest to the first line block in the line blocks of the current frame at the exposure time, so that the smooth filtering information output by performing smooth filtering on the image affine transformation information corresponding to the image block in the first line block of the current frame and the image affine transformation information corresponding to the image block in the last line block of the reference frame is more accurate, and more accurate and reasonable motion adjustment information is further ensured to be obtained.

For example, as shown in fig. 9, if the first image block is a first image block (e.g., image block 901) in a first row block of the current frame and the image block 902 is a first image block in a last row block of the reference frame, the spatio-temporal smoothing filtering module 1201 performs smoothing filtering on the image affine transformation information corresponding to the image block 901 and the image affine transformation information corresponding to the image block 902 (or another image block in the last row block of the reference frame) in a manner of performing temporal smoothing filtering on the image affine transformation information.

Case three:

specifically, the time-space domain smoothing and filtering module 1201 may be configured to: performing smoothing filtering based on first image affine transformation information corresponding to a first image block in a first row of blocks of a current frame and second image affine transformation information corresponding to image blocks located at a previous position and/or a subsequent position of the first image block in the first row of blocks to output first smoothing filtering information in the smoothing filtering information; and performing smoothing filtering on the first image affine transformation information corresponding to the first image block and third image affine transformation information corresponding to the second image block in a row before the first row of blocks to output second smoothing filtering information in the smoothing filtering information.

The motion adjustment module 1202 may be configured to perform linear weighting on the first smooth filtered information and the second smooth filtered information to obtain motion adjustment information.

It should be noted that, in case three, the time-space domain smoothing filtering module 1201 performs space-domain smoothing filtering and time-domain smoothing filtering on the image affine transformation information corresponding to the image block of the current frame to obtain first smoothing filtering information and second smoothing filtering information, and then performs linear weighting processing on the first smoothing filtering information and the second smoothing filtering information through the motion adjusting module 1202 to obtain motion adjusting information, thereby further ensuring that the obtained motion adjusting information is more accurate and reasonable.

In conjunction with the above description, the following describes a specific description of the memory module included in the image noise reduction apparatus 10.

In one possible example, the image noise reduction apparatus 10 may further include a storage module. The storage module may be configured to store at least one of the current frame, the reference frame, the first feature point data window, and the second feature point data window.

In particular, the memory module may include a double rate DDR memory.

The DDR memory may include a double data rate synchronous dynamic random access memory (DDR SDRAM).

It should be noted that the image processing module 110 in this embodiment of the present application may directly obtain the first feature point data window and the current frame in the image block of the reference frame from the storage module, and then upload the current second feature point data window to the storage module for storage, so as to subsequently and directly read the second feature point data window from the storage module without performing feature point detection again to perform search matching in parallel processing, thereby achieving reduction of processing delay in the video denoising process. Meanwhile, the parallel processing of searching matching and feature point detection is realized by directly obtaining the first feature point data window and the current frame, so that the processing time delay in the video noise reduction process is reduced through the parallel processing, and the processing efficiency is improved. In addition, the second characteristic point data window of the current frame is directly uploaded, so that the residual redundant data in the current frame is prevented from being uploaded and stored, and the bandwidth of data uploading is reduced.

In conjunction with the above description, the image processing module 110 is specifically described below.

In one possible example, the image processing module 110 may include a search matching module 1101, an affine transformation module 1102, and a feature point detecting module 1103; wherein the content of the first and second substances,

a search matching module 1101 may be configured to match a first window of feature point data within an image block of the current frame to output a motion vector.

And an affine transformation module 1102, configured to perform data fitting processing on the motion vectors to output image affine transformation information.

The feature point detecting module 1103 may be configured to perform feature point detection on the image block of the current frame to obtain a second feature point data window.

It should be noted that, first, in the embodiment of the present application, the image processing module 110 may execute, in parallel, the search matching of the feature point data window of the reference frame in the image block of the current frame and the feature point detection of the current frame, so as to implement the processing delay reduction in the video denoising process through parallel processing of the search matching and the feature point detection.

Secondly, the feature point detection in the embodiment of the present application includes hash corner detection, so that the feature point detection module 1103 can detect corners in an image frame of a current frame.

Thirdly, searching matching is one of the important components of the video noise reduction technology, and a Motion Vector (MV) of a pixel point is obtained by calculating the relative motion offset of the pixel point in two adjacent frames of images. The search matching module 1101 may determine a search range in the image block of the current frame through a search matching algorithm, and then find a most approximate (or matching) data window within the search range. The search matching algorithm may include a full search matching algorithm, a three-step search algorithm, a diamond search algorithm, a four-step search algorithm, a continuous culling algorithm, a multi-layer culling algorithm, a partial distortion removal algorithm, and the like. Therefore, the search matching module 1101 can perform search matching on the feature point data window of the reference frame in the image block of the current frame to obtain the operation vector.

Finally, the data fitting process of the motion vector in the embodiment of the present application may include a least square method, and the data fitting process is performed on the motion vector by the least square method to obtain image affine transformation (imaging) information. The affine transformation of the image may include a homographic matrix (homographic matrix). The homography transformation can be used to describe the position mapping relationship of the object between the world coordinate system and the pixel coordinate system, and the corresponding homography transformation is called a homography matrix. Therefore, the affine transformation module 1102 performs data fitting processing on the motion vectors to obtain motion adjustment information for performing rotation alignment processing on the reference frame.

Specifically, the search matching module 1101 may be specifically configured to: calculating the sum of absolute errors between the gray value of each pixel point in the first characteristic point data window and the gray value of each pixel point in the data window in the searching range of the image block of the current frame to obtain a parameter value; taking a data window corresponding to the minimum value in the parameter values as a target data window; and calculating a relative displacement of the first feature point data window to the target data window to output a motion vector.

It should be noted that, in the embodiment of the present application, it is considered that a minimum Sum of Absolute Differences (SAD) criterion is used to search and match in the image block of the current frame in the first feature point data window of the reference frame, and a data window corresponding to the minimum value of the SAD values (i.e., parameter values) of the first feature point data window is found. At this time, the data window is the data window corresponding to the first feature point data window that is the best match, i.e., the target data window. Therefore, the first feature point data window and the target data window form a relative position in a time domain to obtain a motion vector, so that motion adjustment information for performing a rotation alignment process on the reference frame is constructed by the motion vector.

In conjunction with the above description, the modules included in the image processing module 110 will be specifically described below.

In particular, the image processing module 110 may further include a smoothing filter module 1104. Wherein, the first and the second end of the pipe are connected with each other,

the smoothing filtering module 1104 may be configured to perform smoothing filtering processing on the current frame and output the current frame after the smoothing filtering processing.

It is noted that the smoothing filter module 1104 can be used to reduce noise and artifacts in the current frame. The smoothing filter module 1104 may include an average filter, a square frame filter, a median filter, a gaussian filter, a bilateral filter, etc.

Specifically, the image processing module 110 may further include an image blocking module 1105. Wherein, the first and the second end of the pipe are connected with each other,

the image blocking module 1105 may be configured to perform image blocking processing on the current frame, and output the current frame after the image blocking processing.

It should be noted that, in the embodiment of the present application, an image frame (including a current frame or a reference frame) of an acquired video stream may be divided into N × N image blocks with the same size and without overlapping with each other by the image partitioning module 1105, so that feature point detection is directly performed on the image blocks in the following process, and the feature point detection efficiency of the image frame is improved.

Specifically, the image processing module 110 may further include a data window uploading module 1106. Wherein the content of the first and second substances,

a data window upload module 1106 may be configured to upload a second feature point data window for storage.

Specifically, the data window uploading module 1106 may be configured to upload the second feature point data window to the storage module.

It should be noted that, in the embodiment of the present application, the data window uploading module 1106 uploads the current second feature point data window to the storage module for storage, so that the second feature point data window is directly read from the storage module in the following without performing feature point detection to perform search matching in parallel processing, thereby reducing the processing delay in the video denoising process. In addition, the second characteristic point data window of the current frame is directly uploaded, so that the residual redundant data in the current frame is prevented from being uploaded and stored, and the bandwidth of data uploading is reduced.

In conjunction with the above description, the noise reduction processing module 130 is specifically described below.

In one possible example, the denoising processing module 130 may specifically include a rotation alignment module 1301, a filtering decision module 1302, and a fusion filtering module 1303. Wherein the content of the first and second substances,

a rotation alignment module 1301 may be configured to perform a rotation alignment process on the reference frame based on the motion adjustment information to output the first image frame.

The filtering decision module 1302 may be configured to determine noise reduction filtering policy information for the current frame and the first image frame based on a gray value of each pixel in an image block of the current frame, a gray value of each pixel in an image block of the first image frame, and a preset threshold, where the noise reduction filtering policy information includes time domain filtering or spatial domain filtering.

The fusion filtering module 1303 may be configured to perform denoising filtering processing on the current frame and/or the first image frame based on the denoising filtering policy information to output a second image frame.

Determining noise reduction filtering strategy information for the current frame and the first image frame based on the gray value of each pixel point in the image block of the current frame, the gray value of each pixel point in the image block of the first image frame and a preset threshold, which may include: performing average absolute error calculation based on the gray value of each pixel point in the image block of the current frame and the gray value of each pixel point in the image block of the first image frame to obtain a parameter value; and determining noise reduction filtering strategy information aiming at the current frame and the first image frame according to the comparison result between the parameter value and a preset threshold value.

It should be noted that, first, in order to reduce a residual value between each pixel in the image block of the current frame and each pixel in the image block of the reference frame, the embodiment of the present application considers that the rotation alignment module 1301 performs a rotation alignment process on the reference frame.

Secondly, in order to overcome the tailing phenomenon of a fast moving object easily caused by time domain filtering, the embodiment of the application considers that the motion intensity is detected by a mean absolute error (MAD) algorithm, so that for objects with different motion intensities, different filtering intensities are adopted by a filter, the tailing phenomenon caused by the fast moving object is avoided, and the image filtering effect is improved. The filtering decision module 1302 calculates a gray value of each pixel in the third image block of the current frame and a gray value of each pixel in the fourth image block of the first image frame through an MAD algorithm to obtain an MAD value (i.e., a parameter value).

Thirdly, the embodiment of the application tests a large number of image sequences to determine an empirical value for measuring the motion intensity, namely a preset threshold, and then determines the motion intensity of the image block of the current frame on the motion track represented by the motion vector through a comparison result between the parameter value and the preset threshold.

If the comparison result between the parameter value and the preset threshold value is that the parameter value is larger than the preset threshold value, it indicates that the image block has stronger motion intensity on the motion track. At this time, the filtering strength of the filter may be reduced (or it is determined that the denoising filtering strategy information is spatial filtering), so that the original information of the filtered image block is kept as much as possible, and the tailing phenomenon is avoided.

If the comparison result between the parameter value and the preset threshold value is that the parameter value is smaller than the preset threshold value, it indicates that the image block has weaker motion intensity on the motion track, i.e. the image block moves stably and slowly. For example, background pictures in a video stream typically do not move. At this time, the filtering strength of the filter may be increased (or the noise reduction filtering policy information is determined to be time-domain filtering), so as to effectively remove the noise. Meanwhile, the low motion intensity of the image block cannot cause trailing, and better filtering effect is guaranteed.

Finally, in order to implement denoising filtering processing on the current frame, the fusion filtering module 1303 according to this embodiment performs fusion denoising filtering processing on the current frame and/or the first image frame according to time-domain filtering or spatial-domain filtering.

Specifically, the spatial filtering may include mean filtering, median filtering, gaussian filtering, or bilateral filtering. Therefore, the current frame is subjected to noise reduction filtering according to spatial filtering to output a second image frame, so that an image block with stronger motion intensity on a motion track is filtered through the spatial filtering, original information in the image block is reserved as much as possible, and the trailing phenomenon is avoided.

In particular, the temporal filtering may comprise linear weighted filtering. Therefore, the second image frame output by performing the fusion noise reduction filtering on the current frame and the first image frame according to the linear weighted filtering satisfies the following formula:

F _out ＝ω*F _t +(1-ω)*F _t-1 ；

wherein, F _out Each pixel value, F, in an image block representing a second image frame _t Representing the values of the pixels in the image block of the current frame, F _t-1 Each pixel value in an image block representing the first image frame, and ω represents a preset weight. In addition, the filtering strength of the filter can be dynamically adjusted by presetting the weight ω.

In summary, an image block with a weak motion intensity on a motion trajectory is filtered through linear weighted filtering, so that noise in the image block is effectively removed.

Specifically, the image blocks of the first image frame may be used to represent at least one image block of the first image frame, which has an equal size and does not overlap with each other. It is understood that, the filtering decision module 1302 of the embodiment of the present application may perform image blocking processing on the input first image frame to obtain N × N image blocks with equal size and without overlapping, where N may have a value of 2, 4, 8, or 16.

In summary, the following describes an exemplary flow of image denoising according to an embodiment of the present application, as shown in fig. 10. In fig. 10, a feature point data window of a reference frame and a current frame to be denoised are directly obtained from a DDR, the reference frame is a previous frame of the current frame, and the current frame is subjected to smooth filtering; performing parallel detection of the feature points of the current frame after smooth filtering and searching and matching of a feature point data window of a reference frame in an image block of the current frame after smooth filtering; obtaining an optimal motion vector, namely a motion matching result, through searching and matching from a feature point data window of a reference frame to a current frame, and performing data fitting processing on the motion matching result to obtain image affine transformation information; performing time-space domain smooth filtering on the affine transformation information of the image to obtain motion adjustment information, wherein the motion adjustment information is used for performing rotation alignment processing on a reference frame; uploading the characteristic point data window of the current frame obtained by the characteristic point detection of the current frame to a DDR for storage; reading a current frame and a reference frame from the DDR, and performing rotation alignment processing on the reference frame through the motion adjustment information; and performing fusion noise reduction filtering on the current frame and the reference frame after the rotation alignment to obtain a second image frame.

Therefore, the parallel processing of searching matching and feature point detection is realized by directly obtaining the feature point data window and the current frame of the reference frame, and the processing time delay in the video noise reduction process is reduced and the processing efficiency is improved by the parallel processing. Meanwhile, the feature point data window of the current frame is only directly uploaded, so that the uploading and storage of the rest redundant data in the current frame are avoided, and the bandwidth of data uploading is reduced. In addition, the motion adjustment information is obtained by performing time-space domain smooth filtering on the affine transformation information of the image, so that more accurate and reasonable motion adjustment information is obtained, the stability and the accuracy of noise reduction filtering processing on the current frame and/or the rotationally aligned reference frame are improved, and the time-domain noise reduction effect presented by the second image frame is better.

With reference to the above description, the image denoising method according to the embodiment of the present application will be described below with reference to fig. 11.

It should be noted that, in the image noise reduction method shown in fig. 2, a reference frame and a current frame to be noise reduced are obtained from a video stream, and the current frame is uploaded to a bus for storage, where the reference frame is a previous frame or a next frame of the current frame; carrying out smooth filtering on the current frame, and carrying out feature point detection on the current frame after smooth filtering to obtain a feature point data window; searching and matching the characteristic point data window of the current frame to a reference frame to obtain a motion vector, namely a motion matching result, and calculating motion adjustment information through the motion matching result, wherein the motion adjustment information is used for performing rotary alignment processing on the reference frame; reading a current frame and a reference frame from a bus, and performing rotation alignment processing on the reference frame through the motion adjustment information; and performing 2-frame fusion noise reduction processing on the current frame and the reference frame after the rotation alignment to obtain a second image frame, thereby realizing time domain noise reduction on the current frame.

For the feature point detection of the current frame, image blocking processing may be performed on the current frame to obtain N × N image blocks with equal size and non-overlapping, where the value of N may be 2, 4, 8, or 16, and feature point detection may be performed on each image block to obtain a plurality of feature point data windows. For example, referring to fig. 3, a current frame is divided into 4 × 4 image blocks, and then each image block is subjected to feature point detection to obtain a plurality of feature point data windows.

In addition, in the process of performing the rotation alignment processing on the reference frame, since the affine transformation information of the image is usually directly used as the motion adjustment information, that is, the affine transformation information of the image is directly used to perform the rotation alignment processing on the reference frame, the rotation alignment effect of the reference frame is poor, and the stability and accuracy of the fusion noise reduction processing performed on the current frame and the reference frame after the rotation alignment are further affected, so that the time domain noise reduction effect presented by the second image frame is poor.

In summary, in order to improve the accuracy of image frame rotation alignment and improve the stability and accuracy of image noise reduction processing, the following description is made specifically.

S1110, acquiring a first feature point data window in the image block of the reference frame and the current frame.

Wherein the reference frame temporally precedes the current frame.

Specifically, the current frame may be used to represent a frame of image in the video stream that is currently to be denoised.

Specifically, the reference frame may be a previous frame image of the current frame.

S1120, matching the first feature point data window in the image block of the current frame to obtain affine transformation information of the image, and performing feature point detection on the image block of the current frame in parallel to obtain a second feature point data window.

Specifically, the feature point detection may include hash corner detection (Harris corner detection). Wherein a corner in an image frame may have the following characteristics: intersections between the contours; for the same scene, the scene still has stable property even if the visual angle changes; the pixel points in the area near the corner point have large changes in the gradient direction or the gradient amplitude.

It should be noted that, if the hash corner detection is adopted to perform the feature point detection on the image block of the current frame to obtain the feature point data window, the feature point data window may be used to represent the corner point in the image block, or the feature point data window may be used to represent an image area composed of the corner point in the image block and pixel points around the corner point, and the like.

Specifically, the image blocks of the current frame may be used to represent at least one image block with equal size and non-overlapping in the current frame.

It can be understood that, in the embodiment of the present application, image blocking processing may be performed on an obtained current frame to obtain N × N image blocks having equal size and no overlap, where a value of N may be 2, 4, 8, or 16.

And S1130, performing time-space domain smooth filtering processing on the affine transformation information of the image to obtain motion adjustment information.

Specifically, the motion adjustment information may be used to perform a rotation alignment process on the reference frame.

And S1140, performing rotation alignment processing on the reference frame based on the motion adjustment information to obtain a first image frame.

It should be noted that, in order to reduce a residual value between each pixel in the image block of the current frame and each pixel in the image block of the reference frame, the embodiment of the present application considers performing a rotation alignment process on the reference frame through the motion adjustment information.

S1150, noise reduction filtering processing is carried out on the current frame and/or the first image frame to obtain a second image frame.

It can be seen that, in the embodiment of the present application, first, a first feature point data window (i.e., feature point detection does not need to be performed) and a current frame in an image block of a reference frame are directly obtained; secondly, matching of a first feature point data window of the reference frame in an image block of the current frame and feature point detection of the image block of the current frame are executed in parallel; thirdly, performing time-space domain smoothing filtering processing on the affine transformation information of the image to obtain motion adjustment information; and finally, performing rotation alignment on the reference frame by adopting the motion adjustment information to obtain a first image frame, and performing noise reduction filtering processing on the current frame and/or the first image frame to output a second image frame. Because the motion adjustment information is obtained by performing the time-space domain smoothing filtering processing on the image affine transformation information, more accurate and reasonable motion adjustment information is obtained, the accuracy of the reference frame rotation alignment is further improved, the stability and the accuracy of the noise reduction filtering processing of the current frame and/or the reference frame after the rotation alignment are improved, and the time domain noise reduction effect presented by the second image frame is better.

In combination with the above description, the following specifically describes the technical solutions of the embodiments of the present application.

Specifically, the step of performing time-space domain smoothing filtering on the affine image transformation information in S1130 to obtain the motion adjustment information may include: carrying out spatial domain smoothing filtering and/or time domain smoothing filtering on image affine transformation information corresponding to the image block of the current frame based on the exposure characteristic of the rolling shutter to obtain smoothing filtering information; and determining the motion adjustment information based on the smoothing filtering information, or performing linear weighting processing on the smoothing filtering information to obtain the motion adjustment information.

Wherein determining the motion adjustment information based on the smoothing filter information may include: and taking the smooth filtering information as the motion adjustment information, or carrying out linear weighting processing on the smooth filtering information to obtain the motion adjustment information.

Further, spatial smoothing filtering may be understood as performing smoothing filtering between affine transformation information of images corresponding to previous and/or next image blocks in the same image frame (current frame), or performing smoothing filtering between affine transformation information of images corresponding to previous and/or next image blocks in the same line block of the same image frame.

Further, the temporal smoothing filtering may be understood as performing smoothing filtering between affine transformation information of images corresponding to image blocks in different image frames (between a current frame and a reference frame), or performing smoothing filtering between affine transformation information of images corresponding to image blocks in different lines of blocks of the same image frame.

It should be noted that, compared to the one-time exposure of the global shutter (global shutter), the exposure characteristic of the rolling shutter (rolling shutter) satisfies one of the following conditions: serial exposure, top to bottom exposure, left to right exposure. It can be seen that the exposure characteristics of the rolling shutter can form image blocks at different exposure times. Therefore, the embodiment of the present application considers that, by using the exposure characteristic of the rolling shutter, each row block of the current frame (reference frame) (as shown in fig. 3, after the image blocking process is performed on the current frame (reference frame), the current frame includes 4 row blocks from top to bottom, each row block includes 4 image blocks from left to right) is represented as a row block at a different exposure time, and each image block from left to right in the same row block is represented as an image block at a different exposure time.

To sum up, the embodiment of the present application performs spatial domain smoothing filtering and/or temporal domain smoothing filtering on the image affine transformation information corresponding to the image block of the current frame based on the exposure characteristic of the rolling shutter, so as to obtain more accurate and reasonable motion adjustment information, thereby being beneficial to improving the accuracy of image frame rotation alignment.

The first situation is as follows:

specifically, the performing, based on the exposure characteristic of the rolling shutter, spatial smoothing filtering and/or temporal smoothing filtering on the image affine transformation information corresponding to the image block of the current frame to obtain smoothing filtering information may include: and performing smoothing filtering on first image affine transformation information corresponding to a first image block in a first row of blocks of the current frame and second image affine transformation information corresponding to image blocks positioned in front and rear positions of the first image block in the first row of blocks to obtain first smoothing filtering information in the smoothing filtering information.

The image block of the current frame is located in a different row block of the current frame.

The first row block may be one of the row blocks of the current frame, the first image block may be at least one of the first row blocks, and the image affine transformation information includes first image affine transformation information and second image affine transformation information.

It should be noted that, in case one, by performing smooth filtering on image affine transformation information corresponding to at least one image block in a certain row block of a current frame and image affine transformation information corresponding to image blocks located at a previous position and/or a subsequent position of the at least one image block in the row block, and so on, spatial domain smooth filtering is performed on the image affine transformation information corresponding to the image block of the current frame to obtain first smooth filtering information, and then the first smooth filtering information is used as motion adjustment information, which is favorable for ensuring that the obtained motion adjustment information is more accurate and reasonable.

The second situation:

specifically, the performing, based on the exposure characteristic of the rolling shutter, spatial smoothing filtering and/or temporal smoothing filtering on the affine transformation information of the image corresponding to the image block of the current frame to obtain smoothing filtering information may include: and performing smoothing filtering on first image affine transformation information corresponding to a first image block in a first row of blocks of a previous frame and third image affine transformation information corresponding to a second image block in a previous row of blocks of the first row to obtain second smoothing filtering information in the smoothing filtering information.

The first row block may be one row block in the row blocks of the current frame, the first image block may be at least one image block in the first row block, the image affine transformation information includes first image affine transformation information and second image affine transformation information, the second image block may be at least one image block in the image blocks of the current frame or at least one image block in the image blocks of the reference frame, and the image affine transformation information may further include third image affine transformation information.

It should be noted that, in case two, by performing smooth filtering on the image affine transformation information corresponding to at least one image block in a certain row block of the current frame and the image affine transformation information corresponding to at least one image block in a previous row block of the row block, and so on, the time-domain smooth filtering on the image affine transformation information corresponding to the image block of the current frame is performed to obtain second smooth filtering information, and then the second smooth filtering information is used as motion adjustment information, which is favorable for ensuring that the obtained motion adjustment information is more accurate and reasonable.

Case three:

specifically, the performing, based on the exposure characteristic of the rolling shutter, spatial smoothing filtering and/or temporal smoothing filtering on the image affine transformation information corresponding to the image block of the current frame to obtain smoothing filtering information may include: smoothing first image affine transformation information corresponding to a first image block in a first row of blocks of a current frame and second image affine transformation information corresponding to image blocks located at the previous position and/or the next position of the first image block in the first row of blocks to obtain first smoothing filtering information in the smoothing filtering information; and performing smoothing filtering on the first image affine transformation information corresponding to the first image block and the third image affine transformation information corresponding to the second image block in the previous row of the first row of blocks to obtain second smoothing filtering information in the smoothing filtering information.

It should be noted that in case three, the embodiment of the present application implements performing spatial smoothing filtering and temporal smoothing filtering on image affine transformation information corresponding to an image block of a current frame to obtain first smoothing filtering information and second smoothing filtering information, and then performing linear weighting processing on the first smoothing filtering information and the second smoothing filtering information to obtain motion adjustment information, thereby further ensuring that the obtained motion adjustment information is more accurate and reasonable.

In conjunction with the above description, how to match the first feature point data window within the image block of the current frame to obtain the image affine transformation information will be described below.

Specifically, the matching the first feature point data window in the image block of the current frame in S1120 to obtain the affine transformation information of the image may include: matching the first characteristic point data window in the image block of the current frame to obtain a motion vector; and carrying out data fitting processing on the motion vector to obtain affine transformation information of the image.

It should be noted that the data fitting process of the motion vector in the embodiment of the present application may include a least square method, and the data fitting process is performed on the motion vector by using the least square method to obtain affine transformation (affine transformation) information of the image. The affine transformation of the image may include a homographic matrix (homographic matrix). The homography transformation can be used to describe the position mapping relationship of the object between the world coordinate system and the pixel coordinate system, and the corresponding homography transformation is called a homography matrix. Therefore, the motion adjustment information for performing the rotation alignment process on the reference frame is obtained by performing the data fitting process on the motion vector.

Further, matching the first feature point data window within the image block of the current frame to obtain the motion vector may include: calculating the sum of absolute errors between the gray value of each pixel point in the first characteristic point data window and the gray value of each pixel point in the data window in the searching range of the image block of the current frame to obtain a parameter value; taking a data window corresponding to the minimum value in the parameter values as a target data window; and calculating the relative displacement from the first characteristic point data window to the target data window to obtain a motion vector.

It should be noted that, in the embodiment of the present application, it is considered that a minimum Sum of Absolute Differences (SAD) criterion is used to search and match in the image block of the current frame in the first feature point data window of the reference frame, and a data window corresponding to the minimum value of the SAD values (i.e., parameter values) of the first feature point data window is found. At this time, the data window is the data window corresponding to the first feature point data window that is the best match, i.e., the target data window. Therefore, the first feature point data window and the target data window form relative positions in the time domain to obtain a motion vector, so that motion adjustment information for performing a rotation alignment process on the reference frame is constructed by the motion vector.

In conjunction with the above description, the following describes how to perform the noise reduction filtering process on the current frame and/or the first image frame to obtain the second image frame.

Specifically, the performing noise reduction filtering processing on the current frame and/or the first image frame to obtain the second image frame in S1150 may include: determining noise reduction filtering strategy information aiming at the current frame and the first image frame based on the gray value of each pixel point in the image block of the current frame, the gray value of each pixel point in the image block of the first image frame and a preset threshold, wherein the noise reduction filtering strategy information comprises time domain noise reduction filtering or space domain noise reduction filtering; and carrying out noise reduction filtering processing on the current frame and/or the first image frame based on the noise reduction filtering strategy information to obtain a second image frame.

Determining noise reduction filtering strategy information for the current frame and the first image frame based on the gray value of each pixel point in the image block of the current frame, the gray value of each pixel point in the image block of the first image frame and a preset threshold, wherein the determining may include: calculating the average absolute error based on the gray value of each pixel point in the image block of the current frame and the gray value of each pixel point in the image block of the first image frame to obtain a parameter value; and determining noise reduction filtering strategy information aiming at the current frame and the first image frame according to the comparison result between the parameter value and a preset threshold value.

The image blocks of the first image frame may be used to represent at least one image block of the first image frame, which has equal size and does not overlap with each other. It is understood that the filtering decision module 1302 of the embodiment of the present application may perform image blocking processing on the input first image frame to obtain N × N image blocks with equal size and without overlapping, where N may be 2, 4, 8, or 16, etc.

It should be noted that, first, in order to overcome the "tailing" phenomenon of a fast moving object that is easily caused by time domain filtering, in the embodiment of the present application, motion intensity detection is performed by using a mean absolute error (MAD) algorithm, so that for objects with different motion intensities, it is ensured that a filter uses different filtering intensities, which is further beneficial to avoiding the "tailing" phenomenon caused by the fast moving object, and improving an image filtering effect. The filtering decision module 1302 calculates a gray value of each pixel in the third image block of the current frame and a gray value of each pixel in the fourth image block of the first image frame by using an MAD algorithm to obtain an MAD value (i.e., a parameter value).

If the comparison result between the parameter value and the preset threshold value indicates that the parameter value is greater than the preset threshold value, it indicates that the image block has stronger motion intensity on the motion track. At this time, the filtering strength of the filter may be reduced (or the denoising filtering strategy information is determined to be spatial filtering), so that the original information of the filtered image block is kept as much as possible, and the phenomenon of "tailing" is avoided.

If the comparison result between the parameter value and the preset threshold is that the parameter value is smaller than the preset threshold, it indicates that the image block has weaker motion intensity on the motion track, i.e. steady and slow motion. For example, background pictures in a video stream typically do not move. At this time, the filtering strength of the filter may be increased (or the noise reduction filtering policy information is determined to be time-domain filtering), so as to effectively remove the noise. Meanwhile, the low motion intensity of the image block cannot cause trailing, and better filtering effect is guaranteed.

Finally, in order to implement the noise reduction filtering processing on the current frame, the embodiment of the present application performs the noise reduction filtering processing on the current frame and/or the first image frame according to the time domain filtering or the spatial domain filtering.

Further, the spatial filtering may include mean filtering, median filtering, gaussian filtering, bilateral filtering, or the like. Therefore, the current frame is subjected to noise reduction filtering according to spatial filtering to output a second image frame, so that an image block with stronger motion intensity on a motion track is filtered through the spatial filtering, original information in the image block is reserved as much as possible, and the trailing phenomenon is avoided.

Further, the temporal filtering may include linear weighted filtering. Therefore, the second image frame output by performing the fusion noise reduction filtering on the current frame and the first image frame according to the linear weighted filtering satisfies the following formula:

F _out ＝ω*F _t +(1-ω)*F _t-1 ；

In summary, an image block with a weaker motion intensity on a motion track is filtered through linear weighted filtering, so that noise in the image block is effectively removed.

With the above description, the following explains a further technical solution of the embodiment of the present application.

Specifically, after the step of obtaining the first feature point data window and the current frame in the image block of the reference frame in S1110, the method further includes: and uploading the second characteristic point data window to a bus for storage.

The bus may include DDR memory, among other things.

It should be noted that, in the embodiment of the present application, only the feature point data window of the current frame is uploaded to the bus for storage, and the other redundant data features of the current frame are not uploaded for storage, which is not only beneficial to avoiding uploading and storing the redundant data in the current frame to reduce the bandwidth of data uploading, but also beneficial to directly reading the feature point data window of the current frame from the storage module without performing feature point detection when searching and matching are performed on the next frame image of the current frame in the following process, thereby reducing the processing delay in the video denoising process and improving the processing efficiency.

Specifically, after the step of obtaining the first feature point data window and the current frame in the image block of the reference frame in S1110, the method further includes: and performing smooth filtering processing on the current frame, and outputting the current frame after the smooth filtering processing.

Specifically, after the step of obtaining the first feature point data window and the current frame in the image block of the reference frame in S1110, the method further includes: and carrying out blocking processing on the current frame to obtain an image block of the current frame.

It should be noted that the present application focuses on the description of various embodiments. Therefore, for technical solutions not described in detail in the embodiment illustrated in fig. 11, reference may be made to the related descriptions in the embodiments illustrated in fig. 1 and fig. 6, and details thereof are not repeated.

In combination with the foregoing description, an embodiment of the present application further provides a processing chip, where the processing chip includes the image noise reduction apparatus as described in fig. 1 or fig. 6. The image noise reduction device includes: the device comprises an image processing module, a filtering processing module and a noise reduction processing module; wherein, the first and the second end of the pipe are connected with each other,

the image processing module is used for matching a first feature point data window in an image block of a reference frame in an image block of a current frame so as to output image affine transformation information, and performing feature point detection on the image block of the current frame in parallel so as to obtain a second feature point data window, wherein the reference frame is positioned in front of the current frame in time sequence;

the filtering processing module is used for performing time-space domain smooth filtering processing on the affine transformation information of the image so as to output motion adjustment information;

and the noise reduction processing module is used for performing rotation alignment processing on the reference frame by adopting the motion adjustment information to obtain a first image frame and performing noise reduction filtering processing on the current frame and/or the first image frame to output a second image frame.

It should be noted that the present application focuses on the description of various embodiments. Therefore, for specific implementation functions of each module of the image noise reduction apparatus in the processing chip, reference may be made to the relevant description in the foregoing embodiments, and details are not repeated here.

Therefore, the image noise reduction device in the processing chip directly obtains the first feature point data window and the current frame to realize parallel processing of searching matching and feature point detection, and further realizes reduction of processing time delay in the video noise reduction process through the parallel processing, and improves processing efficiency. Meanwhile, the second characteristic point data window of the current frame is directly uploaded to avoid uploading and storing the rest redundant data in the current frame so as to reduce the bandwidth of data uploading.

In addition, the image noise reduction device in the processing chip outputs the motion adjustment information by performing time-space domain smoothing filtering processing on the image affine transformation information, so that more accurate and reasonable motion adjustment information is obtained, the accuracy of the rotational alignment of the reference frame is improved, the stability and the accuracy of noise reduction filtering processing on the current frame and/or the rotationally aligned reference frame are improved, the time-domain noise reduction effect presented by the second image frame is better, and the stability and the accuracy of the image noise reduction processing are improved.

For example, please refer to fig. 12, and fig. 12 is a schematic structural diagram of a processing chip according to an embodiment of the present disclosure. The processing chip 1200 includes an image noise reduction device 1210 and a communication bus for connecting the image noise reduction device 1210. In particular, the processing chip 1200 may be a processor. The processor may include a Central Processing Unit (CPU), an Application Processor (AP), a modem processor, a Graphic Processing Unit (GPU), an Image Signal Processor (ISP), a video codec, a Digital Signal Processor (DSP), a Field Programmable Gate Array (FPGA), an application-specific integrated circuit (ASIC), a baseband processor and/or a neural Network Processor (NPU), and the like. The different processors may be independent devices, or may be integrated in the same processor.

Further, the processing chip 1200 may also include a memory for storing instructions and data. In some embodiments, the memory in the processing chip 1200 may be a cache memory. The memory may hold instructions or data that have just been used or recycled by the processor. If the processing chip 1200 needs to reuse the instruction or data, it can be called directly from the memory, thereby avoiding repeated accesses and reducing the latency of the processing chip 1200 to improve system efficiency.

Further, the processing chip 1200 may further include one or more interfaces. The interface may include an integrated circuit (I2C) interface, an integrated circuit built-in audio (I2S) interface, a Pulse Code Modulation (PCM) interface, a universal asynchronous receiver/transmitter (UART) interface, a Mobile Industry Processor Interface (MIPI), a general-purpose input/output (GPIO) interface, a Subscriber Identity Module (SIM) interface, and/or a Universal Serial Bus (USB) interface, and the like.

Specifically, the filtering processing module specifically includes: the device comprises a time-space domain smoothing filtering module and a motion adjusting module; wherein the content of the first and second substances,

the time-space domain smoothing filtering module is used for carrying out space domain smoothing filtering and/or time domain smoothing filtering on the image affine transformation information corresponding to the image block of the current frame based on the exposure characteristic of the rolling shutter to output smoothing filtering information;

and the motion adjusting module is used for determining motion adjusting information based on the smooth filtering information.

Specifically, the time-space domain smoothing filtering module is configured to: smoothing first image affine transformation information corresponding to a first image block in a first row of blocks of a current frame and second image affine transformation information corresponding to image blocks located at a previous position and/or a subsequent position of the first image block in the first row of blocks to output first smoothing filtering information in the smoothing filtering information;

the image affine transformation information comprises first image affine transformation information and second image affine transformation information.

Specifically, the time-space domain smoothing filtering module is configured to: smoothing first image affine transformation information corresponding to a first image block in a first row of blocks of a current frame and third image affine transformation information corresponding to a second image block in a previous row of blocks of the first row to output second smoothing filter information in the smoothing filter information;

wherein the image affine transformation information further comprises third image affine transformation information.

Specifically, if the first row block is the first row block in the row blocks of the current frame, the previous row block of the first row block is the last row block in the row blocks of the reference frame.

Specifically, the image processing module specifically comprises a search matching module, an affine transformation module and a feature point detection module; wherein the content of the first and second substances,

a search matching module for matching the first feature point data window within an image block of the current frame to output a motion vector;

the affine transformation module is used for performing data fitting processing on the motion vector to output image affine transformation information;

and the characteristic point detection module is used for carrying out characteristic point detection on the image block of the current frame to obtain a second characteristic point data window.

Specifically, the noise reduction processing module specifically comprises a rotation alignment module, a filtering decision module and a fusion filtering module; wherein, the first and the second end of the pipe are connected with each other,

a rotation alignment module, configured to perform rotation alignment processing on the reference frame based on the motion adjustment information to output a first image frame;

the filtering decision module is used for determining noise reduction filtering strategy information aiming at the current frame and the first image frame based on the gray value of each pixel point in the image block of the current frame, the gray value of each pixel point in the image block of the first image frame and a preset threshold, wherein the noise reduction filtering strategy information comprises time domain noise reduction filtering or space domain noise reduction filtering;

and the fusion filtering module is used for carrying out noise reduction filtering processing on the current frame and/or the first image frame based on the noise reduction filtering strategy information so as to output a second image frame.

In combination with the above description, an embodiment of the present application further provides an electronic device, where the electronic device includes a memory, a communication interface, and the above image noise reduction apparatus. The image noise reduction device includes: the device comprises an image processing module, a filtering processing module and a noise reduction processing module; wherein the content of the first and second substances,

It should be noted that the description of the embodiments in this application has all its emphasis. Therefore, for specific implementation functions of each module in the image noise reduction apparatus of the electronic device, reference may be made to the relevant description in the foregoing embodiments, and details are not repeated here.

Therefore, the image noise reduction device in the electronic device realizes parallel processing of searching matching and feature point detection by directly obtaining the first feature point data window and the current frame, and further realizes reduction of processing time delay in the video noise reduction process through the parallel processing, thereby improving the processing efficiency. Meanwhile, the second characteristic point data window of the current frame is directly uploaded, so that the situation that other redundant data in the current frame are uploaded and stored is avoided, and the bandwidth of data uploading is reduced.

In addition, the image denoising device in the electronic equipment outputs the motion adjustment information by performing time-space domain smoothing filtering on the affine transformation information of the image, so that more accurate and reasonable motion adjustment information is obtained, the accuracy of the rotational alignment of the reference frame is improved, the stability and the accuracy of denoising filtering processing on the current frame and/or the rotationally aligned reference frame are improved, the time-domain denoising effect presented by the second image frame is better, and the stability and the accuracy of the image denoising processing are improved.

For example, please refer to fig. 13, and fig. 13 is a schematic structural diagram of an electronic device according to an embodiment of the present application. The electronic device 1300 includes a memory 1301, a communication interface 1302, an image noise reduction device 1303, and a communication bus for connecting the memory 1301, the communication interface 1302, and the image noise reduction device 1303.

Specifically, the electronic device 1300 according to the embodiment of the present application may be a handheld device, a vehicle-mounted device, a wearable device, an Augmented Reality (AR) device, a Virtual Reality (VR) device, or other devices connected to a wireless modem, or may be various User Equipments (UEs), terminal devices (terminal devices), mobile phones (smart phones), smart screens, smart tvs, smart watches, laptops, stations (STAs), access Points (APs), mobile Stations (MSs), personal Digital Assistants (PDAs), personal Computers (PCs), or relay devices.

Specifically, the image denoising device 1303 may be a processor. The processor may include a CPU, GPU, ISP, DSP, FPGA, ASIC, baseband processor and/or NPU, among others. When the processor is a CPU, the CPU may be a single-core CPU or a multi-core CPU.

Specifically, the memory 1301 may include a double data rate synchronous dynamic random access memory (DDR SDRAM), a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), and/or a portable read-only memory (CD-ROM), and the memory 1301 is used to store related instructions and data.

In particular, communication interface 1302 is configured to receive and transmit data, signals or instructions or the like.

a motion adjustment module to determine motion adjustment information based on the smooth filtering information.

the affine transformation module is used for carrying out data fitting processing on the motion vector so as to output image affine transformation information;

Specifically, the noise reduction processing module specifically comprises a rotation alignment module, a filtering decision module and a fusion filtering module; wherein the content of the first and second substances,

For the sake of simplicity, the above embodiments are described as a series of combinations of operations. Those skilled in the art should appreciate that the present application is not limited by the order of acts described, as some steps in the embodiments of the present application may occur in other orders or concurrently. Moreover, those skilled in the art should also appreciate that the embodiments described in the specification are preferred embodiments and that acts and modules referred to are not necessarily required to implement the embodiments of the application.

In the above embodiments, the description of each embodiment in the present application has a respective emphasis, and reference may be made to related descriptions of other embodiments for parts that are not described in detail in a certain embodiment.

In the several embodiments provided in the present application, it should be understood by those skilled in the art that the described apparatus may be implemented in other ways. It will be appreciated that the above described apparatus embodiments are merely illustrative. For example, the division of the modules is only one logic function division, and actually, other division modes can be provided. That is, multiple modules may be combined or integrated into a single module, and some features may be omitted or not implemented. In addition, the connections between the modules shown or discussed may be indirect coupling connections, direct coupling connections, or communication connections, may be through some interfaces, and may also be electrical or in other forms.

The above modules, if implemented in the form of software functional units and sold or used as separate products, may be stored in a computer readable storage medium. It is to be understood that the technical solutions of the present application (some of which contribute to the prior art or all or part of which) may be embodied in the form of a computer-readable storage medium. The computer readable storage medium may be stored in a memory and includes instructions for causing a computer device (e.g., a personal computer, an electronic device, or a network device) to perform all or part of the steps of the embodiments of the present application. The computer-readable storage medium may be stored in various memories such as a usb disk, a ROM, a RAM, a removable hard disk, a magnetic disk, or an optical disk.

While the embodiments of the present application have been described in detail, it should be understood by those skilled in the art that the embodiments of the present application are only used for assisting understanding of the core concept of the technical solutions of the present application, and therefore, the embodiments of the present application may be changed in terms of the specific implementation and the application scope. The contents described in the present specification should not be construed as limiting the scope of the present application. In addition, any modification, equivalent replacement, improvement and the like made on the basis of the technical solutions of the embodiments of the present application should be included in the protection scope of the embodiments of the present application.

Claims

1. An image noise reduction apparatus, comprising: the device comprises an image processing module, a filtering processing module and a noise reduction processing module; wherein the content of the first and second substances,

the image processing module is used for matching a first feature point data window in an image block of a reference frame in an image block of a current frame to output image affine transformation information and parallelly detecting feature points of the image block of the current frame to obtain a second feature point data window, wherein the reference frame is positioned in front of the current frame in time sequence;

the filtering processing module is used for performing time-space domain smooth filtering processing on the affine transformation information of the image to output motion adjustment information;

2. The apparatus of claim 1, wherein the filter processing module comprises: the device comprises a time-space domain smoothing filtering module and a motion adjusting module; wherein the content of the first and second substances,

the time-space domain smoothing filtering module is used for performing space domain smoothing filtering and/or time domain smoothing filtering on the image affine transformation information corresponding to the image block of the current frame based on the exposure characteristic of the rolling shutter to output smoothing filtering information;

the motion adjustment module is configured to determine the motion adjustment information based on the smoothing filter information.

3. The apparatus of claim 2, wherein the spatio-temporal domain smoothing filter module is configured to:

smoothing first image affine transformation information corresponding to a first image block in a first row of blocks of the current frame and second image affine transformation information corresponding to image blocks located at a previous position and/or a subsequent position of the first image block in the first row of blocks to output first smoothing filtering information in the smoothing filtering information;

wherein the image affine transformation information comprises the first image affine transformation information and the second image affine transformation information.

4. The apparatus of claim 2, wherein the time-space domain smoothing filter module is configured to:

smoothing first image affine transformation information corresponding to a first image block in a first row of blocks of the current frame and third image affine transformation information corresponding to a second image block in a previous row of blocks of the first row to output second smoothing filter information in the smoothing filter information;

wherein the image affine transformation information further comprises the third image affine transformation information.

5. The apparatus of claim 4, wherein if the first row block is a first row block in the row blocks of the current frame, a previous row block of the first row block is a last row block in the row blocks of the reference frame.

6. The apparatus according to any one of claims 1 to 5, wherein the image processing module includes a search matching module, an affine transformation module, and a feature point detection module; wherein the content of the first and second substances,

the search matching module is used for matching the first feature point data window in the image block of the current frame so as to output the motion vector;

the affine transformation module is used for performing data fitting processing on the motion vectors to output the image affine transformation information;

the feature point detection module is configured to perform feature point detection on the image block of the current frame to obtain the second feature point data window.

7. The apparatus according to any one of claims 1-6, wherein the denoising processing module comprises a rotation alignment module, a filtering decision module, and a fusion filtering module; wherein, the first and the second end of the pipe are connected with each other,

the rotational alignment module is configured to perform rotational alignment processing on the reference frame based on the motion adjustment information to output the first image frame;

the fusion filtering module is configured to perform denoising filtering processing on the current frame and/or the first image frame based on the denoising filtering policy information to output the second image frame.

8. An image noise reduction method, comprising:

performing rotation alignment processing on the reference frame based on the motion adjustment information to obtain a first image frame;

9. The method according to claim 8, wherein the performing time-space domain smoothing filtering on the affine transformation information of the image to obtain motion adjustment information comprises:

performing spatial domain smoothing filtering and/or time domain smoothing filtering on the image affine transformation information corresponding to the image block of the current frame based on the exposure characteristic of the rolling shutter to obtain smoothing filtering information;

determining the motion adjustment information based on the smooth filtering information.

10. The method according to claim 9, wherein the performing spatial smoothing filtering and/or temporal smoothing filtering on the image affine transformation information corresponding to the image block of the current frame based on the exposure characteristic of the rolling shutter to obtain smoothing filtering information comprises:

smoothing first image affine transformation information corresponding to a first image block in a first row of blocks of the current frame and second image affine transformation information corresponding to image blocks located at the previous position and/or the next position of the first image block in the first row of blocks to obtain first smoothing filtering information in the smoothing filtering information;

11. The method according to claim 9, wherein the spatial smoothing and/or temporal smoothing of the affine transformation information of the image corresponding to the image block of the current frame based on the exposure characteristic of the rolling shutter to obtain smoothed filter information comprises:

smoothing first image affine transformation information corresponding to a first image block in a first row of blocks of the current frame and third image affine transformation information corresponding to a second image block in a previous row of blocks of the first row to obtain second smoothing filtering information in the smoothing filtering information;

12. The method of claim 11, wherein if the first row block is a first row block in the row blocks of the current frame, a previous row block of the first row block is a last row block in the row blocks of the reference frame.

13. The method according to any of claims 8-12, wherein said matching said first window of feature point data within an image block of said current frame for image affine transformation information comprises:

matching the first feature point data window in the image block of the current frame to obtain a motion vector;

and performing data fitting processing on the motion vector to obtain the image affine transformation information.

14. The method according to any one of claims 8 to 13, wherein the performing the noise reduction filtering process on the current frame and/or the first image frame to obtain a second image frame comprises:

determining noise reduction filtering strategy information aiming at the current frame and the first image frame based on the gray value of each pixel point in the image block of the current frame, the gray value of each pixel point in the image block of the first image frame and a preset threshold, wherein the noise reduction filtering strategy information comprises time domain noise reduction filtering or space domain noise reduction filtering;

and carrying out noise reduction filtering processing on the current frame and/or the first image frame based on the noise reduction filtering strategy information to obtain the second image frame.

15. A processing chip comprising the image noise reduction device according to any one of claims 1 to 7.

16. An electronic device characterized by comprising a memory, a communication interface and the image noise reduction apparatus according to any one of claims 1 to 7.