CN113112398A

CN113112398A - Image processing method and device

Info

Publication number: CN113112398A
Application number: CN202110461864.3A
Authority: CN
Inventors: 叶晓青; 谭啸; 孙昊
Original assignee: Beijing Baidu Netcom Science and Technology Co Ltd
Current assignee: Beijing Baidu Netcom Science and Technology Co Ltd
Priority date: 2021-04-27
Filing date: 2021-04-27
Publication date: 2021-07-13

Abstract

The disclosure provides an image processing method, an image processing device, electronic equipment, a storage medium and a computer program product, relates to the field of artificial intelligence, in particular to computer vision and deep learning, and can be used in an image splicing scene. The specific implementation scheme is as follows: a first image and a second image to be processed are acquired. And generating a difference map according to the absolute difference value of the first image and the second image. And determining the point with the maximum response from the difference map as the characteristic point. A first offset between the position of the feature point in the first image and the position of the feature point in the second image is calculated. A first homography transform matrix is generated for the first image and the second image according to the first offset. The embodiment reduces the time for extracting, describing and matching the feature points and improves the calculation speed.

Description

Image processing method and device

Technical Field

The present disclosure relates to the field of artificial intelligence, and in particular, to computer vision and deep learning, which may be used in image stitching scenarios, and in particular, to an image processing method, an apparatus, an electronic device, a storage medium, and a computer program product.

Background

Homography (Homography) is a concept in projective geometry, also known as projective transformation. It maps points (three-dimensional homogeneous vectors) on one projection plane onto another projection plane. Homography is a linear transformation of three-dimensional homogeneous vectors, which can be represented by a 3 x 3 non-singular matrix H, called homography. Using this matrix, one point on the projection plane can be projected onto another plane.

At present, for panoramic stitching, the existing method is mainly to perform front and back frame stitching based on feature points directly and map the frames to the same image, but feature point extraction, description and matching are time-consuming and are not suitable for end application scenes with real-time requirements.

Disclosure of Invention

The present disclosure provides an image processing method, apparatus, device, storage medium, and computer program product.

According to a first aspect of the present disclosure, there is provided an image processing method including: a first image and a second image to be processed are acquired. And generating a difference map according to the absolute difference value of the first image and the second image. And determining the point with the maximum response from the difference map as the characteristic point. A first offset between the position of the feature point in the first image and the position of the feature point in the second image is calculated. A first homography transform matrix is generated for the first image and the second image according to the first offset.

According to a second aspect of the present disclosure, there is provided an image processing apparatus comprising: an acquisition unit configured to acquire a first image and a second image to be processed. A generating unit configured to generate a difference map from absolute difference values of the first image and the second image. And a determination unit configured to determine a point having the largest response from the difference map as a feature point. A first calculation unit configured to calculate a first offset between a position of the feature point in the first image and a position of the feature point in the second image. A first transformation unit configured to generate a first homography transformation matrix for the first image and the second image according to the first offset.

According to a third aspect of the present disclosure, there is provided an electronic device for image processing, comprising: at least one processor. And a memory communicatively coupled to the at least one processor. Wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of the first aspect.

According to a fourth aspect of the present disclosure, there is provided a non-transitory computer readable storage medium having stored thereon computer instructions for causing a computer to perform the method according to any one of the first aspect.

According to a fifth aspect of the present disclosure, there is provided a computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of the first aspects.

According to the image processing method and device provided by the embodiment of the disclosure, the feature points are quickly searched through the difference image, the first offset of the feature points in the two images is determined, and the first homography transformation matrix is calculated according to the first offset. The method can simplify the process of searching the feature points, reduce the calculated amount in the matching process and improve the image processing speed. The first homography transformation matrix obtained by transformation can be used in the fields of image registration, panoramic stitching, robot positioning SLAM, AR augmented reality and the like.

It should be understood that the statements in this section do not necessarily identify key or critical features of the embodiments of the present disclosure, nor do they limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description.

Drawings

The drawings are included to provide a better understanding of the present solution and are not to be construed as limiting the present disclosure. Wherein:

FIG. 1 is an exemplary system architecture diagram in which one embodiment of the present disclosure may be applied;

FIG. 2 is a flow diagram for one embodiment of an image processing method according to the present disclosure;

FIG. 3 is a flow diagram of yet another embodiment of an image processing method according to the present disclosure;

FIG. 4 is a schematic diagram of an application scenario of an image processing method according to the present disclosure;

FIG. 5 is a schematic block diagram of one embodiment of an image processing apparatus according to the present disclosure;

fig. 6 is a block diagram of an electronic device for implementing an image processing method of an embodiment of the present disclosure.

Detailed Description

Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, in which various details of the embodiments of the disclosure are included to assist understanding, and which are to be considered as merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present disclosure. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.

Fig. 1 shows an exemplary system architecture 100 to which embodiments of the image processing method or image processing apparatus of the present disclosure may be applied.

As shown in fig. 1, the system architecture 100 may include

terminal devices

101, 102, 103, a network 104, and a server 105. The network 104 serves as a medium for providing communication links between the

terminal devices

101, 102, 103 and the server 105. Network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, to name a few.

The user may use the

terminal devices

101, 102, 103 to interact with the server 105 via the network 104 to receive or send messages or the like. The

terminal devices

101, 102, 103 may have various communication client applications installed thereon, such as an image processing application, a web browser application, a shopping application, a search application, an instant messaging tool, a mailbox client, social platform software, and the like.

The

terminal apparatuses

101, 102, and 103 may be hardware or software. When the

terminal devices

101, 102, and 103 are hardware, they may be various electronic devices having a camera and supporting panorama stitching, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III, mpeg compression standard Audio Layer 3), MP4 players (Moving Picture Experts Group Audio Layer IV, mpeg compression standard Audio Layer 4), laptop portable computers, desktop computers, and the like. When the

terminal apparatuses

101, 102, 103 are software, they can be installed in the electronic apparatuses listed above. It may be implemented as multiple pieces of software or software modules (e.g., to provide distributed services) or as a single piece of software or software module. And is not particularly limited herein.

The server 105 may be a server that provides various services, such as a background image processing server that provides support for images displayed on the

terminal devices

101, 102, 103. The background image processing server may analyze and perform other processing on the received data such as the image processing request, and feed back a processing result (e.g., a homography transformation matrix or a spliced image) to the terminal device.

The server may be hardware or software. When the server is hardware, it may be implemented as a distributed server cluster formed by multiple servers, or may be implemented as a single server. When the server is software, it may be implemented as multiple pieces of software or software modules (e.g., multiple pieces of software or software modules used to provide distributed services), or as a single piece of software or software module. And is not particularly limited herein. The server may also be a server of a distributed system, or a server incorporating a blockchain. The server can also be a cloud server, or an intelligent cloud computing server or an intelligent cloud host with artificial intelligence technology.

It should be noted that the image processing method provided by the embodiment of the present disclosure may be executed by the

terminal devices

101, 102, and 103, or may be executed by the server 105. Accordingly, the image processing apparatus may be provided in the

terminal devices

101, 102, 103, or may be provided in the server 105. And is not particularly limited herein.

It should be understood that the number of terminal devices, networks, and servers in fig. 1 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.

With continued reference to FIG. 2, a flow 200 of one embodiment of an image processing method according to the present disclosure is shown. The image processing method comprises the following steps:

step 201, a first image and a second image to be processed are acquired.

In the present embodiment, the execution subject of the image processing method (e.g., the terminal device shown in fig. 1) can capture the first image and the second image by the camera. For example, a user holding a camera at the same location takes a picture of the same building from an angle that is never the same. The image processing method may also be performed by a server that acquires the first image and the second image from the terminal device. The first image and the second image are required to be the same size, and if the sizes are different, the first image and the second image need to be cut to the same size.

Step 202, a difference map is generated according to the absolute difference value of the first image and the second image.

In the present embodiment, two graphs I are given₁，I₂First, make the absolute difference value I of the gray level of each pixel point of the two graphs_diff。

I_diff＝abs(I₁-I₂)

And forming a difference image according to the absolute difference of the gray level of each pixel point.

And step 203, determining the point with the maximum response from the difference map as a characteristic point.

In this embodiment, the point of the difference map where the gray scale value is the maximum is the point of the maximum response. This point is taken as a feature point. The feature points may be mapped to the same positions of the first image and the second image, respectively.

In step 204, a first offset between the position of the feature point in the first image and the position of the feature point in the second image is calculated.

In this embodiment, a first offset between the position of the feature point in the first image and the position of the feature point in the second image may be calculated by calling an existing software product, for example opencv.

The first offset here is a coarse grain offset, which may be sufficient for some application scenarios. For some application scenarios, the fine-grained offset needs to be further calculated, which is described in step 301-306.

Step 205, generating a first homography transformation matrix of the first image and the second image according to the first offset.

In this embodiment, the first offset of the two images is

The first homography transform matrix is denoted as H_initThen, there are:

wherein the content of the first and second substances,

is the offset between the abscissa of the feature point in the first image and the abscissa of the feature point in the second image,

is the offset between the ordinate of the feature point in the first image and the ordinate of the feature point in the second image.

The method provided by the above embodiment of the present disclosure quickly determines the feature points of the two images through the difference map, and uses the deviation of the feature points as the deviation of the two images, so that the feature point detection speed can be increased, and the calculation amount can be reduced. Therefore, the method is suitable for end application scenes with real-time requirements, such as shooting of a camera panorama.

In some optional implementations of this embodiment, the method further includes: the first image is transformed according to the first homography matrix to generate a mapped image of the second image. And splicing the second image to the mapping image to obtain a spliced image. The splicing process comprises the following specific steps:

1. the position of the strongest matching point (which may be directly the feature point determined in step 203) in the first image and the second image is found.

2. The position coordinate of the strongest matching point of the image 1 obtained by the homography matrix transformation is projected on a new image after being mapped

3. At the mapping coordinates of the strongest matching point on the new image, the two images are joined, the left image of the point being completely the first image and the right being completely the second image.

And the visual field shot by the camera is enlarged through image splicing to form a panoramic image. But also can be applied to other fields, such as robot positioning SLAM, AR augmented reality. The method can be used for quickly splicing the images, improves the splicing speed, reduces the image processing time delay and improves the shooting experience of users.

In some optional implementations of this embodiment, after generating the difference map from the absolute difference values of the first image and the second image, the method further comprises: and filtering the difference image through a block filter to obtain a filtered difference image. The window of the block filter is W × W for performing denoising processing. And the block filter takes the average value of the gray values of each W-W area in the original difference graph as the gray value of the central pixel point of the area. Thereby reducing the influence of noise and improving the accuracy of the generated difference map, and thus improving the accuracy of the calculated offset.

In some optional implementations of this embodiment, calculating a first offset between the position of the feature point in the first image and the position of the feature point in the second image includes: a first image area and a second image area of a predetermined size are cut out from the first image and the second image, respectively, with the feature point as a center. And respectively carrying out fast Fourier transform on the first image area and the second image area to obtain two frequency domain expressions. And solving the cross-power spectrum according to the two frequency domain expressions. And carrying out inverse Fourier transform on the cross-power spectrum to obtain a first offset. The predetermined size may coincide with the window size of the block filter, all W x W.

Obtaining a dirac function (pulse function) by solving the inverse Fourier transform of the cross-power spectrum, and then searching the coordinates corresponding to the peak points of the function, thus obtaining the registration points to be obtained. In practice, in computer processing, the continuous domain is replaced by a discrete domain, which transforms the dirac function into the form of a sequence of discrete time unit impulse functions. In practical operation, the inverse transformation of the cross-power spectral phase of two images always contains a correlation peak value representing the registration point of the two images and some non-correlation peak values, and the correlation peak value directly reflects the degree of consistency between the two images. This way the offset can be calculated quickly.

With further reference to fig. 3, a flow 300 of yet another embodiment of an image processing method is shown. The flow 300 of the image processing method comprises the following steps:

step 301, translating the first image according to the first offset to obtain a third image.

In the present embodiment, the steps 301-. The executing subject translates the first image according to the first offset calculated in step 204, resulting in a new image, referred to herein as the third image. Let the drawing I₁Through coarse-grained translation

Then is I₁′。

Step 302, performing corner detection on the second image and the third image respectively, and determining a predetermined number of corners.

In this embodiment, in FIG. I₁' and scheme I₂Corner detection is applied separately. The current corner detection algorithm can be classified into 3 types: detecting corners based on gray level images, detecting corners based on binary images, and detecting corners based on contour curves. Any angular point detection method commonly used in the prior art can be adopted, and details are not repeated here. Corner points may be detected in the second image and the third image, respectively, and the reliability of each corner point may be output. The number of corner points detected on each image is not necessarily the same, e.g. 8 corner points are detected on the second image and 12 corner points are detected on the third image. Corner points with a reliability greater than a predetermined value may be retained for performing subsequent steps. The number of corner points that are finally retained is at least 4. The same may be retained on the second image and the third image, respectivelyA different number of corner points may also be reserved. The positions of the detected corner points on the second image and the third image may also be different, and the corner points detected by the image may be mapped to another image, and the total number of corner points reaches at least 4.

In some optional implementation manners of this embodiment, corner detection is performed on the second image to obtain the first candidate corner set and the reliability of each first candidate corner. And carrying out corner detection on the third image to obtain a second candidate corner set and the reliability of each second candidate corner. A predetermined number of corners with the highest degree of reliability is selected from the union of the first set of candidate corners and the second set of candidate corners. For example, in FIG. I₁' and scheme I₂The corner detection is respectively applied, then the reliability maps of two corners are merged to obtain scoremap, and then 16 points with the highest score value are selected from the scoremap and recorded as

It is preferable to reserve 16 corner points. The subsequent steps can be performed as long as more than 4 corner points remain. Therefore, effective angular points are screened out, and the registration efficiency and accuracy are improved.

Step 303 calculates a second offset between the position in the second image and the position in the third image of each of the predetermined number of corner points.

In this embodiment, the second offset may be calculated by directly using an existing software tool, and the calculation method may be the same as the method for calculating the first offset in step 204, and therefore, the description thereof is omitted.

In some optional implementations of the present embodiment, for each of a predetermined number of corner points, a first image block and a second image block of the corner point of a predetermined size are cut out from the second image and the third image, respectively, centering on the corner point. And respectively carrying out fast Fourier transform on the first image block and the second image block of each angular point to obtain two frequency domain expressions of each angular point. And solving the cross-power spectrum of each corner point according to the two frequency domain expressions of each corner point. And performing inverse Fourier transform on the cross-power spectrum of each corner point to obtain a second offset between the position of each corner point in the second image and the position of each corner point in the third image.

Based on the figure I₁' and scheme I₂And the remaining corner points, referring to step 204, respectively cutting image blocks with each corner point as the center and the window size of W x W, performing FFT and calculating offset

Which represents the shift of the abscissa in the direction of the axis,

indicating the ordinate offset, i being the number of the corner point. The present disclosure takes 16 corner points as an example, and the number of actual corner points is not limited to 16, as long as it is not less than 4.

By the method, the angular point registration speed can be increased, so that the method can be applied to terminal equipment with real-time requirements.

And step 304, calculating a final offset of each corner point according to the first offset and the second offset of each corner point.

In this embodiment, based on the 16 corner points obtained by the previous calculation, each corner point corresponding to the original image I1 and the image I are obtained by calculation₂Offset amount of (2):

for the purpose of the final shift of the abscissa,

is the final ordinate offset.

And 305, determining the position of each corner point in the first image according to the final offset of each corner point and the position of each corner point in the third image.

In this embodiment, the original drawing I₁The 16 angular positions on the upper plate are:

step 306, determining a second homography matrix of the first image and the second image according to the position of each corner point in the first image and the position of each corner point in the second image.

In this embodiment, openHomoghy calculations that can call opencv are based on the original image I₁And I₂The corresponding position of the corner point on the upper panel is calculated to obtain a homography matrix, and the homography matrix is named as a second homography matrix here for distinguishing from the homography matrix obtained in step 205.

In some optional implementations of this embodiment, the method further includes: and converting the first image according to the second homography matrix to generate a mapping image of the second image. And splicing the second image to the mapping image to obtain a spliced image. The splicing process is similar to the splicing process in the flow 200, and thus is not described in detail.

As can be seen from fig. 3, compared with the embodiment corresponding to fig. 2, the flow 300 of the image processing method in the present embodiment represents the step of calculating the offset from the refinement point to the fine point. Therefore, the scheme described in the embodiment can greatly improve the calculation speed, and obtain a more accurate homography matrix, thereby improving the image splicing effect and speed.

With continued reference to fig. 4, fig. 4 is a schematic diagram of an application scenario of the method of image processing according to the present embodiment. In the application scenario of fig. 4, the following steps are mainly included:

1. two-frame subtraction preprocessing

Given two graphs I₁,I₂First, the absolute difference value I of the two graphs is made_diffThen, a block filter (boxfilter) with a window size W x W is applied.

I_diff＝abs(I₁-I₂)

The difference value after block filtering is shown as I_box.

2. Coarse grain offset calculation

For the difference value graph I obtained in the step 1_boxTaking the point P with the maximum response, taking the point P as the center, in the graph I₁,I₂On which an image area C of W x W is cut₁,C₂And respectively performing fast Fourier transform to obtain corresponding frequency domain expression of

Assume that the two center offsets that one wants to solve for are

Then, according to the time domain and frequency domain transformation relation, solving the cross power spectrum:

and then, solving the inverse Fourier transform of the cross-power spectrum to obtain the coarse grain offset of the two images, namely the first offset:

the initial homography transformation matrix is denoted as H_initThen, there are:

3. corner detection and extraction

Let the drawing I₁Through coarse-grained translation

Then is I₁'. In the figure I₁' and scheme I₂In this example, for convenience, only 2 corner points are detected on each image, and then the detected corner points are mapped to the same positions of another image, so that 4 corner points are finally provided on each image. In practice, 8 corner points can be detected in each image, and then the corner points in another image are mapped, and each image has 16 corner points.

4. Refined offset calculation

Based on the figure I₁' and scheme I₂And the detected angular points and the mapped angular points, referring to the steps 1 and 2, respectively cutting image blocks which take each angular point as the center and have the window size of W x W, performing FFT and calculating offset

n is the number of corner points on each graph. In this example, only the calculation process of the second offset of 1 corner point is schematically listed, and the other corner points perform the same operation, so that the FFT calculation of the window corresponding to all corner points is not shown.

5. Computing homography transformations between preceding and following frames

Calculating to obtain an original image I based on the angular points obtained by the previous step₁Each corner point corresponding to the upper part of the figure I₂Final offset of (c):

according to the final offset and the graph I₁' Upper corner position gets the original picture I₁The upper corner positions are:

then calls opencv findHomogright calculation based on the original image I₁And I₂And calculating to obtain a homography matrix at the position corresponding to the upper angular point.

The homography matrix may be used to map I₁And I₂Spliced together.

With further reference to fig. 5, as an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of an image processing apparatus, which corresponds to the embodiment of the method shown in fig. 2, and which is particularly applicable in various electronic devices.

As shown in fig. 5, the image processing apparatus 500 of the present embodiment includes: an acquisition unit 501, a generation unit 502, a determination unit 503, a first calculation unit 504, and a first transformation unit 505. Wherein, the acquiring unit 501 is configured to acquire a first image and a second image to be processed. A generating unit 502 configured to generate a difference map from the absolute difference values of the first image and the second image. A determining unit 503 configured to determine a point having the largest response from the difference map as a feature point. A first calculation unit 504 configured to calculate a first offset between the position of the feature point in the first image and the position of the feature point in the second image. A first transformation unit 505 configured to generate a first homography transformation matrix for the first image and the second image according to the first offset.

In the present embodiment, specific processing of the acquisition unit 501, the generation unit 502, the determination unit 503, the first calculation unit 504, and the first transformation unit 505 of the image processing apparatus 500 may refer to step 201, step 202, step 203, step 204, and step 205 in the corresponding embodiment of fig. 2.

In some optional implementations of this embodiment, the apparatus 500 further comprises a second transformation unit (not shown in the drawings) configured to: and translating the first image according to the first offset to obtain a third image. And respectively carrying out corner detection on the second image and the third image to determine a predetermined number of corners. A second offset is calculated between the position of each of the predetermined number of corner points in the second image and the position in the third image. A final offset for each corner point is calculated based on the first offset and the second offset for each corner point. And determining the position of each corner point in the first image according to the final offset of each corner point and the position of each corner point in the third image. And determining a second homography matrix of the first image and the second image according to the position of each corner point in the first image and the position of each corner point in the second image.

In some optional implementations of this embodiment, the apparatus 500 further comprises a splicing unit (not shown in the drawings) configured to: and converting the first image according to the first homography matrix or the second homography matrix to generate a mapping image of the second image. And splicing the second image to the mapping image to obtain a spliced image.

In some optional implementations of this embodiment, the generating unit 502 is further configured to: and after generating a difference map according to the absolute difference value of the first image and the second image, filtering the difference map through a block filter to obtain a filtered difference map.

In some optional implementations of this embodiment, the first computing unit 504 is further configured to: a first image area and a second image area of a predetermined size are cut out from the first image and the second image, respectively, with the feature point as a center. And respectively carrying out fast Fourier transform on the first image area and the second image area to obtain two frequency domain expressions. And solving the cross-power spectrum according to the two frequency domain expressions. And carrying out inverse Fourier transform on the cross-power spectrum to obtain a first offset.

In some optional implementations of this embodiment, the second transformation unit is further configured to: and carrying out corner detection on the second image to obtain a first candidate corner set and the reliability of each first candidate corner. And carrying out corner detection on the third image to obtain a second candidate corner set and the reliability of each second candidate corner. A predetermined number of corners with the highest degree of reliability is selected from the union of the first set of candidate corners and the second set of candidate corners.

In some optional implementations of this embodiment, the second transformation unit is further configured to: for each of a predetermined number of corner points, a first image block and a second image block of the corner point of a predetermined size are cut out from the second image and the third image, respectively, centered on the corner point. And respectively carrying out fast Fourier transform on the first image block and the second image block of each angular point to obtain two frequency domain expressions of each angular point. And solving the cross-power spectrum of each corner point according to the two frequency domain expressions of each corner point. And performing inverse Fourier transform on the cross-power spectrum of each corner point to obtain a second offset between the position of each corner point in the second image and the position of each corner point in the third image.

The present disclosure also provides an electronic device, a readable storage medium, and a computer program product according to embodiments of the present disclosure.

FIG. 6 illustrates a schematic block diagram of an example electronic device 600 that can be used to implement embodiments of the present disclosure. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the disclosure described and/or claimed herein.

As shown in fig. 6, the apparatus 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a Read Only Memory (ROM)602 or a computer program loaded from a storage unit 608 into a Random Access Memory (RAM) 603. In the RAM603, various programs and data required for the operation of the device 600 can also be stored. The calculation unit 601, the ROM 602, and the RAM603 are connected to each other via a bus 604. An input/output (I/O) interface 605 is also connected to bus 604.

A number of components in the device 600 are connected to the I/O interface 605, including: an input unit 606 such as a keyboard, a mouse, or the like; an output unit 607 such as various types of displays, speakers, and the like; a storage unit 608, such as a magnetic disk, optical disk, or the like; and a communication unit 609 such as a network card, modem, wireless communication transceiver, etc. The communication unit 609 allows the device 600 to exchange information/data with other devices via a computer network such as the internet and/or various telecommunication networks.

The computing unit 601 may be a variety of general and/or special purpose processing components having processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various dedicated Artificial Intelligence (AI) computing chips, various computing units running machine learning model algorithms, a Digital Signal Processor (DSP), and any suitable processor, controller, microcontroller, and so forth. The calculation unit 601 performs the respective methods and processes described above, such as method image processing. For example, in some embodiments, the method image processing may be implemented as a computer software program tangibly embodied in a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program may be loaded and/or installed onto the device 600 via the ROM 602 and/or the communication unit 609. When the computer program is loaded into the RAM603 and executed by the computing unit 601, one or more steps of the method image processing described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform method image processing by any other suitable means (e.g., by means of firmware).

Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuitry, Field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), system on a chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, receiving data and instructions from, and transmitting data and instructions to, a storage system, at least one input device, and at least one output device.

Program code for implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the program codes, when executed by the processor or controller, cause the functions/operations specified in the flowchart and/or block diagram to be performed. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package partly on the machine and partly on a remote machine or entirely on the remote machine or server.

In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic, speech, or tactile input.

The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), Wide Area Networks (WANs), and the Internet.

The computer system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server may be a server of a distributed system or a server incorporating a blockchain. The server can also be a cloud server, or an intelligent cloud computing server or an intelligent cloud host with artificial intelligence technology. The server may be a server of a distributed system or a server incorporating a blockchain. The server can also be a cloud server, or an intelligent cloud computing server or an intelligent cloud host with artificial intelligence technology.

It should be understood that various forms of the flows shown above may be used, with steps reordered, added, or deleted. For example, the steps described in the present disclosure may be executed in parallel, sequentially, or in different orders, as long as the desired results of the technical solutions disclosed in the present disclosure can be achieved, and the present disclosure is not limited herein.

The above detailed description should not be construed as limiting the scope of the disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made in accordance with design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present disclosure should be included in the scope of protection of the present disclosure.

Claims

1. An image processing method comprising:

acquiring a first image and a second image to be processed;

generating a difference map according to the absolute difference value of the first image and the second image;

determining a point with the maximum response from the difference map as a characteristic point;

calculating a first offset between the position of the feature point in the first image and the position of the feature point in the second image;

generating a first homography transformation matrix for the first image and the second image according to the first offset.

2. The method of claim 1, wherein the method further comprises:

translating the first image according to the first offset to obtain a third image;

respectively carrying out corner detection on the second image and the third image to determine a predetermined number of corners;

calculating a second offset between the position of each of the predetermined number of corner points in the second image and the position in the third image;

calculating the final offset of each corner point according to the first offset and the second offset of each corner point;

determining the position of each corner point in the first image according to the final offset of each corner point and the position of each corner point in the third image;

and determining a second homography matrix of the first image and the second image according to the position of each corner point in the first image and the position of each corner point in the second image.

3. The method of claim 2, wherein the method further comprises:

converting the first image according to the first homography matrix or the second homography matrix to generate a mapping image of the second image;

and splicing the second image to the mapping image to obtain a spliced image.

4. The method of claim 1, wherein after the generating a difference map from absolute difference values of the first and second images, the method further comprises:

and filtering the difference image through a block filter to obtain a filtered difference image.

5. The method of claim 1, wherein said calculating a first offset between the location of the feature point in the first image and the location of the feature point in the second image comprises:

cutting out a first image area and a second image area of a predetermined size from the first image and the second image, respectively, with the feature point as a center;

respectively carrying out fast Fourier transform on the first image area and the second image area to obtain two frequency domain expressions;

solving a cross-power spectrum according to the two frequency domain expressions;

and carrying out inverse Fourier transform on the cross-power spectrum to obtain a first offset.

6. The method of claim 2, wherein the performing corner detection on the second image and the third image, respectively, determining a predetermined number of corners comprises:

performing corner detection on the second image to obtain a first candidate corner set and the reliability of each first candidate corner;

performing corner detection on the third image to obtain a second candidate corner set and the reliability of each second candidate corner;

a predetermined number of corners with a maximum degree of reliability is selected from the union of the first candidate set of corners and the second candidate set of corners.

7. The method of claim 2, wherein said calculating a second offset between a position of each of the predetermined number of corner points in the second image and a position in the third image comprises:

for each of the predetermined number of corner points, cutting out a first image block and a second image block of the corner point with a predetermined size from the second image and the third image respectively by taking the corner point as a center;

respectively carrying out fast Fourier transform on the first image block and the second image block of each angular point to obtain two frequency domain expressions of each angular point;

solving a cross-power spectrum of each corner point according to the two frequency domain expressions of each corner point;

and performing inverse Fourier transform on the cross-power spectrum of each corner point to obtain a second offset between the position of each corner point in the second image and the position of each corner point in the third image.

8. An image processing apparatus comprising:

an acquisition unit configured to acquire a first image and a second image to be processed;

a generating unit configured to generate a difference map from absolute difference values of the first image and the second image;

a determination unit configured to determine a point having the largest response from the difference map as a feature point;

a first calculation unit configured to calculate a first offset between a position of the feature point in the first image and a position of the feature point in the second image;

a first transformation unit configured to generate a first homography transformation matrix for the first image and the second image according to the first offset.

9. The apparatus of claim 8, wherein the apparatus further comprises a second transformation unit configured to:

10. The apparatus of claim 9, wherein the apparatus further comprises a stitching unit configured to:

and splicing the second image to the mapping image to obtain a spliced image.

11. The apparatus of claim 8, wherein the generating unit is further configured to:

and after generating a difference map according to the absolute difference value of the first image and the second image, filtering the difference map through a block filter to obtain a filtered difference map.

12. The apparatus of claim 8, wherein the first computing unit is further configured to:

13. The apparatus of claim 9, wherein the second transform unit is further configured to:

14. The apparatus of claim 9, wherein the second transform unit is further configured to:

15. An electronic device for image processing, comprising:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,

the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-7.

16. A non-transitory computer readable storage medium having stored thereon computer instructions for causing the computer to perform the method of any one of claims 1-7.

17. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1-7.