CN115661478A

CN115661478A - Image processing method, image processing device, electronic equipment and storage medium

Info

Publication number: CN115661478A
Application number: CN202211366531.3A
Authority: CN
Inventors: 胡伟东; 张演龙; 杨尊程; 滕禹桥
Original assignee: Beijing Baidu Netcom Science and Technology Co Ltd
Current assignee: Beijing Baidu Netcom Science and Technology Co Ltd
Priority date: 2022-11-02
Filing date: 2022-11-02
Publication date: 2023-01-31

Abstract

The disclosure provides an image processing method, and relates to the technical field of artificial intelligence, deep learning, image processing and computer vision. The specific implementation scheme is as follows: selecting N initial matching point pairs from the characteristic point matching regions of the first image and the second image to generate an initial homography matrix; selecting M verification matching point pairs from the feature point matching area to verify the initial homography matrix to obtain an inner point set; under the condition that the interior point set meets a preset condition, generating a candidate homography matrix by using the interior point set; determining the maximum return times; responding to the fact that the current return times are smaller than the maximum return times, returning to the step of generating the initial homography matrix until the current return times are not smaller than the maximum return times, and obtaining at least one candidate homography matrix; and determining a target homography matrix from the at least one candidate homography matrix, and splicing the first image and the second image by using the target homography matrix. The present disclosure also provides an image processing apparatus, an electronic device, and a storage medium.

Description

Image processing method, image processing device, electronic equipment and storage medium

Technical Field

The present disclosure relates to the field of artificial intelligence technology, and more particularly, to techniques for deep learning, image processing, and computer vision. More particularly, the present disclosure provides an image processing method, apparatus, electronic device, and storage medium.

Background

Homography matrices (also called homography matrices), which refer to transformation matrices that map from one point in one image to a corresponding point in the other image, usually describe the mapping of three-dimensional points on the same plane in space in both images.

Disclosure of Invention

The disclosure provides an image processing method, an apparatus, a device and a storage medium.

According to a first aspect, there is provided an image processing method comprising: selecting N initial matching point pairs from the characteristic point matching regions of the first image and the second image to generate an initial homography matrix, wherein N is an integer larger than 1; selecting M verification matching point pairs from the feature point matching region to verify the initial homography matrix to obtain a verified internal point set, wherein M is an integer larger than 1; under the condition that the internal point set meets the preset condition, generating a candidate homography matrix by using the internal point set; determining the maximum return times according to the expected probability that the M verification matching point pairs are all interior points; responding to the situation that the current return times are smaller than the maximum return times, returning to the step of generating the initial homography matrix, increasing the return times until the current return times are not smaller than the maximum return times, and obtaining at least one candidate homography matrix; and determining a target homography matrix from the at least one candidate homography matrix, and stitching the first image and the second image by using the target homography matrix.

According to a second aspect, there is provided an image processing apparatus comprising: the first generation module is used for selecting N initial matching point pairs from the characteristic point matching regions of the first image and the second image to generate an initial homography matrix, wherein N is an integer larger than 1; the verification module is used for selecting M verification matching point pairs from the characteristic point matching area to verify the initial homography matrix to obtain a verified internal point set, wherein M is an integer larger than 1; the second generation module is used for generating a candidate homography matrix by using the internal point set under the condition that the internal point set meets the preset condition; the first determining module is used for determining the maximum returning times according to the expected probability that the M verification matching point pairs are all interior points; the processing module is used for responding to the fact that the current return times are smaller than the maximum return times, returning to the step of generating the initial homography matrix, increasing the return times until the current return times are not smaller than the maximum return times, and obtaining at least one candidate homography matrix; and the splicing module is used for determining a target homography matrix from the at least one candidate homography matrix and splicing the first image and the second image by using the target homography matrix.

According to a third aspect, there is provided an electronic device comprising: at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform methods provided in accordance with the present disclosure.

According to a fourth aspect, there is provided a non-transitory computer readable storage medium having stored thereon computer instructions for causing a computer to perform a method provided in accordance with the present disclosure.

According to a fifth aspect, a computer program product is provided, comprising a computer program stored on at least one of a readable storage medium and an electronic device, which computer program, when executed by a processor, implements a method provided according to the present disclosure.

It should be understood that the statements in this section are not intended to identify key or critical features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description.

Drawings

The drawings are included to provide a better understanding of the present solution and are not to be construed as limiting the present disclosure. Wherein:

FIG. 1 is a schematic diagram of an exemplary system architecture to which the image processing method and apparatus may be applied, according to one embodiment of the present disclosure;

FIG. 2 is a flow diagram of an image processing method according to one embodiment of the present disclosure;

FIG. 3 is a flow diagram of a method of determining a target homography matrix according to one embodiment of the present disclosure;

FIG. 4 is a schematic diagram of feature point matching regions of a first image and a second image according to one embodiment of the present disclosure;

FIG. 5 is a block diagram of an image processing apparatus according to one embodiment of the present disclosure;

fig. 6 is a block diagram of an electronic device of an image processing method according to one embodiment of the present disclosure.

Detailed Description

Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, in which various details of the embodiments of the disclosure are included to assist understanding, and which are to be considered as merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present disclosure. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.

Homography matrices have many applications in the image domain. For example, in a scene of a panoramic image, two images including the same scene can be spliced by using the homography matrix to obtain the panoramic image.

For two images to be spliced, a point in one image and a corresponding point in the other image can form a matching point pair, and a homography matrix can be obtained through calculation according to the matching point pair. But often multiple iterations are required to obtain a homography that best fits all the pairs of matching points. Therefore, the calculation of the homography matrix has the problems of slow speed and long time consumption, and the image processing efficiency is low.

It should be noted that besides image stitching, the homography matrix can also be applied to various image processing scenarios such as image correction (e.g., returning an image with an oblique angle to the original), view angle conversion (e.g., converting all points in one image to the plane of another image), and the like. In addition, the homography matrix can also be applied to various scenes such as camera pose estimation and augmented reality.

In the technical scheme of the disclosure, the collection, storage, use, processing, transmission, provision, disclosure and other processing of the personal information of the related user are all in accordance with the regulations of related laws and regulations and do not violate the good customs of the public order.

In the technical scheme of the disclosure, before the personal information of the user is obtained or collected, the authorization or the consent of the user is obtained.

Fig. 1 is a schematic diagram of an exemplary system architecture to which the image processing method and apparatus may be applied, according to one embodiment of the present disclosure. It should be noted that fig. 1 is only an example of a system architecture to which the embodiments of the present disclosure may be applied to help those skilled in the art understand the technical content of the present disclosure, and does not mean that the embodiments of the present disclosure may not be applied to other devices, systems, environments or scenarios.

As shown in fig. 1, the system architecture 100 according to this embodiment may include

terminal devices

101, 102, 103, a network 104 and a server 105. Network 104 is the medium used to provide communication links between

terminal devices

101, 102, 103 and server 105. Network 104 may include various connection types, such as wired and/or wireless communication links, and so forth.

The user may use the

terminal devices

101, 102, 103 to interact with the server 105 via the network 104 to receive or send messages or the like. The

terminal devices

101, 102, 103 may be various electronic devices including, but not limited to, smart phones, tablet computers, laptop computers, and the like.

The image processing method provided by the embodiments of the present disclosure may be generally performed by the server 105. Accordingly, the image processing apparatus provided by the embodiment of the present disclosure may be generally provided in the server 105. The image processing method provided by the embodiment of the present disclosure may also be performed by a server or a server cluster that is different from the server 105 and is capable of communicating with the

terminal devices

101, 102, 103 and/or the server 105. Accordingly, the image processing apparatus provided by the embodiment of the present disclosure may also be disposed in a server or a server cluster different from the server 105 and capable of communicating with the

terminal devices

101, 102, 103 and/or the server 105.

Fig. 2 is a flowchart of an image processing method according to one embodiment of the present disclosure.

As shown in fig. 2, the image processing method 200 may include operations S210 to S260.

In operation S210, N initial matching point pairs are selected from the feature point matching regions of the first image and the second image to generate an initial homography matrix.

For example, the first image and the second image may be two images to be stitched, the first image and the second image containing an overlapping region. The first image and the second image may be adjacent video frames in a video for a preset scene. The preset scene may be a scene containing characters, scenes, or text contents.

For example, the feature point matching region of the first image and the second image may be an overlapping region of the first image and the second image. The feature point matching region may include a plurality of matching point pairs, each of which is composed of a feature point of the first image and a feature point of the second image that correspond to each other, and the feature points of the first image and the feature points of the second image that correspond to each other may be regarded as feature points of the same object (e.g., the same object, the same word). The feature points may refer to points (e.g., corner points) in the image, which have distinctive features and can effectively reflect essential features of the image and can identify objects in the image.

For example, feature extraction may be performed on the first image and the second image respectively by using a deep learning model, so as to obtain a feature point set of the first image and a feature point set of the second image. The deep learning model is constructed based on, for example, ORB (organized Fast and Rotated Brief) algorithm, which is improved based on Fast (Features from estimated segment Test) algorithm and Brief (Binary route Independent element Feature) algorithm. Then, a KNN (K-Nearest-Neighbors, nearest rule classification) algorithm may be used to perform bidirectional matching on the feature point set of the first image and the feature point set of the second image, so as to obtain a matching point pair set.

For example, N matching point pairs (N is an integer greater than 1, e.g., N = 100) may be randomly selected from the feature point matching area as initial matching point pairs, and an initial homography matrix representing a mapping relationship between two feature points corresponding to each initial matching point pair may be obtained according to a positional relationship between the two feature points.

It will be appreciated that the initial homography may characterize the mapping between two feature points in the initial pair of matching points, in other words, the initial pair of matching points conforms to the mapping described by the initial homography. However, in the feature point matching region, not all the matching point pairs conform to the mapping relationship described in the initial homography matrix.

In operation S220, M verification matching point pairs are selected from the feature point matching region to verify the initial homography matrix, and a verified interior point set is obtained.

For example, M (M is an integer greater than 1, for example, M = 100) matching point pairs may be randomly selected from the remaining matching point pairs excluding the initial matching point in the feature point matching region as verification matching point pairs. And verifying the initial homography matrix by using the M verification matching point pairs to obtain a verified inner point set and a non-verified outer point set.

For example, the matching point pairs in the inner point set are in accordance with the mapping relationship described by the initial homography matrix, and the matching point pairs in the outer point set are not in accordance with the mapping relationship described by the initial homography matrix. Therefore, in order to obtain the homography matrix most suitable for all the matching point pairs, a new homography matrix (candidate homography matrix) needs to be further calculated.

In operation S230, in the case where the internal point set meets a preset condition, a candidate homography matrix is generated using the internal point set.

For example, the sum of the number of matching point pairs in the inner point set and the number of matching point pairs in the outer point set is M, and the ratio between the number of matching point pairs in the inner point set and the number M of verification matching point pairs may be referred to as an inner point proportion.

If the ratio of the interior points meets the preset condition (for example, greater than 0.5), which indicates that the effect of the current initial homography matrix is still better, a new homography matrix with better effect can be generated by using the matching point pairs in the verified interior point set as the candidate homography matrix. If the internal point ratio does not meet the preset condition, it indicates that the effect of the current initial homography matrix is not good, and the operation may return to operation S210 to recalculate the initial homography matrix.

In operation S240, a maximum number of returns is determined according to an expected probability that the M verification matching point pairs are all inliers.

For example, a process from the calculation of the initial homography matrix in operation S210 to the calculation of the candidate homography matrix in operation S230 may be referred to as a process of finding the candidate homography matrix. After operation S230, the process may return to operation S210 to perform the next round of search process of the candidate homography matrix.

However, in order to avoid the situation that the quality of the matching point pair is poor (for example, the correlation between two images is small, and the ratio of the interior points is small), the number of times of returning to calculate the initial homography matrix (the number of times of searching for the candidate homography matrix) is high, which causes the problems of high peak value of the number of times of returning and long time consumption, and the maximum number of times of returning can be calculated once in each round.

For example, in each round, the maximum number of returns may be calculated according to the expected probability that the selected M verification matching point pairs are all inliers. The desired probability may be a preset value, and the higher the desired probability (e.g., desired probability P = 0.99), the better the effect of the initial homography matrix is required.

For example, the expected probability P that M verification matching point pairs selected each time in the K maximum return times are all inliers may be set to reach a preset value (e.g., 0.99), thereby reversing the maximum return times.

In operation S250, it is determined whether the current return time is less than the maximum return time. If so, return to operation S210 and increment the number of returns; otherwise, operation S260 is performed.

For example, if the current number of returns is less than the maximum number of returns K, the operation S210 may be returned, the number of returns is increased by 1, and the search process of the homography matrix is continued. And if the current return times are more than or equal to the maximum return times K, ending the search process of the homography matrix. And obtaining the candidate homography matrix obtained in each turn, and obtaining at least one candidate homography matrix.

In operation S260, a target homography matrix is determined from the at least one candidate homography matrix, and the first image and the second image are stitched using the target homography matrix.

For example, the candidate homography matrix with the highest ratio of interior points can be selected from the at least one homography matrix as the target homography matrix. According to the target homography matrix, the characteristic points of the first image and the characteristic points of the second image which belong to the same three-dimensional point can be in one-to-one correspondence, the characteristic points of the first image and the characteristic points of the second image which are corresponding to each other are spliced, the first image and the second image can be spliced, and the panoramic image is obtained.

According to the embodiment of the disclosure, in the process of searching the candidate homography matrix in each round, the maximum return times are determined according to the expected probability that M verification matching point pairs are all interior points, so that the maximum return times are reduced along with the increment of the current return times, and the problems of high time consumption and long time consumption of the times of searching the candidate homography matrix can be avoided. Therefore, the efficiency of searching the candidate homography matrix can be improved, and the image splicing efficiency is further improved.

According to an embodiment of the present disclosure, determining the maximum number of return times according to the expected probability that M verification matching point pairs are all inliers includes: determining expected probability that M verification matching point pairs selected each time are the interior points after K times of returning according to the interior point proportion, wherein K is the maximum returning time; and determining the maximum return times K according to the expected probability and the current return times.

For example, the desired probability may be determined according to the following equation (1):

P＝1-(1-w ^M ) ^K (1)

wherein P is the expected probability, w is the internal point proportion, M is the number of verification matching point pairs, and K is the maximum return time.

It can be understood that w ^M Represents the probability that the selected M verification matching point pairs are all interior points, 1-w ^M Indicating the probability of at least one outlier in the selected M pairs of verified matching points, (1-w) ^M ) ^K Representing M verification matches selected each time through K roundsProbability of at least one outer point in the pairing point pair, P =1- (1-w) ^M ) ^K And representing the expected probability that the M verification matching point pairs selected each time are all the interior points after K rounds.

For example, the maximum number of returns may be determined according to the following equation (2):

wherein K is the maximum return time, P is the expected probability, i is the current return time, and b is a constant. The desired probability P may take a preset value (e.g., 0.99).

It will be appreciated that the maximum number of returns K to reverse the thrust is linearly decreasing according to the desired probability P. And a constraint term (i/b + 1) is added in the formula for reversely deducing the maximum return times K, so that the maximum return times K are reduced in a step shape. For example, b =10, when the current number of returns i =10, the denominator of equation (2) may be increased by 2 times, and thus, the maximum number of returns K may be decreased by 2 times. Therefore, the maximum number of returns K is reduced by a factor of two for every 10 passes.

Fig. 3 is a flow diagram of a method of determining a target homography matrix according to one embodiment of the present disclosure.

As shown in fig. 3, the method includes operations S310 to S390.

In operation S310, a feature point set of the first image is matched with a feature point set of the second image, so as to obtain a plurality of feature point matching regions.

For example, after the feature point set of the first image and the feature point set of the second image are subjected to bidirectional matching by using a KNN algorithm, a matching point pair set is obtained. Matching point pairs which may have mismatching in the matching point pair set may be removed by using a GMS (Grid-based Motion Statistics) algorithm, and the matching point pair set is divided into a plurality of feature point matching regions.

For example, the GMS algorithm can convert the statistics of motion smoothness into statistics that cull false matches. In the overlapping region of the first image and the second image, there are more matching point pairs around the correct matching point pair to support it, and less matching point pairs around the wrong matching point pair to support it. Therefore, the GMS algorithm may be used to eliminate the wrong matching point pairs according to the distances between the matching point pairs in the matching point pair set, so as to obtain a plurality of feature point matching regions containing more correct matching point pairs.

In operation S320, an initial homography matrix is generated using N initial matching point pairs selected from the plurality of feature point matching regions.

In operation S330, the initial homography matrix is verified using M verification matching point pairs selected from the plurality of valid matching regions, resulting in an interior point set.

For example, N matching point pairs may be selected from each of the feature point matching regions as initial matching point pairs, so as to obtain N initial matching point pairs. And for each characteristic point matching region, determining residual matching point pairs except the initial matching points in the characteristic point matching region, and then selecting M matching point pairs from the residual matching point pairs in each characteristic point matching region as verification matching point pairs to obtain M verification matching point pairs.

For example, there are 200 feature point matching regions, n is 1, and 1 matching point pair may be selected from each of the 200 feature point matching regions as an initial matching point pair, so that 200 initial matching point pairs may be obtained. And m is 1, and 1 matching point pair can be selected from the remaining matching point pairs except the initial matching point pair in each feature point matching region as verification matching point pairs, so that 200 verification matching point pairs can be obtained.

For example, an initial homography matrix is generated by using 200 selected initial matching point pairs, and the initial homography matrix is verified by using 200 verification matching point pairs, so that a verified inner point set and a non-verified outer point set are obtained.

In operation S340, an interior point ratio is calculated.

For example, the ratio between the number of matching point pairs in the inner point set and the number M of verification matching point pairs may be calculated to obtain the inner point proportion w.

In operation S350, a maximum number of iterations k is calculated.

For example, the maximum number of returns K may be calculated according to the above formula (2) based on the inlier proportion w and the expected probability P that M verification matching point pairs are all inliers.

In operation S360, it is determined whether the interior point proportion is greater than a threshold value. If so, perform operation S370; otherwise, operation 380 is performed.

In operation S370, a candidate homography matrix is generated using the set of interior points.

In operation S380, it is determined whether the current return time i is less than the maximum return time K. If so, return to operation S320; otherwise, operation S390 is performed.

In operation S390, a target homography matrix is determined from at least one candidate homography matrix.

For example, in the case that the internal point ratio is greater than a threshold (e.g., 0.5), a candidate homography matrix may be generated using the set of internal points, and then it may be determined whether the current number of returns i is less than the maximum number of returns K. And under the condition that the internal point occupation ratio is not greater than the threshold value, directly judging whether the current return frequency i is less than the maximum return frequency K.

If i is less than K, the operation returns to step S320, the number of times of return is increased by 1, and the search process of the candidate homography matrix of the next round is performed. If i is larger than or equal to K, the searching process of the homography matrix is ended.

Then, at least one searched candidate homography matrix can be obtained, and the candidate homography matrix with the highest ratio of internal points is selected as the target homography matrix.

The method for determining the target homography matrix provided by the embodiment can effectively reduce the peak value of the return times, reduce time consumption, reduce the calculation amount, and improve the calculation efficiency, so that the image processing efficiency is improved.

Fig. 4 is a schematic diagram of feature point matching regions of a first image and a second image according to one embodiment of the present disclosure.

As shown in fig. 4, the first image 410 and the second image 420 are two images to be stitched, and an overlapping area between the first image 410 and the second image 420 is an overlapping area 430. The overlap region 430 includes a set of matching point pairs of the first image 410 and the second image 420, which may be divided into a plurality of feature point matching regions, such as a feature point matching region 431, a feature point matching region 432, a feature point matching region 433, and the like, using a GMS algorithm.

For example, n (e.g., n = 1) matching point pairs may be selected as initial matching point pairs from the feature point matching region 431, the feature point matching region 432, and the feature point matching region 433, respectively, to generate an initial homography matrix. The initial homography matrix may be verified by selecting m (e.g., m = 1) matching point pairs from the feature point matching region 431, the feature point matching region 432, and the feature point matching region 433, respectively, as verification matching point pairs.

In the embodiment, the matching point pair set is divided into a plurality of feature point matching regions, and the initial matching point pair and the verification matching point pair are respectively selected from each feature point matching region.

Fig. 5 is a block diagram of an image processing apparatus according to one embodiment of the present disclosure.

As shown in fig. 5, the image processing apparatus 500 includes a first generation module 501, a verification module 502, a second generation module 503, a first determination module 504, a processing module 505, and a stitching module 506.

The first generation module 501 is configured to select N initial matching point pairs from the feature point matching regions of the first image and the second image to generate an initial homography matrix, where N is an integer greater than 1.

The verification module 502 is configured to select M verification matching point pairs from the feature point matching area to verify the initial homography matrix, so as to obtain a verified internal point set, where M is an integer greater than 1.

The second generating module 503 is configured to generate a candidate homography matrix by using the internal point set when the internal point set meets the preset condition.

The first determining module 504 is configured to calculate and determine the maximum number of times of returning according to the expected probabilities that the M verification matching point pairs are all inliers.

The processing module 505 is configured to, in response to that the current return times are smaller than the maximum return times, return to the step of generating the initial homography matrix, and increment the return times until the current return times are not smaller than the maximum return times, so as to obtain at least one candidate homography matrix.

The stitching module 506 is configured to determine a target homography matrix from the at least one candidate homography matrix, and stitch the first image and the second image using the target homography matrix.

According to an embodiment of the present disclosure, the image processing apparatus 500 further includes a calculation module.

The calculation module is used for calculating the proportion between the number of the matching point pairs in the inner point set and the number M of the verification matching point pairs to obtain the inner point proportion.

The second generating module 503 is configured to generate a candidate homography matrix by using the interior point set when the interior point ratio meets a preset condition.

The first determination module includes a first determination unit and a second determination unit.

The first determining unit is used for determining the expected probability that the M verification matching point pairs selected each time are all the interior points after K times of returning according to the interior point ratio, wherein K is the maximum returning time.

The second determining unit is used for determining the maximum returning times K according to the expected probability and the current returning times.

According to an embodiment of the present disclosure, the first determination unit is configured to determine the desired probability according to the following formula:

P＝1-(1-w ^M ) ^K

A second determining unit for determining the maximum number of returns according to the following formula:

wherein, P is the expected probability, i is the current return times, and b is a constant.

According to an embodiment of the present disclosure, the image processing apparatus 500 further includes a second determination module.

The second determining module is used for determining whether the current return times are less than the maximum return times under the condition that the internal point proportion does not accord with the preset condition.

Wherein the processing module 505 is configured to return to the first generating module 501 and the number of returns is incremented in response to the second determining module determining that the current number of returns is less than the maximum number of returns.

The splicing module 506 is configured to determine a candidate homography matrix with the highest interior point ratio from the at least one candidate homography matrix as a target homography matrix.

According to an embodiment of the present disclosure, the feature point matching regions of the first image and the second image include a plurality of feature point matching regions. The image processing apparatus 500 further comprises a third determining module and a matching module.

The third determination module is configured to determine a set of feature points of the first image and a set of feature points of the second image.

The matching module is used for matching the feature point set of the first image with the feature point set of the second image to obtain a plurality of feature point matching regions, wherein each feature point matching region comprises at least one matching point pair.

The first generating module 501 is configured to select N matching point pairs from each feature point matching region as initial matching point pairs to obtain N initial matching point pairs, where N is an integer greater than or equal to 1.

The verification module 502 is configured to determine, for each feature point matching region, remaining matching point pairs in the feature point matching region except for the initial matching point; and selecting M matching point pairs from the residual matching point pairs in each characteristic point matching area as verification matching point pairs to obtain M verification matching point pairs, wherein M is an integer greater than or equal to 1.

The matching module comprises a matching unit and a dividing unit.

The matching unit is used for matching the feature point set of the first image with the feature point set of the second image to obtain a matching point pair set.

The dividing unit is used for dividing the matching point pair set into a plurality of characteristic point matching areas according to the distance between a plurality of matching point pairs in the matching point pair set.

The present disclosure also provides an electronic device, a readable storage medium, and a computer program product according to embodiments of the present disclosure.

FIG. 6 illustrates a schematic block diagram of an example electronic device 600 that can be used to implement embodiments of the present disclosure. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the disclosure described and/or claimed herein.

As shown in fig. 6, the apparatus 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a Read Only Memory (ROM) 602 or a computer program loaded from a storage unit 608 into a Random Access Memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the device 600 can also be stored. The calculation unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input/output (I/O) interface 605 is also connected to bus 604.

A number of components in the device 600 are connected to the I/O interface 605, including: an input unit 606 such as a keyboard, a mouse, or the like; an output unit 607 such as various types of displays, speakers, and the like; a storage unit 608, such as a magnetic disk, optical disk, or the like; and a communication unit 609 such as a network card, modem, wireless communication transceiver, etc. The communication unit 609 allows the device 600 to exchange information/data with other devices via a computer network such as the internet and/or various telecommunication networks.

The computing unit 601 may be a variety of general and/or special purpose processing components having processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various dedicated Artificial Intelligence (AI) computing chips, various computing units running machine learning model algorithms, a Digital Signal Processor (DSP), and any suitable processor, controller, microcontroller, and so forth. The calculation unit 601 executes the respective methods and processes described above, such as the image processing method. For example, in some embodiments, the image processing method may be implemented as a computer software program tangibly embodied in a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program may be loaded and/or installed onto the device 600 via the ROM 602 and/or the communication unit 609. When the computer program is loaded into the RAM 603 and executed by the computing unit 601, one or more steps of the image processing method described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform the image processing method by any other suitable means (e.g. by means of firmware).

Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuitry, field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), system on a chip (SOCs), complex Programmable Logic Devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, receiving data and instructions from, and transmitting data and instructions to, a storage system, at least one input device, and at least one output device.

Program code for implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These program code may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the program code, when executed by the processor or controller, causes the functions/acts specified in the flowchart and/or block diagram to be performed. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.

In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic, speech, or tactile input.

The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Networks (WANs), and the Internet.

The computer system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client → server relationship to each other.

It should be understood that various forms of the flows shown above may be used, with steps reordered, added, or deleted. For example, the steps described in the present disclosure may be executed in parallel or sequentially or in different orders, and are not limited herein as long as the desired results of the technical solutions disclosed in the present disclosure can be achieved.

The above detailed description should not be construed as limiting the scope of the disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made, depending on design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present disclosure should be included in the scope of protection of the present disclosure.

Claims

1. An image processing method comprising:

selecting N initial matching point pairs from the characteristic point matching regions of the first image and the second image to generate an initial homography matrix, wherein N is an integer larger than 1;

selecting M verification matching point pairs from the feature point matching region to verify the initial homography matrix to obtain a verified internal point set, wherein M is an integer larger than 1;

under the condition that the interior point set meets a preset condition, generating a candidate homography matrix by using the interior point set;

determining the maximum return times according to the expected probability that the M verification matching point pairs are all interior points;

responding to the fact that the current return times are smaller than the maximum return times, returning to the step of generating the initial homography matrix, increasing the return times until the current return times are not smaller than the maximum return times, and obtaining at least one candidate homography matrix; and

and determining a target homography matrix from the at least one candidate homography matrix, and splicing the first image and the second image by using the target homography matrix.

2. The method of claim 1, further comprising:

calculating the proportion between the number of the matching point pairs in the inner point set and the number M of the verification matching point pairs to obtain an inner point proportion;

and under the condition that the internal point ratio meets a preset condition, generating a candidate homography matrix by using the internal point set.

3. The method of claim 2, wherein said determining a maximum number of returns based on an expected probability that said M verified matching point pairs are all inliers comprises:

determining expected probability that M verification matching point pairs selected each time are all the interior points after K times of return according to the interior point proportion, wherein K is the maximum return time; and

and determining the maximum return times K according to the expected probability and the current return times.

4. The method of claim 3, wherein,

determining the expected probability that the M verification matching point pairs selected each time are all the interior points after K times of returning according to the interior point proportion comprises the following steps:

determining the desired probability according to the formula:

P＝1-(1-w ^M ) ^K

wherein, P is the expected probability, w is the internal point proportion, M is the number of the verification matching point pairs, and K is the maximum return times;

the determining the maximum number of return times K according to the expected probability and the current number of return times includes:

determining the maximum number of returns according to the following formula:

5. The method of claim 2, further comprising: determining whether the current return times are less than the maximum return times or not under the condition that the internal point ratio does not meet the preset condition;

and returning to the step of generating the initial homography matrix in response to the current return times being smaller than the maximum return times, wherein the return times are increased progressively.

6. The method of any of claims 2 to 5, wherein the determining a target homography matrix from the at least one candidate homography matrix comprises:

and determining the candidate homography matrix with the highest interior point ratio from the at least one candidate homography matrix as the target homography matrix.

7. The method of any of claims 1 to 6, wherein the feature point matching regions of the first and second images comprise a plurality of feature point matching regions; further comprising:

determining a set of feature points of the first image and a set of feature points of the second image;

matching the feature point set of the first image with the feature point set of the second image to obtain a plurality of feature point matching regions, wherein each feature point matching region comprises at least one matching point pair;

the selecting N initial matching point pairs from the feature point matching regions of the first image and the second image comprises:

selecting N matching point pairs from each characteristic point matching area as initial matching point pairs to obtain the N initial matching point pairs, wherein N is an integer greater than or equal to 1;

the selecting M verification matching point pairs from the feature point matching region to verify the initial homography matrix comprises:

for each feature point matching region, determining the remaining matching point pairs except the initial matching point in the feature point matching region;

and selecting M matching point pairs from the remaining matching point pairs in each feature point matching region as verification matching point pairs to obtain the M verification matching point pairs, wherein M is an integer greater than or equal to 1.

8. The method of claim 7, wherein the matching the feature point set of the first image and the feature point set of the second image to obtain the plurality of feature point matching regions comprises:

matching the feature point set of the first image with the feature point set of the second image to obtain a matching point pair set; and

and dividing the matching point pair set into the plurality of characteristic point matching areas according to the distance between a plurality of matching point pairs in the matching point pair set.

9. An image processing apparatus comprising:

the first generation module is used for selecting N initial matching point pairs from the characteristic point matching regions of the first image and the second image to generate an initial homography matrix, wherein N is an integer larger than 1;

a verification module, configured to select M verification matching point pairs from the feature point matching region to verify the initial homography matrix, so as to obtain a verified internal point set, where M is an integer greater than 1;

the second generation module is used for generating a candidate homography matrix by using the interior point set under the condition that the interior point set meets the preset condition;

the first determining module is used for determining the maximum returning times according to the expected probability that the M verification matching point pairs are all interior points;

the processing module is used for responding to the fact that the current return times are smaller than the maximum return times, returning to the step of generating the initial homography matrix, increasing the return times until the current return times are not smaller than the maximum return times, and obtaining at least one candidate homography matrix; and

and the splicing module is used for determining a target homography matrix from the at least one candidate homography matrix and splicing the first image and the second image by using the target homography matrix.

10. The apparatus of claim 9, further comprising:

the calculation module is used for calculating the proportion between the number of the matching point pairs in the inner point set and the number M of the verification matching point pairs to obtain the inner point ratio;

the second generating module is configured to generate a candidate homography matrix by using the interior point set when the interior point proportion meets a preset condition.

11. The apparatus of claim 10, the first determining module comprising:

a first determining unit, configured to determine, according to the ratio of the interior points, an expected probability that M verification matching point pairs selected each time are all interior points after K times of returning, where K is the maximum number of returning times; and

and the second determining unit is used for determining the maximum return times K according to the expected probability and the current return times.

12. The apparatus of claim 11, wherein,

the first determining unit is configured to determine the expected probability according to the following formula:

P＝1-(1-w ^M ) ^K

the second determining unit is configured to determine the maximum number of returns according to the following formula:

13. The apparatus of claim 10, further comprising:

the second determining module is used for determining whether the current return times are less than the maximum return times or not under the condition that the internal point ratio does not meet the preset condition;

the processing module is used for returning to the first generating module in response to the second determining module determining that the current return times are smaller than the maximum return times, and the return times are increased progressively.

14. The apparatus according to any one of claims 10 to 13, wherein the stitching module is configured to determine, as the target homography matrix, a candidate homography matrix with a highest inlier ratio from the at least one candidate homography matrix.

15. The apparatus of any of claims 9 to 14, wherein the feature point matching regions of the first and second images comprise a plurality of feature point matching regions; the device further comprises:

a third determining module, configured to determine a feature point set of the first image and a feature point set of the second image;

a matching module, configured to match the feature point set of the first image with the feature point set of the second image to obtain multiple feature point matching regions, where each feature point matching region includes at least one matching point pair;

the first generation module is used for selecting N matching point pairs from each feature point matching area as initial matching point pairs to obtain the N initial matching point pairs, wherein N is an integer greater than or equal to 1;

the verification module is used for determining the remaining matching point pairs except the initial matching point in each feature point matching area; and selecting M matching point pairs from the residual matching point pairs in each feature point matching area as verification matching point pairs to obtain the M verification matching point pairs, wherein M is an integer greater than or equal to 1.

16. The apparatus of claim 15, wherein the matching module comprises:

the matching unit is used for matching the characteristic point set of the first image with the characteristic point set of the second image to obtain a matching point pair set; and

a dividing unit, configured to divide the matching point pair set into the multiple feature point matching regions according to distances between multiple matching point pairs in the matching point pair set.

17. An electronic device, comprising:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein, the first and the second end of the pipe are connected with each other,

the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 8.

18. A non-transitory computer readable storage medium having stored thereon computer instructions for causing the computer to perform the method of any one of claims 1 to 8.

19. A computer program product comprising a computer program stored on at least one of a readable storage medium and an electronic device, the computer program when executed by a processor implementing the method according to any one of claims 1 to 8.