TWI752478B - Image processing method and image processing system - Google Patents

Abstract

An image processing method includes analyzing multiple images data based on Illumination-invariant Feature Network (IF-NET) with an image processing device to generate corresponding sets of eigenvector, in which image data includes a first image data related to at least a first feature of the sets of eigenvector, and a second image data related to at least a second feature of the sets of eigenvector; choosing a corresponding first training set of tiles and second training set of tiles from the first image data and second image data with an image processing device based on IF-NET, and computing on both training set of tiles to generate a least one loss value; and adjusting IF-NET based on a least one loss value. An image processing system is also disclosed herein.

Description

Image processing method and image processing system

The present disclosure relates to an image processing method and an image processing system, in particular to an image processing method and an image processing system based on deep learning anti-light interference matching.

Feature matching is widely used in many computer vision fields such as image retrieval, camera positioning, etc. The features of images must be invariant and unique to scale, orientation, viewing angle, and light.

However, in the field of environmental scene similarity comparison, existing feature matching systems or methods are not optimized for light differences and scene field changes, resulting in unsatisfactory matching results.

The present disclosure provides an image processing method, which includes the steps of: analyzing complex image data based on a feature extraction model by an image data processing device to generate a feature vector set corresponding to the image data, wherein the image data includes at least one of the feature vector sets a plurality of first image data related to the first feature and a plurality of second image data related to at least one second feature in the feature vector set; the image data processing device extracts the model from the first image data and the second image by the image data processing device In the data, the corresponding first training image block group and the second training image block group are respectively selected, and the first training image block group and the second training image block group are operated to generate corresponding at least one loss function value; and according to The at least one loss function value adjusts the feature extraction model such that when the image data processing apparatus analyzes the image data based on the adjusted feature extraction model, the degree of matching of the first image data and the second image data increases.

The present disclosure provides an image processing system, which includes an image capturing device and an image data processing device. an image capture device for capturing a plurality of image data; and an image data processing device, coupled to the image capture device, and used for performing a process on the plurality of first image data and the plurality of second image data in the image data based on the feature extraction model A comparison operation is performed, and an image positioning result is output according to the result of the comparison operation; wherein the first image data is related to at least one first feature, the second image data is related to at least one second feature, and the feature extraction model is based on the first image data. The data and the second image data are adjusted by at least one loss function value generated by the operation.

The above-mentioned image processing method and image processing system can improve the accuracy of the feature matching system when the outdoor scene light changes greatly.

The following is a detailed description of the embodiments in conjunction with the accompanying drawings, but the specific embodiments described are only used to explain the present invention, not to limit the present invention, and the description of the structure and operation is not used to limit the order of its execution, any The structure of the recombination of the components, the resulting devices with equal efficacy, are all within the scope of the disclosure of the present invention.

Unless otherwise specified, the terms used throughout the specification and the scope of the patent application generally have the ordinary meaning of each term used in the field, in the content disclosed herein and in the specific content. Certain terms used to describe the present disclosure are discussed below or elsewhere in this specification to provide those skilled in the art with additional guidance in describing the present disclosure.

FIG. 1 is a schematic diagram of an image processing system 100 according to some embodiments. As shown in FIG. 1, the image processing system 100 includes a video The image capturing device 110 and the image data processing device 120 are included. The image data processing device 120 is coupled to the image capturing device 110 . The image capturing device 110 is used for capturing a plurality of image data 300 , such as various photos or patterns, as described in FIG. 3 , and streaming the image data to the image data processing device 120 .

In some embodiments, the image capturing device 110 can be implemented by a camera lens of a smart phone, a camera camera lens, or a program software with a screenshot function.

In some embodiments, the image data processing device 120 may be implemented by a computer system such as a notebook computer, a desktop computer, or the like.

In some embodiments, the image data processing apparatus 120 includes a feature extraction model 130 and an instruction library 140, wherein the feature extraction model 130 is preconfigured in the image data processing apparatus 120, and its architecture is based on IF-Net (Illumination Neural Network Network). ) deep learning network architecture.

In some embodiments, the feature extraction model 130 uses the IF-Net deep learning-based convolutional neural network (Convolutional Neural Network, CNN) to train the feature descriptor (descriptor) generated by the feature extraction model 130, and uses It is learned and trained to find feature descriptors with high adaptability. In some embodiments, the above-mentioned feature descriptor with high adaptability can be used to solve the feature matching error in the case where the light difference of the outdoor scene varies greatly.

In some embodiments, the instruction library 140 stores operation instructions, which are accessed and executed by a processor (not shown) in the image data processing device 120 .

FIG. 2 is a flowchart illustrating the operation of the image processing system 100 according to some embodiments. As shown in FIG. 2, the operation flow of the image processing system 100 includes step S210, step S220, step S230, step S240, step S250, step S260 and step S270. For the convenience of clear description, the image processing method 200 shown in FIG. 2 is described with reference to FIG. 1, but is not limited thereto.

In step S210 , the image capturing device 110 captures the current environment image as the image data 300 (as shown in FIG. 3 ), which is input to the image data processing device 120 . Next, in step S220, the image data processing device 120 loads the feature extraction model 130, and loads an environment scene model in step S230.

In step S240, environmental features are extracted from the environmental scene model and the image data 300, respectively, and in step S250, the image data processing device 120 performs a comparison operation of the similarity of environmental features on the image data 300. Next, in step S260, the image data processing device 120 performs spatial positioning according to the comparison result in step S250, and outputs an image positioning result according to the above-mentioned spatial positioning in step S270.

In some embodiments, in step S240 , the image data processing device 120 analyzes the complex image data 300 based on the feature extraction model 130 to generate a feature vector set corresponding to the image data 300 . The image data 300 includes a plurality of first image data 310 related to at least one first feature vector in the above-mentioned feature vector set, and a complex number of second image data 310 related to at least one second feature vector in the above-mentioned feature vector set. Video material 320. An example will be described below with reference to FIG. 3 .

FIG. 3 is a schematic diagram of classification of image data 300 according to some embodiments. The first image data 310 includes image data under different observation distances or observation angles, and the second image data 320 includes image data observed under different brightness or light, and the second image data 320 is observed under darker light or the brightness is darker The image data of , are filled with slashed lines in the illustration in Figure 3. In other words, in some embodiments, the at least one first eigenvector is related to at least one of an image observation angle and an image observation distance, and the at least one second eigenvector is related to at least one of image brightness, image gamma, and image contrast. one.

Continuing from the above, in the field of environmental feature similarity comparison, the feature extraction model 130 in the prior art is less resistant to interference from light differences and changes in the scene field of view, resulting in unsatisfactory matching results. Therefore, the present disclosure provides an image processing method to adjust the feature extraction model 130 to thereby improve the matching results.

In some embodiments, the above-mentioned image processing method includes using the image data processing device 120 to select a corresponding first training block group and a first training block group from the first image data 310 and the second image data 320 based on the feature extraction model 130 , respectively. Two sets of training tiles. An example will be described below with reference to FIGS. 4A and 4B .

4A and 4B are schematic diagrams of a sample screening mechanism 400 according to some embodiments. Referring to FIG. 4A , in some embodiments, the image data processing device 120 selects the first training image block group and the second training image block group from the first image data 310 and the second image data 320 according to the sample screening mechanism 400 .

In some embodiments, the first image data 310 and the second image data 320 respectively include a base image block (anchor), a plurality of similar image blocks (positive), and a plurality of heterogeneous image blocks (negative). Similar tiles and base tiles have higher matching values, so there is a shorter Euclidean distance between similar tiles and basic tiles. In contrast to homogeneous tiles, heterogeneous tiles have a lower match value with the base tile. Therefore, there is a long Euclidean distance between the heterogeneous tiles and the base tiles.

In some embodiments, as shown in FIG. 4B , the Euclidean distance represents the distance between the blocks output by the feature descriptor generated based on the measurement feature extraction model 130 . For example, the feature vector set of the same type of tile and the feature vector set of the basic tile have a shorter Euclidean distance L1 in the space of the feature descriptor, while the feature vector set of the heterogeneous tile and the feature vector set of the basic tile have a shorter Euclidean distance L1. There is a long Euclidean distance L2.

As mentioned above, in different embodiments, the distance between the output blocks of the feature descriptor generated by the trained feature extraction model 130 will vary. For example, the feature vector set of the same type of tile and the feature vector set of the basic tile will have an Euclidean distance L3 smaller than the Euclidean distance L1 in the space of the feature descriptor, while the feature vector set of the heterogeneous tile and the basic tile will have an Euclidean distance L3. The set of eigenvectors will have an Euclidean distance L4 greater than the Euclidean distance L2. In other words, the features extracted by the image data processing device 120 based on the trained feature extraction model 130 will have a higher matching degree than the original ones.

Therefore, in some embodiments, the step of selecting the first training block group by the image data processing device 120 includes selecting the degree of matching with the base block of the first image data 310 from the plurality of similar blocks of the first image data 310 The lowest at least one similar image block is used as the first training image block group, and from the plural heterogeneous image blocks of the first image data 310, at least one heterogeneous image block with the highest degree of matching with the basic image data 310 is selected as the first training block group. In other words, the first training block group includes the same type of block with the longest Euclidean distance from the base block in the first image data 310 , and the heterogeneous block with the shortest Euclidean distance from the base block in the first image data 310 .

In some other embodiments, the step of selecting the second training block group by the image data processing device 120 includes selecting, from the plurality of similar blocks of the second image data 320, the least matching degree with the base block of the second image data 320 at least one similar block of the second image data 320 is used as the second training block group, and from the plurality of heterogeneous image blocks of the second image data 320, at least one heterogeneous image block with the highest degree of matching with the basic image block of the second image data 320 is selected as the The second set of training tiles. In other words, the second training block group includes similar blocks with the longest Euclidean distance from the base block in the second image data 320 , and heterogeneous blocks with the shortest Euclidean distance from the base block in the second image data 320 .

Through the steps described in the above-mentioned embodiments, the Euclidean distance between the base block and the heterogeneous blocks can be effectively maximized, and the Euclidean distance between the base block and similar blocks can be shortened, so that the feature extraction model 130 can generate more Representative feature descriptors.

L為損失函數值；n為圖塊總數目；w _p與w _n為權重值； d _M(a _i,p _i)代表基礎圖塊與同類圖塊的歐式距離，而d _m(a _i,n _i)則代表基礎圖塊與異類圖塊的歐式距離。 In some embodiments, the image data processing device 120 executes operation instructions in the instruction library 140 to perform operations on the first training image block group and the second training image block group to generate at least one corresponding loss function value. The above step of performing operations includes performing operations on the first training image block group and the second training image block group by using an outlier loss function to generate at least one loss function value. Among them, the outlier loss function is as follows:

L is the loss function value; n is the total number of tiles; w _p and _wn are the weight values; d _M (a _i , p _i ) represents the Euclidean distance between the basic tile and similar tiles, and d _m (a _i , n _i ) represents the Euclidean distance between the base tile and the heterogeneous tile.

The weight value w _p is the average of the Euclidean distances between the same type of image data 300 in the same batch and the base image block, and the weight value w _n is the sum of the Euclidean distances between the heterogeneous image blocks in the same batch and the base image block. average. Such as the following formula:

As mentioned above, if there is noise mixed in the same batch of data groups that are operated on, and the noise is an outlier relative to the training data, it will have an impact on the training performance. Therefore, in some embodiments, the above-mentioned loss function value can reduce the influence of noise when training the IF-Net deep network, so as to more effectively achieve the effect of making the feature extraction model 130 converge during the training process.

In some embodiments, the image processing method includes adjusting the feature extraction model 130 according to the at least one loss function value, so that when the image data processing device 120 analyzes the image data 300 based on the adjusted feature extraction model 130, the first image data 310 and the The degree of matching of the second image data 320 increases.

In some embodiments, the step of adjusting the feature extraction model 130 further includes inputting the first image data 310 and the second image data 320 into the feature extraction model 130 sharing the parameters of the deep neural network model (shared-weight) to generate respectively. corresponding different loss function values, and storing and updating at least one network parameter in the feature extraction model 130 with the different loss function values corresponding to the first image data and the second image data. An example will be described below with reference to FIG. 5 .

FIG. 5 is a flowchart illustrating a step 500 of sharing parameters among groups according to some embodiments. As shown in FIG. 5 , in some embodiments, the image data processing device 120 performs operations on the first image data 310 based on the operation instructions in the feature extraction model 130 accessing the instruction library 140 to generate the first loss function value, and An operation is performed on the second image data 320 to generate a second loss function value. The image data processing device 120 stores the first loss function value and the second loss function value and updates the network parameters in the feature extraction model 130 .

In some embodiments, the above-mentioned method of inputting the first image data 310 and the second image data 320 to the IF-Net that shares the parameters of the deep neural network model (shared-weight) and updating the network parameters at one time can be More effectively, the feature extraction model 130 has the ability to process different types of data.

In some embodiments, the image data processing device 120 performs at least one of the following operations based on the adjusted feature extraction model 130 , such as comparing the first image data 310 and the second image data 320 as in the above step S250 operation, performing the spatial positioning operation in step S270 according to the result of the above comparison operation, and outputting the image positioning result according to the result of the spatial positioning operation.

Based on the above, the image data processing device 120 will improve the matching degree of the image data 300 in step S250 based on the adjusted feature extraction model 130, and the image positioning result in step S270 will be more accurate. The following table (1) is a comparison table of matching degree data generated according to the embodiments in the present disclosure. Table I) HardNet IF-Net Verification Inter 93.85% 93.83% Intra 93.53% 93.57% Retrieval 60.85% 93.57% Matching 41.64% 48.65% In the case of large changes in light or scene, according to the ratio of the number of features extracted from the image data 300 by the feature extraction model 130 to determine the correct rate, it can be clearly seen from Table (1) that the image processing method proposed in this disclosure is related to the The system can improve the matching accuracy.

Although the present disclosure has been disclosed as above in embodiments, it is not intended to limit the present disclosure. Anyone with ordinary knowledge in the art can make various changes and modifications without departing from the spirit and scope of the present disclosure. Therefore, the protection scope of the present disclosure shall be determined by the scope of the appended patent application.

110: Image capture device 120: Video data processing device 130: Feature extraction model 140: Instruction library 200: Image Processing Methods S210, S220, S230, S240, S250, S260, S270: Steps 300: Video data 310: The first video data 320: Second image data 400: Sample Screening Mechanism L1~L4: Euclidean distance 500: Steps of grouping and sharing parameters

FIG. 1 is a schematic diagram of an image processing system according to some embodiments.

FIG. 2 is a flowchart illustrating an operation of an image processing system according to some embodiments.

FIG. 3 is a schematic diagram of classification of image data according to some embodiments.

4A and 4B are schematic diagrams of sample screening mechanisms drawn according to some embodiments.

FIG. 5 is a flow chart of steps of sharing parameters among groups according to some embodiments.

200: Image Processing Methods S210, S220, S230, S240, S250, S260, S270: Steps

Claims (9)

An image processing method, comprising the steps of: analyzing complex image data based on a feature extraction model (IF-NET) by an image data processing device to generate a feature vector set corresponding to the image data, wherein the image data includes and Complex first image data related to at least one first feature vector in the feature vector set and complex second image data related to at least one second feature vector in the feature vector set; by the image data processing device based on the The feature extraction model selects a corresponding first training image block group and a second training image block group respectively from the first image data and the second image data, and selects the first training image block group and the first training image block group. Two training image block groups are operated to generate corresponding at least one loss function value; and the feature extraction model is adjusted according to the at least one loss function value, wherein the step of selecting the first training image block group includes: selecting the first training image block group from the first Among the plurality of similar blocks of image data, at least one similar block with the lowest degree of matching with one of the basic blocks of the first image data is selected as the first training block group; and from the first image data Among the plurality of heterogeneous picture blocks, at least one heterogeneous picture block with the highest matching degree with the basic picture block of the first image data is selected as the first training picture block group. The image processing method of claim 1, wherein when the image data processing device analyzes the image data based on the adjusted feature extraction model, the first image data matches the second image data degree increased. The image processing method of claim 1, further comprising: performing at least one of the following operations on the first image data and the second image data based on the adjusted feature extraction model by the image data processing device performing a comparison operation on the image data; performing a spatial positioning operation according to the result of the comparison operation; and outputting an image positioning result according to the result of the spatial positioning operation. The image processing method of claim 1, wherein the at least one first eigenvector is related to at least one of an image observation angle and an image observation distance, the at least one second eigenvector is related to image brightness, image gamma, and At least one of image contrast. The image processing method according to claim 1, wherein the step of selecting the second training block group comprises: selecting a base image corresponding to the second image data from a plurality of similar blocks of the second image data at least one similar block with the lowest degree of block matching is used as the second training block group; and from the plurality of heterogeneous blocks of the second image data, selecting a block matching the basic block of the second image data At least one heterogeneous image block with the highest degree is used as the second training image block group. The image processing method of claim 1, wherein the step of performing operations on the first training image block group and the second training image block group comprises: using an outlier loss function on the first training image block group and The second training image block group is operated to generate the corresponding at least one loss function. The image processing method according to claim 1, wherein the step of adjusting the feature extraction model further comprises: inputting the first image data and the second image data into the feature of a shared-weight model of a deep neural network extracting a model to generate corresponding different loss function values respectively; and storing and updating at least one network parameter in the feature extraction model for the different loss function values corresponding to the first image data and the second image data. An image processing system includes: an image capture device for capturing a plurality of image data; and an image data processing device coupled to the image capture device, the image data processing device analyzes the image data based on a feature extraction model image data to generate a feature vector set corresponding to the image data; wherein the image data includes a plurality of first image data related to at least one first feature vector in the feature vector set and a plurality of first image data related to the feature vector set At least one second feature vector is associated with a plurality of second image data; the image data processing device executes based on the feature extraction model: From the plurality of similar picture blocks of the first image data, select at least one similar picture block with the lowest degree of matching with a basic picture block of the first image data as the first training picture block group; Among a plurality of heterogeneous blocks of image data, at least one heterogeneous block with the highest degree of matching with the basic block of the first image data is selected as the first training block group; and from the second image data Select a corresponding second training image block group from the , and perform operations on the first training image block group and the second training image block group to generate at least one corresponding loss function value; the feature extraction model is based on the The at least one loss function value generated by the operation of the first image data and the second image data is adjusted. The image processing system of claim 8, wherein the first eigenvector is related to at least one of an image observation angle and an image observation distance, and the at least one second eigenvector is related to image brightness, image gamma, and image contrast at least one of them.

Images