TWI796610B - Image processing method, chip, and electronic device - Google Patents

Abstract

The invention provides an image processing method, the method includes: acquiring a template image collected by an image collection device; acquiring a first set of key points of the template image and a second set of descriptors of the key points; and determining whether any description area of the descriptors exceeds an edge of the template image. The method further includes marking the descriptor whose description area exceeds the edge of the template image as a target descriptor, and mark the key point of the target descriptor is a target key point; and regenerating a target descriptor based on the target key point and a sample image, and updating the template image based on the regenerated target descriptor. The invention also provides a chip and an electronic device. The invention can improve an accuracy of image matching.

Description

Image processing method, chip and electronic device

The present application relates to the technical field of image recognition, in particular to an image processing method, a chip and an electronic device.

The traditional image matching process basically extracts key points first, and then divides a region according to the position of the key points to extract descriptors. However, some images (such as fingerprints) have a relatively small area. When the position of the key point is close to the edge of the image, the area of the descriptor generated according to the key point is incomplete, resulting in incomplete information of the descriptor, which will lead to matching does not work well.

In view of the above problems, the present application proposes an image processing method, a chip and an electronic device to improve the accuracy of image matching.

The first aspect of the present application provides an image processing method, the method comprising: acquiring a sample image collected by an image acquisition device; extracting key points in the sample image to obtain a key point set; Generate a descriptor for each key point of the descriptor to obtain a descriptor set; confirm whether the description area of the descriptor in the descriptor set exceeds the edge of the template image; When the edge, mark the descriptor that describes the region beyond the edge of the template image as the target descriptor, and mark the key points in the target descriptor as target key points; according to The target keypoints and sample images regenerate target descriptors; and update the sample images based on the regenerated target descriptors.

According to some embodiments of the present application, the method further includes: calculating the size of an area in the target descriptor beyond the edge of the template image; comparing whether the calculated area is greater than or equal to a preset area; when the When the calculated area is greater than or equal to the preset area, the target descriptor is regenerated according to the target key points and the sample image.

According to some embodiments of the present application, regenerating the target descriptor according to the target descriptor and the sample image includes: covering the target descriptor with the sample image; obtaining the target key points according to the target descriptor ; Determine the target position of the target key point in the sample image; use the target position as the key point of the sample image, and regenerate the target descriptor according to the key point of the sample image.

According to some embodiments of the present application, the sample image is an image matched with the template image.

According to some embodiments of the present application, the updating the sample image based on the regenerated object descriptor includes: confirming key points in the sample image based on the regenerated object descriptor; an object descriptor in the template image; replacing the matched object descriptor with the regenerated object descriptor map, so as to update the template image.

The second aspect of the present application provides an image processing chip, the chip includes: an acquisition module, used to acquire a sample image collected by an image acquisition device; an extraction module, used to extract key points in the sample image , to obtain the key point set; the generation module is used to generate a descriptor based on each key point in the key point set to obtain the descriptor set; the confirmation module is used to confirm whether there is a description of the descriptor in the descriptor set The area exceeds the edge of the template image; the marking module is used to mark the descriptor whose description area exceeds the edge of the template image as the target description when the description area of the existing descriptor exceeds the edge of the template image character; the generation module is also used to The image regenerates the target descriptor; and an updating module, configured to update the template image based on the regenerated target descriptor.

According to some embodiments of the present application, the generation module is further used to: calculate the size of the area beyond the edge of the template image in the target descriptor; compare whether the calculated area is greater than or equal to the preset area ; When the calculated area is greater than or equal to the preset area, regenerate the target descriptor according to the target key points and the sample image.

According to some embodiments of the present application, the generation module is further configured to: use the sample image to cover the target descriptor; obtain the target key points according to the target descriptor; determine the target key points A target position in the sample image; using the target position as a key point of the sample image, and regenerating a target descriptor according to the key point of the sample image.

According to some embodiments of the present application, the update module is further configured to: confirm the key points in the sample image based on the regenerated target descriptor; match the target in the sample image according to the confirmed key points Descriptor: use the regenerated target descriptor to replace the matched target descriptor, and update the template image.

The third aspect of the present application provides an electronic device, the electronic device includes: a processor; and a memory, wherein a plurality of program modules are stored in the memory, and the plurality of program modules are loaded by the processor And execute the image processing method as mentioned above.

In the image processing method, chip, and electronic device of the present application, by determining the key points at the edge of the image in the sample image, and updating the descriptors corresponding to the key points through the sample image, the key points at the edge of the image can be increased. The possibility of the point being matched, thereby improving the accuracy of the subsequent matching of the image.

A:Template image

B: sample image

A1~A9, B1~B3: key points

D _A1 ~ D _A9 、 D _B1 ~ D _B3 : Descriptor

10: Electronic device

11: Memory

12: Processor

13: Computer program

20: Image processing chip

201: Get the module

202: Extract module

203: Generate modules

204: Confirm the module

205: Marking Module

206:Update module

FIG. 1 is a schematic flowchart of an image processing method provided by an embodiment of the present application.

FIG. 2 is a schematic diagram of key points in a sample image A provided by an embodiment of the present application.

Fig. 3 is a schematic diagram of a descriptor generated according to key points in a sample image A provided by an embodiment of the present application.

FIG. 4 is a schematic diagram of a sample image B covering the sample image A provided by an embodiment of the present application.

FIG. 5 is a schematic diagram of an image processing chip provided by an embodiment of the present application.

FIG. 6 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present application.

In order to more clearly understand the above objects, features and advantages of the present application, the present application will be described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that, in the case of no conflict, the embodiments of the present application and the features in the embodiments can be combined with each other.

Many specific details are set forth in the following description to facilitate a full understanding of the application, and the described embodiments are only a part of the embodiments of the application, rather than all the embodiments.

Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the technical field to which this application belongs. The terms used herein in the specification of the application are only for the purpose of describing specific embodiments, and are not intended to limit the application.

Please refer to FIG. 1 . FIG. 1 is a schematic flowchart of an image processing method provided by an embodiment of the present application. According to different requirements, the order of the steps in the flowchart can be changed, and some steps can be omitted. For ease of description, only the parts related to the embodiment of the present application are shown. As shown in Fig. 1, the image processing method includes the following steps.

Step S1. Obtain a template image collected by an image collection device.

In one embodiment, the image collection device may be a fingerprint collection device, and the fingerprint collection device may be installed in a smart terminal such as a mobile phone, a tablet computer, or an industrial device. To collect user fingerprints for identity authentication. The fingerprint collection device can also be set in the attendance device, and is used to collect user fingerprints for attendance and so on. The fingerprint collection device may be a device that collects fingerprint images by means of optical fingerprint collection technology, capacitive sensor fingerprint collection technology, ultrasonic fingerprint collection technology, or electromagnetic wave fingerprint collection technology.

In other implementation manners, the image acquisition device may also be a camera. The template image may be an image of a person, an animal, a scene, etc. captured by the camera.

In this implementation manner, when the images acquired by the image acquisition device subsequently during use need to be matched, the template images may be used to match the subsequently acquired images. However, because the sample image may have incomplete descriptor regions of key points at the edges, the matching effect is not good. In order to improve the matching accuracy, the sample image may be processed by the image processing method provided in the present application to complement the descriptor area of the key point at the edge of the sample image.

Step S2, extracting key points in the sample image to obtain a key point set.

The key points represent points associated with features or characteristics of the template image, and may be referred to as interest points or feature points. For example, key points may be located at the outlines of objects in the template image. For example, the key points may be minutiae points in the fingerprint image, for example, endpoints of fingerprint lines, bifurcation points, bifurcation points, isolated points, ring points, central points of fingerprints, triangular points, and the like. The key points may also be the left eye area, eye area, nose area, left mouth corner area, right mouth corner area, etc. in the face image.

In this embodiment, the algorithm for extracting key points in the sample image includes Harris corner detection algorithm, scale-invariant feature transform (Scale-invariant feature transform, SIFT) feature detection algorithm, accelerated robustness Feature (Speeded Up Robust Features, SURF) feature detection algorithm, ORB (Oriented FAST and Rotated BRIEF) feature detection algorithm, etc.

For example, for the sample image A shown in Figure 2, the key points A1 to A9 in the sample image A are extracted to obtain the key point set {A1, A2, A3, A4, A5, A6, A7 , A8, A9}.

It should be noted that, before step S2, the image processing method may further include: a step of preprocessing the template image. Specifically, the preprocessing of the sample image includes gray-scaling the sample image, binarizing the feature to be recognized, and the like.

Step S3, generating a descriptor based on each key point in the key point set to obtain a descriptor set.

In this embodiment, appropriate key points are generated according to the sample image, and are used to create corresponding descriptors to characterize the image. In one embodiment, the descriptor D may be a SIFT-type descriptor. Specifically, for each key point, the local area around the key point is rotated clockwise to ensure its rotation invariance. In the rotated area, the 16×16 rectangular window with the key point position as the center is evenly divided into 16 sub-areas, and the descriptor window is 16 4×4 sub-blocks. Calculate the cumulative gradient value of eight directions n*π/4 (n=0,1,...7) on each sub-block. A total of 128 values are obtained for 16 sub-blocks. This 1×128 vector is defined as A descriptor D of a keypoint.

Furthermore, even though SIFT-type descriptors are referred to in the discussed examples, similar considerations apply to employing different types of descriptors (e.g., accelerated robust features (SURF) and histograms of oriented gradients (HOG) or possibly other type) case. Additionally, in other embodiments, in addition to descriptors related to gradients, different types of descriptors may be considered, including, for example, those related to hue gradients, saturation gradients, or even colors (including lightness, saturation, and hue). degree) gradient-related information.

In this embodiment, the descriptor may be implemented as a multi-dimensional descriptor (eg, 128 dimensions) for a specific radius (support region) around a given keypoint. For example, the specific radius is set to 15 primitives.

For example image A shown in FIG. 3 , descriptors can be generated based on each key point in the key point set {A1, A2, A3, A4, A5, A6, A7, A8, A9}. For example, based on the key point A1, the descriptor D _A1 can be generated; based on the key point A2, the descriptor D _A2 can be generated; and so on, based on the key point A9, the descriptor D _A9 can be generated. The resulting descriptor set is {D _A1 , D _A2 , D _A3 , D _A4 , D _A5 , D _A6 , D _A7 , D _A8 , D _A9 }.

Step S4 , confirming whether the description area of any descriptor in the descriptor set exceeds the edge of the template image. When the description area with descriptors exceeds the edge of the sample image, the process enters step S5; when the description area without descriptors exceeds the edge of the sample image, the process ends.

In order to ensure the separability of the descriptor of the key point, the description area of the descriptor will be selected as large as possible. In order to ensure the number of key points in the sample image, the key points may be distributed in the edge area of the sample image. This may cause the description area of the descriptor to exceed the edge of the image. For example, in the sample image A shown in FIG. 3 , the descriptors D _A1 , D _A2 , D _A3 , D _A8 , and D _A9 are all beyond the edge of the sample image.

Step S5 , marking a descriptor whose description area exceeds the edge of the sample image as an object descriptor, and marking a key point in the object descriptor as an object key point.

Due to the incomplete information of the descriptors beyond the edge of the template image, poor matching effect may occur in the subsequent image matching process. Therefore, in order to solve the problem of poor matching effect in this case, it is necessary to complete the descriptors beyond the edge of the template image to improve the accuracy of image matching.

In an implementation manner, after marking the object descriptor, it may first be determined whether the object descriptor needs to be updated. When the description area of the target descriptor is relatively small beyond the edge of the sample image, the description area of the target descriptor can accurately describe the sample image without updating the target descriptor; When the description area of the target descriptor exceeds the area of the edge of the sample image is relatively large, the target descriptor cannot accurately describe the sample image, and the target descriptor needs to be updated.

Specifically, calculating the size of an area beyond the edge of the sample image in the description area of the target descriptor, and comparing whether the calculated area is greater than or equal to a preset area. When the calculation When the calculated area is greater than or equal to the preset area, it is confirmed that the target descriptor needs to be updated, and the process enters step S6; when the calculated area is smaller than the preset area, it is confirmed that the target descriptor does not need to be updated, The process ends.

For example, in the sample image A shown in Figure 3, the areas beyond the edge of the sample image in the description areas of the descriptors D _A1 and D _A9 are small, and there is no need to update the descriptors D _A1 and D _A9 ; In the description areas of the descriptors D _A2 , D _A3 and D _A8 , the areas beyond the edge of the template image are relatively large, and the descriptors D _A2 , D _A3 and D _A8 need to be updated.

Step S6, regenerate the target descriptor according to the target key points and the sample image.

In this implementation manner, the sample image is an image that matches the template image, and the sample image is complete and of good quality. For example, the similarity between the sample image and the sample image is greater than a first preset value, or the proportion of the key points in the sample image matching the key points in the sample image is greater than the first preset value. Two default values. It should be noted that the sample image has complete descriptor information of the key points located at the edge in the sample image, that is, the region described by the descriptor of the edge of the sample image must be a corresponding region in the sample image. complete.

It can be understood that, in this embodiment, the sample image may be an image pre-stored in the electronic device, or an image collected by the image acquisition device.

Specifically, regenerating the target descriptor according to the target key points and the sample image includes: covering the target descriptor with the sample image; obtaining the target key points according to the target descriptor; determining the target A target position of the key point in the sample image; using the target position as a key point of the sample image, and regenerating a target descriptor according to the key point of the sample image.

It should be noted that when the sample image covers the target descriptor, the range The key point on the edge in this image A falls within the overlapping area with the sample image B. As shown in Figure 4, the target key points A2, A3 and A8 in the sample image all fall within the overlapping area between the sample image and the sample image, that is, the sample image B covered: determine the position of the target key point in the sample image B, and use the determined position as the key point of the sample image B, such as key points B1, B2 and B3. In the sample image, target descriptors DB1 , DB2 and DB3 are respectively generated with the key points B1 , B2 and B3 .

Step S7, updating the template image based on the regenerated target descriptor.

In this implementation manner, the target descriptor in the template image may be replaced with a regenerated target descriptor. Specifically, confirm the key points in the sample image based on the regenerated target descriptor; match the target descriptor in the sample image according to the confirmed key point; replace the matching with the regenerated target descriptor map target descriptor to update the template image.

It should be noted that the key points of the regenerated target descriptor correspond to the key points of the target descriptor in the sample image. For example, as shown in Figure 4, based on the regenerated target descriptor DB1, confirm the key point A2 in the sample image, match the target descriptor DA2 in the sample image according to the confirmed key point A2, and finally use The regenerated object descriptor DB1 replaces the object descriptor DA2 in the sample image. Similarly, the object descriptor DA3 in the sample image is replaced with the regenerated object descriptor DB2, and the object descriptor DA8 in the sample image is replaced with the regenerated object descriptor DB3.

It should be noted that, when updating the template image based on the regenerated target descriptor, it is also possible to increase the integrity of the image area and the control of image quality to ensure the correctness of the update.

1-4 describe the image processing of the present application in detail, and the efficiency and accuracy of the image processing can be improved through the method. The functional modules of the software chip and the architecture of the hardware device for realizing the image processing will be introduced below with reference to FIG. 5 and FIG. 6 . It should be understood that the embodiments are only for illustration, and are not limited by the structure in terms of the scope of the patent application.

FIG. 5 is a structural diagram of an image processing chip provided by an embodiment of the present application.

In some implementations, the image processing chip 20 may include a plurality of functional modules composed of program code segments to realize image processing functions.

Referring to FIG. 5 , in this embodiment, the image processing chip 20 can be divided into a plurality of functional modules according to the functions it performs, and each functional module is used to execute the steps in the corresponding embodiment in FIG. 1 , so as to Realize the function of image processing. In this embodiment, the functional modules of the image processing chip 20 include: an acquisition module 201 , an extraction module 202 , a generation module 203 , a confirmation module 204 , a marking module 205 and an updating module 206 . The functions of each functional module will be described in detail in the following embodiments.

The obtaining module 201 is used to obtain the sample image collected by the image collection device.

The extraction module 202 is used to extract key points in the sample image to obtain a key point set.

In one embodiment, the key points may be minutiae points in the fingerprint image, for example, endpoints of fingerprint lines, bifurcation points, bifurcation points, isolated points, ring points, central points of fingerprints, triangle points, etc. . The key points may also be the left eye area, eye area, nose area, left mouth corner area, right mouth corner area, etc. in the face image.

In this embodiment, the algorithm for extracting key points in the sample image includes Harris corner detection algorithm, scale-invariant feature transform (Scale-invariant feature transform, SIFT) feature detection algorithm, accelerated robustness Feature (Speeded Up Robust Features, SURF) feature detection algorithm, ORB (Oriented FAST and Rotated BRIEF) feature detection algorithm, etc.

The generating module 203 is configured to generate a descriptor based on each key point in the key point set to obtain a descriptor set.

The confirming module 204 is used for confirming whether the description area of any descriptor in the descriptor set exceeds the edge of the sample image.

The labeling module 205 is used to label descriptors whose description areas exceed the edge of the sample image as target descriptors, and mark keypoints in the target descriptors as target keypoints.

In one embodiment, after marking the target descriptor, the marking module 205 is further configured to determine whether the target descriptor needs to be updated. When the description area of the target descriptor is relatively small beyond the edge of the sample image, the description area of the target descriptor can accurately describe the sample image without updating the target descriptor; When the description area of the target descriptor exceeds the area of the edge of the sample image is relatively large, the target descriptor cannot accurately describe the sample image, and the target descriptor needs to be updated.

The generating module 204 is also used to regenerate object descriptors according to the object key points and sample images.

Specifically, the generating module 204 is used to cover the target descriptor with the sample image; obtain the target key point according to the target descriptor; determine that the target key point is in the sample image the target position; using the target position as the key point of the sample image, and regenerating the target descriptor according to the key point of the sample image.

The updating module 206 is used for updating the template image based on the regenerated object descriptor.

In this implementation manner, the target descriptor in the template image may be replaced with a regenerated target descriptor. Specifically, confirm the key points in the sample image based on the regenerated target descriptor; match the target descriptor in the sample image according to the confirmed key point; replace the matching with the regenerated target descriptor map target descriptor to update the template image.

FIG. 6 is a schematic diagram of functional modules of an electronic device provided by an embodiment of the present application. The electronic device 10 includes a memory 11, a processor 12 and stored in the memory 11 and can be A computer program 13 running on the processor 12, such as an image processing program.

In this embodiment, the electronic device 10 may be, but not limited to, a smart phone, a tablet computer, a smart industrial device, a fingerprint time attendance machine, and the like.

When the processor 12 executes the computer program 13, it implements the steps of image processing in the above method embodiment, which is used to identify the fingerprint image collected by the fingerprint collection unit 11 . Alternatively, the processor 13 executes the computer program 13 to implement the functions of the modules/units in the above chip embodiment.

Exemplarily, the computer program 13 can be divided into one or more modules/units, and the one or more modules/units are stored in the memory 11 and executed by the processor 12 , to complete this application. The one or more modules/units may be a series of computer program instruction segments capable of accomplishing specific functions, and the instruction segments are used to describe the execution process of the computer program 13 in the electronic device 10 . For example, the computer program 13 can be divided into modules 201-206 in FIG. 5 .

Those skilled in the art can understand that the schematic diagram 6 is only an example of the electronic device 10, and does not constitute a limitation to the electronic device 10. The electronic device 10 may include more or less components than those shown in the figure, or combine certain components. , or different components, for example, the electronic device 10 may also include input and output devices and the like.

The so-called processor 12 may be a central processing unit (Central Processing Unit, CPU), and may also include other general-purpose processors, digital signal processors (Digital Signal Processor, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC) , Ready-made programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc., the processor 12 is the control center of the electronic device 10, and uses various interfaces and lines to connect various parts of the entire electronic device 10 .

The memory 11 can be used to store the computer program 13 and/or module/unit, and the processor 12 runs or executes the computer program and/or module/unit stored in the memory 11, and Various functions of the electronic device 10 are realized by calling the data stored in the memory 11 . The memory 11 may include an external storage medium or a memory. In addition, memory 11 can include high-speed random access memory, and can also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart memory card (Smart Media Card, SMC), secure digital (Secure Digital, SD) card, flash memory card (Flash Card), at least one disk memory component, flash memory device, or other volatile solid state memory components.

If the integrated modules/units of the electronic device 10 are realized in the form of software function units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above-mentioned embodiments in the present application can also be completed by instructing related hardware through computer programs, and the computer programs can be stored in a computer-readable storage medium. When the computer program is executed by the processor, it can realize the steps of the above-mentioned various method embodiments. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable Excludes electrical carrier signals and telecommunication signals.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application without limitation. Although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be Make modifications or equivalent replacements without departing from the spirit and scope of the technical solutions of the present application.

Claims (9)

An image processing method applied to an electronic device, the method comprising: acquiring a sample image collected by an image acquisition device; extracting key points in the sample image to obtain a key point set; based on each key point set in the key point set A key point generates a descriptor to obtain a descriptor set; confirm whether the description area of the descriptor in the descriptor set exceeds the edge of the template image; when there is a description area of the descriptor beyond the edge of the template image , mark the descriptor whose description area exceeds the edge of the sample image as the target descriptor, and mark the key points in the target descriptor as target key points; regenerate the target description according to the target key points and the sample image and updating the sample image based on the regenerated target descriptor, including: confirming key points in the sample image based on the regenerated target descriptor; matching in the sample image according to the confirmed key points the target descriptor; using the regenerated target descriptor to replace the matched target descriptor, and update the template image. The image processing method according to claim 1, the method further includes: calculating the size of an area in the target descriptor beyond the edge of the template image; comparing whether the calculated area is greater than or equal to a preset area; When the calculated area is greater than or equal to the preset area, regenerate the target descriptor according to the target key points and the sample image. According to the image processing method described in Claim 2, regenerating the object descriptor according to the object descriptor and the sample image includes: covering the object descriptor with the sample image; obtaining the object descriptor according to the object descriptor key points of the target; Determine the target position of the target key point in the sample image; use the target position as the key point of the sample image, and regenerate the target descriptor according to the key point of the sample image. In the image processing method according to claim 1, the sample image is an image matched with the template image. An image processing chip, the chip includes: an acquisition module, used to acquire a sample image collected by an image acquisition device; an extraction module, used to extract key points in the sample image, and obtain a key point set; generate A module for generating descriptors based on each key point in the key point set to obtain a descriptor set; a confirmation module for confirming whether the description area of any descriptor in the descriptor set exceeds the template image The edge of the mark module, used to mark the descriptor whose description area exceeds the edge of the sample image as the target descriptor when the description area of the existing descriptor exceeds the edge of the sample image, and mark the target The key points in the descriptor are target key points; the generation module is also used to regenerate the target descriptor according to the target key points and the sample image; and the update module is used to regenerate the target descriptor based on Update said template image. The image processing chip as described in claim 5, the generating module is also used to: calculate the size of the area beyond the edge of the template image in the target descriptor; compare whether the calculated area is greater than or equal to A preset area: when the calculated area is greater than or equal to the preset area, regenerate the target descriptor according to the target key points and the sample image. In the image processing chip described in claim 5, the generation module is also used for: Use the sample image to cover the target descriptor; obtain the target key point according to the target descriptor; determine the target position of the target key point in the sample image; use the target position as the target position key points of the sample image, and regenerate the target descriptor according to the key points of the sample image. In the image processing chip as described in claim 5, the update module is further used to: confirm the key points in the template image based on the regenerated target descriptor; match the key points in the template image according to the confirmed key points The target descriptor in ; use the regenerated target descriptor to replace the matching target descriptor, and update the template image. An electronic device, comprising: a processor; and a memory, wherein a plurality of program modules are stored in the memory, and the plurality of program modules are loaded and executed by the processor as in claim 1 To the image processing method described in any one of Claim 4.

Images