WO2019148442A1

WO2019148442A1 - Template optimization method and device, electronic device and computer program product

Info

Publication number: WO2019148442A1
Application number: PCT/CN2018/075055
Authority: WO
Inventors: 王洛威; 王恺; 廉士国
Original assignee: 深圳前海达闼云端智能科技有限公司
Priority date: 2018-02-02
Filing date: 2018-02-02
Publication date: 2019-08-08
Also published as: CN108369731A; CN108369731B

Abstract

A template optimization method and device, an electronic device and a computer program product, the method comprising: acquiring an augmented reality (AR) image template; according to image features comprised in the image template, determining that the quality of the image template does not meet a first standard; and adding an enhanced image feature to said image template. In the present application, it is possible to optimize an AR image template, thus increasing the correct matching rate for image templates and increasing the efficiency of feature matching.

Description

Template optimization method, device, electronic device and computer program product

Technical field

The present application relates to the field of augmented reality technologies, and in particular, to a template optimization method, apparatus, electronic device, and computer program product.

Background technique

AR (Augmented Reality) technology is a technology that increases the user's perception of the real world through the information provided by the computer system, and superimposes the computer-generated virtual object, scene or system prompt information into the real scene, thereby realizing the reality. Enhanced." The technology collects real-world images in real time and sets the position of the virtual object according to the position of the camera relative to the real object, so that the virtual object can be correctly superimposed on the real object.

The commonly used AR technology mostly uses flat objects (such as cards, book covers) as targets for superposition, and these targets for superposition are AR image templates. First, the planar image of the template (such as the image of the cover of the book) is pre-stored, the video frame is recognized in real time at runtime, and the image matching algorithm is used to match the image template, and the pose information of the camera is calculated according to the matching result. The image matching refers to identifying a point of the same name between two or more images by a certain matching algorithm. For example, in the two-dimensional image matching, by comparing the correlation coefficients of the same size window in the target area and the search area, the search area is taken. The center point of the window corresponding to the largest correlation coefficient is used as the point of the same name. The essence is to use the best search problem of matching criteria under the condition of primitive similarity. Image matching can be mainly divided into gray-based matching and feature-based matching. When natural images are used as templates, matching is usually based on image features.

Feature matching refers to an algorithm that performs feature matching by extracting features (points, lines, faces, etc.) of two or more images, and then using the described parameters to perform matching. Features processed by feature-based matching typically include features such as color features, texture features, shape features, spatial location features, and the like. Feature matching Firstly, the image is preprocessed to extract its high-level features, and then the matching correspondence between the two images is established. The commonly used feature primitives have some features, edge features and regional features. Feature matching requires many mathematical operations such as matrix operations, gradient solutions, and Fourier transforms and Taylor expansion. Commonly used feature extraction and matching methods are: statistical methods, geometric methods, model methods, signal processing methods, boundary feature methods, Fourier shape description methods, geometric parameter methods, shape invariant moment methods, and so on. In addition, the feature matching algorithm is mainly composed of the following four elements: (1) Feature space, which is composed of image features participating in matching. Selecting features can improve matching performance, reduce search space, reduce noise, etc. The influence of deterministic factors on the matching algorithm; (2) the similarity measure, which measure is used to determine the similarity between the features to be matched, which is usually defined as a form of a cost function or a distance function; 3) Image matching transformation type, image geometric transformation is used to solve geometric position difference between two images, including rigid body transformation, affine transformation, projection transformation, polynomial transformation, etc.; (4) search strategy, search strategy is appropriate The search method finds the optimal estimation of the transformation parameters such as translation and rotation in the search space, so that the similarity between the transformed images is the largest. After the feature matching is successful, the camera pose is obtained by solving the template object and the homography matrix of the target in the video frame, so as to set the corresponding virtual object pose.

The shortcomings of the prior art are:

When the quality of the AR image template is poor, the corresponding pose calculation and virtual object superposition cannot be correctly performed according to the matching of the image templates with the features of the real image.

Summary of the invention

The embodiment of the present application proposes a template optimization method, device, device and computer program product, which are mainly used to optimize some AR image templates so as to correctly perform corresponding pose calculation and virtualization according to the matching of the image templates with the features of the real image. Object overlay.

In one aspect, the embodiment of the present application provides a template optimization method, where the method includes: acquiring an AR image template; determining, according to image features included in the image template, that the image template quality does not meet the first standard. Adding enhanced image features to the image template.

In another aspect, the embodiment of the present application provides a template optimization apparatus, where the apparatus includes: an obtaining module, configured to acquire an AR image template; and a quality determining module, configured to: according to the image included in the image template The feature determines that the image template quality does not satisfy the first criterion; and the optimization module is configured to add an enhanced image feature to the image template.

In another aspect, an embodiment of the present application provides an electronic device, including: a memory, one or more processors; and one or more modules, the one or more modules being Stored in the memory and configured to be executed by the one or more processors, the one or more modules including instructions for performing the various steps of the above methods.

In another aspect, embodiments of the present application provide a computer program product for use in conjunction with an electronic device, the computer program product comprising a computer program embedded in a computer readable storage medium, the computer program comprising An instruction to cause the electronic device to perform the various steps in the above methods.

The beneficial effects of the embodiments of the present application are as follows:

In the present application, the AR image template can be optimized, the image template matching accuracy rate is improved, and the feature matching efficiency is improved.

DRAWINGS

Specific embodiments of the present application will be described below with reference to the accompanying drawings, in which:

1 is a schematic flow chart of a template optimization method in Embodiment 1 of the present application;

2 is a schematic structural diagram of a template optimization apparatus in Embodiment 2 of the present application;

FIG. 3 is a schematic structural diagram of an electronic device in Embodiment 3 of the present application.

Detailed ways

The exemplary embodiments of the present application are further described in detail below with reference to the accompanying drawings, in which the embodiments described are only a part of the embodiments of the present application, but not all embodiments. An exhaustive example. And in the case of no conflict, the features in the embodiments and the embodiments in the description can be combined with each other.

The inventor noticed during the invention that when some AR image templates contain image features that are not robust or too small, they will not correctly match the image poses with the features of the real images to complete the corresponding pose calculations and virtual objects. Superimposed.

In view of the above-mentioned deficiencies, the present application provides a template optimization method for optimizing an AR image template by adding an enhanced image feature when determining an image template quality according to an image feature of the AR image template. In the present application, the AR image template can be optimized, the image template matching accuracy rate is improved, and the feature matching efficiency is improved.

The essence of the technical solution of the embodiment of the present invention is further clarified by specific examples below.

Embodiment 1:

FIG. 1 is a schematic flowchart of a template optimization method according to Embodiment 1 of the present application. As shown in FIG. 1, the template optimization method includes:

Step 101: Acquire an AR image template.

Step 102: Determine, according to image features included in the image template, that the image template quality does not meet the first criterion;

Step 103, adding an enhanced image feature to the image template.

In step 101, an AR image template is obtained, that is, an AR image template that is input by the user and uploaded by the terminal or retrieved by the cloud server, and the AR image template is an overlay template when the virtual image is superimposed using the AR technology, and is usually Printed on a flat object.

In step 102, it is determined whether the image template quality satisfies the first criterion according to the image features included in the image template, that is, whether the AR image template is of poor quality needs to be optimized.

The image feature is the basis of AR image template matching. The image feature determination method may be different in different image matching algorithms, and the number and location of image features may also be different. When applying various image matching algorithms, the image template quality can be evaluated according to the image features contained in the image template to determine whether it meets the preset first criterion, and the image features specifically refer to various states or information of the image features. , which may include the number of image features, the density of image features (the number of image features per unit area or a certain amount of pixels), the degree of aggregation of image features (whether the image features are scattered or gathered), or the significantness of image features. A combination of one or more of the degree (under certain image recognition or matching algorithms, whether each image feature is easy to recognize), etc., when the image quality is judged by various image features, there are different standards, that is, The first criterion may be one or a combination of a number threshold of an image feature, a density threshold, an aggregation degree threshold, or a significance degree threshold. If the image feature reaches a preset standard, the image template is considered to have rich image features, good robustness, and is suitable for feature matching in the AR; if the preset standard cannot be achieved, the subsequent steps are added to the image. Features for template optimization.

In some embodiments, determining that the image template quality does not meet the first criterion is based on a density and/or a degree of aggregation of image features included in the image template.

Taking the density of image features as an example, the density threshold of the image features may be defined, that is, the first criterion is N image features per thousand pixels (N may be an integer or a non-integer), according to the number of pixels in the current image template. And the number of image features n determines whether the quality of the image template satisfies the first criterion. When the number of pixels per thousand pixels in the image template is greater than N, it is considered that the image template quality satisfies the first criterion.

Taking the degree of aggregation of image features as an example, the threshold C of the degree of aggregation of image features may be defined, that is, the first criterion is C, and the calculation formula of the degree of aggregation of image features is as follows:

In the above formula, c represents the degree of aggregation of image features in the image template, ∑d is the sum of the Euclidean distances between the two image features in the image template, n is the number of image features in the image template, and s is the area of the image template. . When c ≥ C, the quality of the image template is considered to satisfy the first criterion. It should be noted that the calculation of the degree of aggregation of image features is not limited to the above algorithm, and commonly used algorithms capable of characterizing the degree of scatter aggregation can be used, and correspondingly need to set different first standards.

In addition, the threshold of the density and the degree of aggregation of the image features can be set simultaneously as the first criterion, and the image template is regarded as satisfying the first criterion when it reaches two thresholds at the same time. A relatively simple algorithm can be employed to quickly determine whether an image template needs to be optimized when performing quality assessments on image templates according to the first standard.

Step 103, adding an enhanced image feature to the image template.

When the quality of the image template does not meet the preset criteria, an image feature may be actively added to the image template, for example, adding image features of points, lines, and faces having special colors, textures, shapes, etc., and adding new enhanced image features into the image template. After the image template, the image template matching accuracy rate can be improved to a certain extent, and the feature matching efficiency is improved. Of course, geometric patterns such as stars and rectangles can be selected as enhanced image features. These pattern features are more obvious, resulting in sensor noise, image changes caused by changes in viewing angles, target movement and deformation, illumination or environmental changes. Under the influence of image changes, the similarity of these features is not high, and it is not easy to be affected by various effects. It can be easily distinguished in the feature matching process, further improving the matching accuracy of image templates and improving feature matching efficiency.

In some embodiments, before the step 103, the image template is further divided into a plurality of image regions; the step 103 is specifically to add an enhanced image feature to the quality difference region of the image template, the quality The difference area is an image area whose image quality determined according to image characteristics included in each image area does not satisfy the second standard.

That is, after it is determined in step 102 that the image template quality does not satisfy the preset first criterion, the image template may be divided into regions, for example, divided into two regions, four regions, and divided into nine regions by the "#" type. Or divided into multiple regions in other random or non-random ways.

After the regions are divided, the image quality of each image region is evaluated, and the image quality of each image region is determined to satisfy the second standard by using various states or information similar to the image features included in each region in step 102, and the image quality is not satisfied. The image area of the second standard is listed as a quality difference area, and an enhanced image feature is added in the quality difference area. The image features and the corresponding second standard are similar to those in step 102 and will not be described again. It can be appreciated that the addition of enhanced image features to areas of relatively poor image quality can significantly improve the overall image quality of the image template.

In some embodiments, on the basis of the above-described embodiment of adding an enhanced image feature to the qualitative difference region, when there is a high quality region and determining that the high quality region can be divided according to the resolution of the high quality region, The high quality area is divided into a plurality of areas, and the high quality area is an area in which the image quality determined according to the image features included in each image area satisfies the second standard.

After the image template is divided into a plurality of image regions and the image quality of each image region is evaluated, whether the image quality of each image region is satisfied is determined by using various states or information according to the image features included in each region in step 102. In the second standard, the image area that satisfies the second standard is listed as a high-quality area, and the enhanced image feature is not added to the high-quality area in the divided image area, but is further divided, and the area divided by the high-quality area is included. In the quality difference area, the enhanced image feature is added to the quality difference area. If the classified area is still a high quality area, further division is continued until the high quality areas cannot continue to be divided. The judging whether the high-quality area can continue to be divided may be determined according to the resolution of the current area. When the resolution is less than a certain threshold, the high-quality area may not be divided, because the partitioned area is likely to be a quality difference area, but Adding enhanced image features does not significantly indicate the quality of the image template. The image features and corresponding second standards in this embodiment are similar to those in step 102, and are not described again. It can be understood that adding enhanced image features to regions with relatively poor image quality can significantly improve the image quality of the image template as a whole. When some image regions are high-quality regions, they may still contain poor quality parts, so the quality is excellent. Further segmentation of the regions to identify these regions and the addition of enhanced image features can enhance the quality of the image template more specifically.

In some embodiments, the second criterion is determined based on a mean threshold and a variance threshold of Harris response values corresponding to all image features included in the image region.

When further performing quality evaluation on a divided image region to determine whether it is a poor quality region or a high quality region, a relatively complicated evaluation standard may be adopted, for example, a Harris response value corresponding to each image feature in the image region may be calculated, and It is calculated whether the average value and the variance of the corresponding Harris response values of all image features in the image region satisfy a preset threshold criterion, that is, a second criterion, to determine that each image region is a quality region or a quality difference region.

Harris corner detection is a feature point detection method, which uses the gray point difference of adjacent pixels to determine whether it is a corner point, an edge, and a smooth area. The basic principle is to calculate the gray value in the image by using a moving window. The main flow includes converting images into grayscale images, calculating differential images, Gaussian smoothing, calculating local extrema, confirming corners, and so on. The calculated response value of each image feature can represent the degree of saliency of each image feature to a certain extent. In this embodiment, the saliency threshold is used as the second criterion to determine whether each image feature is significant, and the degree of significance is significant. Whether or not the evaluation is similar, it is possible to determine whether it is a quality region or a quality difference region according to the degree of significance of the image feature in the image region, so as to further determine whether it is necessary to add an enhanced image feature or whether further division is required.

In this embodiment, when the image processing operation is involved, when the algorithm has a large amount of computation, the cloud server can complete the correlation calculation, and feed the calculation result to the front end of each of the initiated AR image template optimization requests.

In the present application, the AR image template can be optimized, the image template matching accuracy rate is improved, and the feature matching efficiency is improved. It is possible to determine which region of the image template to add the enhanced image feature to based on the division of the image region of the image template and the image quality evaluation of the sub-region, so as to significantly improve the image quality of the image template as a whole; the quality of the segmented image region is still good. The time division is further divided to more accurately determine the image area of poor quality to more specifically add enhanced image features. In addition, the present application can quickly determine the image template to be optimized by using reasonable evaluation criteria, and can also determine whether the divided image regions need further optimization by using reasonable evaluation criteria.

Embodiment 2:

Based on the same inventive concept, a template optimization device is also provided in the embodiment of the present application. Since the principle of solving the problem is similar to the template optimization method, the implementation of the device may refer to the implementation of the method, and the repeated description is not repeated. As shown in FIG. 2, the template optimization apparatus 200 includes:

The obtaining module 201 is configured to acquire an AR image template.

The quality determining module 202 is configured to determine, according to the image features included in the image template, that the image template quality does not satisfy the first criterion;

The optimization module 203 is configured to add an enhanced image feature to the image template.

In some embodiments, the quality determining module 202 is specifically configured to determine that the image template quality does not satisfy the first criterion according to the density and/or the degree of aggregation of the image features included in the image template.

In some embodiments, the apparatus 200 further includes a partitioning module 204 for dividing the image template into a plurality of image regions before adding the enhanced image features to the image template;

The optimization module 203 is specifically configured to add an enhanced image feature to the quality difference region of the image template, where the quality difference region is an image region whose image quality determined according to image features included in each image region does not satisfy the second standard.

In some embodiments, the partitioning module 204 is further configured to divide the high quality area into multiples when there is a high quality area and the quality of the quality area is determined according to the resolution of the quality area. The area, the high quality area is an area in which the image quality determined according to the image features included in each image area satisfies the second criterion.

Embodiment 3:

Based on the same inventive concept, an electronic device is also provided in the embodiment of the present application. Since the principle is similar to the template optimization method, the implementation of the method may refer to the implementation of the method, and the repeated description is not repeated. As shown in FIG. 3, the electronic device 300 includes: a memory 301, one or more processors 302; and one or more modules, the one or more modules being stored in the memory and configured to Executed by the one or more processors, the one or more modules include instructions for performing the various steps of any of the above methods.

Embodiment 4:

Based on the same inventive concept, an embodiment of the present application further provides a computer program product for use in combination with an electronic device, the computer program product comprising a computer program embedded in a computer readable storage medium, the computer program comprising An instruction to cause the electronic device to perform each of the steps of any of the above methods.

For the convenience of description, the various parts of the above-described apparatus are separately described by functions into various modules. Of course, the functions of each module or unit may be implemented in the same software or hardware in the implementation of the present application.

Those skilled in the art will appreciate that embodiments of the present application can be provided as a method, system, or computer program product. Thus, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment in combination of software and hardware. Moreover, the application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) including computer usable program code.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (system), and computer program products according to embodiments of the present application. It will be understood that each flow and/or block of the flowchart and/or block diagrams, and combinations of flows and/or blocks in the flowcharts and/or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing device to produce a machine for the execution of instructions for execution by a processor of a computer or other programmable data processing device. Means for implementing the functions specified in one or more of the flow or in a block or blocks of the flow chart.

The computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture comprising the instruction device. The apparatus implements the functions specified in one or more blocks of a flow or a flow and/or block diagram of the flowchart.

These computer program instructions can also be loaded onto a computer or other programmable data processing device such that a series of operational steps are performed on a computer or other programmable device to produce computer-implemented processing for execution on a computer or other programmable device. The instructions provide steps for implementing the functions specified in one or more of the flow or in a block or blocks of a flow diagram.

While the preferred embodiment of the present application has been described, it will be apparent that those skilled in the art can make further changes and modifications to the embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and the modifications and

Claims

A template optimization method, the method comprising:

Obtain an AR image template;

Determining that the image template quality does not satisfy the first criterion according to the image features included in the image template;

An enhanced image feature is added to the image template.
The method according to claim 1, wherein the determining, according to the image features included in the image template, that the image template quality does not satisfy the first criterion comprises:

The image template quality is determined to not satisfy the first criterion based on the density and/or degree of aggregation of the image features contained in the image template.
The method according to any one of claims 1 to 2, further comprising: before adding the enhanced image feature to the image template,

Dividing the image template into a plurality of image regions;

Adding enhanced image features to the image template includes:

An enhanced image feature is added to the quality difference region of the image template, the quality difference region being an image region whose image quality determined according to image features included in each image region does not satisfy the second standard.
The method of claim 3, wherein the method further comprises:

When there is a high quality area and it is determined that the high quality area can be divided according to the resolution of the high quality area, the high quality area is divided into a plurality of areas, and the high quality area is an image according to each image area The feature-determined image quality satisfies the area of the second standard.
The method of any of claims 3 or 4, wherein the second criterion is determined based on a mean threshold and a variance threshold of Harris response values corresponding to all image features included in the image region.
A template optimization device, characterized in that the device comprises:

Obtaining a module, configured to acquire an AR image template;

a quality determining module, configured to determine, according to the image features included in the image template, that the image template quality does not meet the first criterion;

An optimization module for adding enhanced image features to the image template.
The apparatus according to claim 6, wherein the quality determining module is configured to determine that the image template quality does not satisfy the first criterion according to the density and/or the degree of aggregation of the image features included in the image template.
The apparatus according to any one of claims 6 or 7, wherein said apparatus further comprises a partitioning module for dividing said image template into a plurality of said image templates prior to said adding an enhanced image feature to said image template Image area;

The optimization module is specifically configured to add an enhanced image feature to the quality difference region of the image template, where the quality difference region is an image region whose image quality determined according to image features included in each image region does not satisfy the second standard.
The device according to claim 8, wherein the partitioning module is further configured to: when there is a high quality area and determine that the high quality area can be divided according to the resolution of the high quality area, The excellent area is divided into a plurality of areas, and the high quality area is an area in which the image quality determined according to the image features included in each image area satisfies the second standard.
The apparatus according to any one of claims 8 or 9, wherein said second criterion is determined based on a mean threshold and a variance threshold of Harris response values corresponding to all image features included in the image region.
An electronic device, comprising:

a memory, one or more processors; and one or more modules, the one or more modules being stored in the memory and configured to be executed by the one or more processors, the one or The plurality of modules includes instructions for performing the various steps of the method of any of claims 1 to 5.
A computer program product for use in conjunction with an electronic device, the computer program product comprising a computer program embedded in a computer readable storage medium, the computer program comprising means for causing the electronic device to perform claims 1 to 5 The instructions of the various steps in any of the methods described.