WO2019084726A1

WO2019084726A1 - Marker-based camera image processing method, and augmented reality device

Info

Publication number: WO2019084726A1
Application number: PCT/CN2017/108404
Authority: WO
Inventors: 谢俊
Original assignee: 深圳市柔宇科技有限公司
Priority date: 2017-10-30
Filing date: 2017-10-30
Publication date: 2019-05-09
Also published as: CN111344740A

Abstract

The present invention relates to a marker-based camera image processing method, and an augmented reality device. The method comprises the steps of: selecting or extracting a marker image, performing perspective transformation on the marker image, and obtaining a marker image sequence; extracting sequence feature points of each sequence image in the marker image sequence; obtaining a current camera image, extracting image feature points of the current camera image, matching the image feature points of the current camera image with the sequence feature points of the sequence image, and obtaining feature point pairs that are successfully matched; and in combination with an intrinsic parameter matrix of a camera, computing an extrinsic parameter matrix of a current frame camera according to the feature point pairs that are successfully matched. In the present invention, a marker can be effectively recognized, the recognition rate of the marker can be increased in a great angle, the accuracy rate of feature point matching in a feature point matching process between a marker image and a camera image can also be improved, the computing amount is small, and accordingly, the present invention is applicable to mobile devices.

Description

Camera-based image processing method based on markers, augmented reality device

[0001] The present invention relates to the field of augmented reality technology, and more particularly to a marker-based camera image processing method and an augmented reality device.

Background technique

[0002] The existing Augmented Reality (AR) generally uses a computer vision technology to actually capture the relative positional relationship between a scene and a marker symbol, and input an image that is actually captured.

By comparing the marker image with the image captured by the real image, the following is: searching the captured image for the connected region corresponding to the image identifying the marker, and the connected region is used as a candidate to obtain the contour of each connected region. Line, if it can extract four intersecting straight edges, it can be used as a possible marker; use the angular features found by four right-angled sides to perform deformation correction, and then obtain the correspondence between the marker image and the captured image.

[0003] However, the marker image selected by the method is unique. When the camera changes the angle and moves 吋 during the shooting process, the obtained image and the marker image have a large gap, and then, the feature comparison is performed. , the amount of data to be compared is more, the operation speed is slower, and the recognition rate and accuracy of the markers are worse.

[0004] The technical problem to be solved by the present invention is to provide a marker-based camera image processing method and apparatus for effectively identifying a marker and effectively improving the accuracy in the feature point matching process, and the like Methods and apparatus for implementing augmented reality, and computer readable storage media. Problem solution

Technical solution

[0005] The technical solution adopted by the present invention to solve the technical problem thereof is: Constructing a marker-based camera image processing method, comprising the following steps:

[0006] A: selecting or extracting a marker image, performing perspective transformation on the marker image to obtain a marker image sequence; [0007] B: extracting sequence feature points of each sequence image in the sequence of marker images;

[0008] C: obtaining a current camera image;

[0009] D: extracting image feature points of the current camera image, pairing image feature points of the current camera image with sequence feature points of the sequence image, and obtaining feature point pairs that are successfully matched;

[0010] E: calculating an outer parameter matrix of the current frame camera according to the matching feature point pair of the camera, and the outer parameter matrix of the current frame camera is a feature that the marker image and the camera image are successfully matched. The coordinate correspondence of the points.

[0011] The present invention also provides a marker-based camera image processing apparatus, including:

[0012] a marker image sequence acquisition module, configured to select or extract a marker image, perform perspective transformation on the marker image, and obtain a marker image sequence;

[0013] a first feature point extraction module, configured to extract sequence feature points of each sequence image in the sequence of marker images;

[0014] a current camera image acquisition module, configured to acquire a current camera image;

[0015] a feature point pairing module, configured to extract image feature points of the current camera image, and pair image feature points of the current camera image with sequence feature points of the sequence image to obtain a feature point pair with successful matching ;

[0016] an external parameter matrix calculation module, configured to calculate, according to the feature point pair that is successfully matched, an external parameter matrix of the current frame camera according to the internal parameter matrix of the camera, where the external parameter matrix of the current frame camera is a marker image and The camera image matches the coordinate correspondence of the feature points that are successful. .

[0017] The present invention also provides a method for realizing augmented reality, which uses the above-described marker-based camera image processing method to obtain an external parameter matrix of a camera.

[0018] The present invention also provides an apparatus for implementing augmented reality, comprising a processor for executing a computer program stored in a memory to implement the steps of the method as described above.

The present invention also provides a computer readable storage medium having stored thereon a computer program, the computer program being executed by a processor to implement the steps of the method as described above.

Advantageous effects of the invention

Beneficial effect

[0020] The invention can effectively identify the marker, and can increase the recognition rate of the marker under a large angle, and can also mention The accuracy of the feature point matching between the high marker image and the camera image feature point matching process is small, and is suitable for mobile devices.

Brief description of the drawing

DRAWINGS

[0021] The present invention will be further described below in conjunction with the accompanying drawings and embodiments, in which:

1 is a schematic flow chart of a first embodiment of a method for processing a camera image based on a marker according to the present invention;

2 is a schematic flow chart of a second embodiment of a method for processing a camera image based on a marker according to the present invention;

[0024] FIG. 3-1 is a schematic diagram of a sequence of marker images;

[0025] FIG. 3-2 is a schematic diagram of an original marker image;

3-3 is a schematic diagram of an image generated by two perspective transformations of a marker image in each direction;

4 is a schematic diagram of self-matching results of feature points of a marker image;

[0028] FIG. 5 is a schematic diagram of a region of interest and a region of non-interest;

6 is a schematic diagram of typical image processing;

[0030] FIG. 7 is a schematic diagram of error analysis of a camera external reference matrix;

8 is a schematic diagram of functional modules of a camera function relationship acquisition device based on a marker according to the present invention.

BEST MODE FOR CARRYING OUT THE INVENTION

[0032] In order to more clearly understand the technical features, objects and effects of the present invention, the specific embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0033] Referring to FIG. 1, FIG. 1 is a schematic flow chart of a first embodiment of a method for processing a camera image based on a marker according to the present invention.

. The marker-based camera image processing method of the present embodiment can be applied to augmented reality technology.

[0034] As shown in FIG. 1, the marker-based camera image processing method of this embodiment includes the following steps: [0035] Step A: selecting or extracting a marker image, performing perspective transformation on the marker image to obtain a logo Object image sequence.

[0036] The marker image sequence can be obtained by performing pose transformation on the selected or extracted marker image by using a preset transformation matrix. Among them, the posture transformation performed on the selected or extracted marker image includes translation and rotation.

[0037] The selected or extracted marker image is a marker image previously stored in the memory, wherein the marker The object image may be an image directly called from the image library, or may be a real shot image obtained by field shooting and stored in a memory. The source of the marker image of the present invention is not specifically required.

[0038] The preset transformation matrix can be calculated by preset the marker image and the distance from the camera in the conventional use scene, and the transformation used can be a perspective transformation.

[0039] Further, before step A, the marker-based camera image processing method of the present invention further includes

[0040] Step A1: Obtain an internal parameter matrix of the camera, and the internal reference matrix of the camera includes parameter information of the camera.

[0041] wherein the parameter information of the camera is various parameters of the camera itself, for example, the number of horizontal pixels of the camera itself, the number of vertical pixels, the horizontal and vertical focal lengths of the camera, and the like. These parameters can be obtained by pre-calibrating the camera, or by directly reading the camera parameter information (pixel, focal length, etc.). This embodiment does not require specific requirements.

[0042] Step A2: Initialize the system environment and configure system parameters. This step mainly includes building a system hardware platform.

Set the drawing environment that can support 2D and 3D graphics, allocate image buffer space, identify cameras, and more.

[0043] Step B: Extracting sequence feature points of each sequence image in the sequence of marker images.

[0044] Further, before the step C, the embodiment further includes:

[0045] B11: Feature point extraction is performed on all sequence images in the sequence of marker images by using a feature point extraction algorithm. For example, a surf feature point extraction algorithm, a sift feature point extraction algorithm, or an ORB feature point extraction algorithm may be employed.

In the present embodiment, the ORB feature point extraction algorithm of the present invention extracts sequence feature points of each sequence image.

Compared with the surf feature and the sift feature, the present embodiment uses the ORB algorithm to extract the feature points of the feature points of the marker image, has rotation invariance, and the extraction speed is faster, thereby making the solution applicable to mobile devices. .

[0048] B12: Perform self-matching on the sequence feature points of each sequence image extracted in step B11.

[0049] The sequence feature points of each of the extracted sequence images are self-matched, that is, each sequence image's own sequence feature points are matched with their own sequence feature points. Feature points with high similarity in each sequence image can be obtained by self-matching the sequence feature points of each sequence image. [0050] It can be understood that in the embodiment, the threshold method can be used for self-matching, that is, the value of the corresponding position of the ORB feature sequence of any two sequence feature points in each sequence image is subtracted, and then the absolute value is taken. The value is added to obtain the absolute value of the accumulated value, which is the pairing value of the feature point matching. If the accumulated value is greater than the threshold, the matching failure is judged, and the matching is less than the threshold to determine that the matching is successful, that is, the two sequences are The sequence feature points of the image match.

[0051] B13: The sequence feature points that are successfully matched from the sequence are deleted, and the sequence feature points that fail from the matching are retained.

[0052] Since the self-matching sequence feature points in each sequence image obtained in step B12 may cause subsequent matching confusion, in this step, the self-matching sequence feature points in each sequence image are successfully matched. Deleted to reduce the impact of these sequence feature points on subsequent operations, further speed up the processing, reduce the matching, the amount of operations, and improve the accuracy of feature point matching; the same holds the sequence feature points of the self-matching failure for subsequent steps. Pairing.

[0053] Step C: Acquire the current camera image.

[0054] It can be understood that the current camera image is a frame in the real environment captured by the camera, which is a real acquired image.

[0055] Step D: Extracting the acquired image feature points of the current camera image, and pairing the image feature points of the current camera image with the sequence feature points of the sequence image to obtain a feature point pair with successful matching.

[0056] Further, step D specifically includes:

[0057] D11: Identify a region of interest in the current camera image based on the preset foreign parameter matrix, and remove the non-interest region.

[0058] In step C, the current camera image is acquired, that is, the current frame captured by the camera, and the search is performed in the acquired current camera image according to the preset external parameter matrix, and the flag obtained by matching the previous frame is recognized. The area of the object image in the camera image, which is the region of interest in the current camera image, and the non-interest region is removed based on the region of interest. In the specific operation process, the regions outside the region of interest are replaced by color patches (if the non-interest regions are all filled with black), when the non-interest regions are all filled with black, the feature points outside the region of interest will not be extracted. , thus speeding up the calculation.

[0059] It should be noted that if the preset external reference matrix is not obtained according to the first frame camera image processing, the preset external reference matrix is the previous frame. The marker-based camera image processing method according to the present invention. The external parameter matrix of the obtained camera. After obtaining the external parameter matrix of the camera in the previous frame, it is stored in the memory for use as a preset external parameter matrix for the next frame camera image processing. If the preset external reference matrix is obtained according to the first frame camera image processing, the first frame camera external reference matrix can be obtained by using an existing image processing scheme and stored in the memory for use as the second frame camera image processing. Preset the external parameter matrix.

[0060] It can be understood that if the external parameter matrix 摄像 of the camera is not obtained in the previous frame, the step is not performed, that is, the step D1 l does not need to be performed, that is, the acquired current camera image does not need to be sensed. Identification of areas of interest.

[0061] D12: Extracting the feature points of interest of the region of interest in the current camera image.

[0062] The feature points of interest of the region of interest in the current camera image may be extracted by using a feature point extraction algorithm. In this embodiment, the ORB feature point extraction algorithm is preferably used for extraction.

[0063] D13: pairing the feature points of interest of the region of interest with the sequence feature points retained in step B13

[0064] Further, step D13 specifically includes:

[0065] D131: According to the preset external parameter matrix, a typical image is obtained from the sequence of the marker image, and typical feature points of the typical image are obtained.

[0066] A typical image is a marker image that is closest to the coordinate correspondence of matching feature points described by the outer parameter matrix, that is, in all sequence images of the marker image sequence, the typical image is the current camera image captured with the current camera. The closest sequence image of angle, position, and state. It can be understood that when the outer frame matrix 摄像 of the camera is not acquired in the previous frame, it is not necessary to select a typical image.

[0067] Specifically, the acquisition of a typical image can be achieved by the following steps:

[0068] D1311: acquiring sequence vertex coordinates corresponding to each sequence image in the sequence of marker images;

[0069] D1312: calculating lengths of each side length of each sequence image based on the sequence vertex coordinates, sequentially storing, and obtaining a first side length length sequence of each sequence image;

[0070] D1313: normalizing the first side length sequence of each obtained sequence image; [0071] D1314: obtaining, according to the preset outer parameter matrix, coordinates of the vertex of interest of the region of interest;

[0072] D1315: calculating a second side length length sequence of the region of interest based on the obtained vertex coordinates of the region of interest, and normalizing the calculated second side length sequence of the region of interest; [0073] D1316: respectively calculating a first side length length sequence of all sequence images of the normalized processing in step D1313 and a second side length length sequence of the normalized processing region of interest in step D1315 Distance or Manhattan distance.

[0074] D1317: Determine a typical image according to the obtained Euclidean distance or Manhattan distance.

[0075] determining, by calculating a second side length sequence of the current region of interest and a Euclidean distance or Manhattan distance of the first side length sequence of the marker image, selecting a minimum Euclidean distance or a minimum Manhattan distance, Further, a typical image is determined, that is, a sequence image having the smallest Euclidean distance or the smallest Manhattan distance is a typical image.

[0076] It can be understood that, after the sequence feature points of each sequence image in the marker image sequence are obtained in step B, the sequence feature points of each sequence image are correspondingly saved. Therefore, when a typical image contained in the sequence image is obtained in step D1317, the corresponding sequence feature points can be obtained from the obtained typical image, and these sequence feature points are typical feature points of a typical image.

[0077] D132: Pairing the feature points of interest of the region of interest with typical feature points of the typical image.

[0078] By selecting a typical image from the sequence of marker images, and then pairing the extracted feature points of the region of interest with the typical feature points of the typical image, the number of pairs can be greatly reduced, and the pairing time can be shortened and accelerated. The speed of operation, and the accuracy of pairing is higher.

[0079] Further, step D132 specifically includes the steps of:

[0080] D1321: Using the threshold method, the typical feature points of the typical image and the feature points of interest of the region of interest are performed.

[1321] D1322: Determine whether a pairing value of a typical feature point of the typical image and a feature point of interest of the region of interest is greater than a threshold, and if so, extract a feature point of the typical marker image whose pairing value is greater than a threshold.

[0082] For example, when one feature point of a typical image can match a plurality of feature points of interest in the region of interest, it cannot be determined which pair of feature points of the typical image and which feature point in the region of interest are paired with Correct, easy to cause confusion, in order to avoid confusion, interfere with correct pairing, you need to extract the typical feature points in the typical image that are easy to cause confusion.

[0083] D1323: In the matching result, the typical feature points of the typical image whose pairing value is greater than the threshold value are removed, and the feature point pairs that match the success are obtained.

[0084] In step D1322, after extracting a typical feature point of a typical image whose pairing value is greater than a threshold, after matching In the result, the typical feature points whose pairing value is greater than the threshold value are removed; the typical feature points whose pairing value is greater than the threshold value are saved as the difference feature points in the difference feature point table obtained in the previous frame.

[0085] In other words, in step D132, each of the typical feature points and the feature points of interest are respectively calculated according to typical feature points of the typical image (ie, typical feature point sets) and the feature points of interest of the region of interest. a pairing value of each of the feature points in interest, and comparing the calculated pairing value with a threshold value. If the threshold value is greater than the threshold value, the pairing fails. If the threshold value is less than the threshold value, the pairing is successful, that is, the two feature points whose pairing value is smaller than the threshold value. To match successful feature point pairs. For example, a typical feature point of a typical image is set to X, a feature point of interest in the region of interest is Y, and the pair of typical feature points X and the feature point of interest is Μ. If Μ is greater than the threshold, the typical feature point X Pairing with the feature points of interest fails, and the typical feature point X is saved as a difference feature point to the difference feature point table; if Ν is less than the threshold, the typical feature point X is successfully paired with the feature point of interest, and the typical feature point X A feature point pair that matches the feature point of interest as a match.

[0086] It can be understood that if the current frame is the first frame, the difference feature point table is empty.

[0087] Step Ε: According to the matching feature point pair, the external parameter matrix of the current frame camera is calculated according to the internal parameter matrix of the camera, and the outer parameter matrix of the current frame camera is the coordinate of the feature point where the marker image and the camera image match successfully. Correspondence relationship.

[0088] Specifically, using the RPP (Robust Planar Pose) algorithm, according to the matching feature point pairs obtained in step D, and calculating the external parameter matrix of the camera of the current frame in combination with the internal parameter matrix of the camera.

[0089] The external parameter matrix of the camera, which is a camera for photographing markers, can capture the current captured image state of the marker by translating and rotating in space. That is, the correspondence relationship between the marker image and the point coordinates of the camera image captured by the camera is shown. The correspondence is represented by a function, and this function is described by a matrix, that is, an external parameter matrix. In other words, the external parameter matrix of the camera is A correspondence relationship between the marker image and the coordinates of the point of the camera image is indicated.

[0090] Further, the marker-based camera image processing method of the embodiment further includes after step E

[0091] F: Perform error calculation on the outer parameter matrix of the current frame camera acquired in step E, and obtain an error result.

[0092] Step F specifically includes the steps of:

[0093] F1: based on the outer frame matrix of the current frame camera acquired in step E, combining the sequence image with the current camera Calculating the coordinates of all the feature points that match the success of the image, and calculating the calculated coordinates of the sequence feature point coordinates of the sequence image in the camera coordinate system;

[0094] F2: calculating an error distance between the calculated coordinates obtained in step F1 and the matching coordinates of the feature points that are successfully matched in the current camera image;

[0095] F3: Calculate the average error distance based on all the error distances obtained.

[0096] 0: According to the error result, it is verified whether the external parameter matrix of the camera of the current frame is correct.

[0097] 01: determining whether the average error distance is greater than an average error distance threshold;

[0098] 02: If the average error distance is greater than the average error distance threshold, the external parameter matrix of the camera obtained in step E is incorrect, otherwise, the outer parameter matrix of the current frame camera obtained in step E is determined to be correct.

[0099] Preferably, if it is determined in step G2 that the outer parameter matrix of the current frame camera obtained in step E is correct, the following steps are further performed:

[0100] updating the region of interest and the difference feature point table saved in the previous frame;

[0101] The outer parameter matrix of the current frame camera is saved. The saved outer frame matrix of the current frame camera can be used for the preset outer parameter matrix of the camera image of the next frame.

[0102] It can be understood that if it is verified in step G2 that the outer parameter matrix of the current frame camera obtained in step E is erroneous, the current image processing fails, and the obtained outer parameter matrix is not saved, and the same is cleared. The saved region of interest and the difference feature point table.

[0103] Referring to FIG. 2, FIG. 2 is a schematic flow chart of a second embodiment of a method for processing a camera image based on a marker according to the present invention. The marker-based camera image processing method of this embodiment can be used to implement a reality enhancement technique.

[0104] As shown in FIG. 2, the marker-based camera image processing method of this embodiment includes steps 201 to 2

09. specifically:

[0105] Step 201: Acquire an internal parameter matrix of the camera, where the internal parameter matrix of the camera includes parameter information of the camera

[0106] wherein the parameter information of the camera is various parameters of the camera itself, for example, the number of horizontal pixels of the camera itself, the number of vertical pixels, the horizontal and vertical focal lengths of the camera, and the like. These parameters can be obtained by pre-calibrating the camera, or by directly reading the camera parameter information (pixel, focal length, etc.). This embodiment does not require specific requirements.

[0107] Step 202: Select or extract a marker image, perform perspective transformation on the marker image, and obtain a target A sequence of images of volunteers.

[0108] It is understood that the essence of step 202 is to simulate a marker image that is captured when the camera image is selected or extracted by the camera that is not perpendicular to the marker (which is the original marker image).

[0109] Specifically, the simulation mode may be acquired by perspective transformation, and the preset transformation matrix used may be a preset distance matrix between the target image and the conventional use scene.

[0110] FIG. 3 is a sequence of a marker image obtained by performing perspective transformation by a transformation matrix of FIG. 3-2, wherein the perspective transformation matrix can virtually construct an external parameter matrix by simulating a posture change of the camera, and combining the camera The internal parameter matrix is calculated.

[0111] It can be understood that, in this embodiment, FIG. 3-1 only generates four directions (one direction is changed every 90 degrees), and each direction is one tilt angle, but if a better effect is needed, You can increase the direction and angle of each direction (as shown in Figure 3-3); in Figure 3-3, 2 perspective transformations are performed in each direction to generate 2 images.

[0112] The transformation matrix is preset, and each time a new marker image is used, a sequence of marker images can be generated using the same preset transformation matrix.

[0113] Step 203: Extract sequence feature points of each sequence image in the sequence of marker images, and perform self-matching on sequence feature points of each sequence image to remove sequence feature points that are successfully matched.

[0114] In this embodiment, the extraction of sequence feature points of each sequence image in the sequence of marker images may be adopted.

The ORB algorithm performs extraction. The ORB feature extracted by the ORB algorithm has rotation invariance and fast extraction speed

Suitable for mobile devices to run during the move.

[0115] The ORB feature is an integer sequence of length 64. The matching process is to subtract the values of the corresponding positions of the ORB feature sequence of the two feature points, then take the absolute value, and add these absolute values to obtain absolute The accumulated value of the value, which is the pairing value of the feature point matching. Threshold method

If the accumulated value is greater than the threshold, the matching failure is judged, and if the value is smaller than the threshold, the matching is successful.

[0116] In this embodiment, the purpose of self-matching all sequence feature points of each sequence image is to remove feature points with high similarity in the current marker image. As shown in Figure 4, the three points represented by a, b, and c are the feature points that we want to remove. These three feature points have high similarity, which will cause confusion.

[0117] Further, for the ORB feature point calculated by the marker image generated by the change, the coordinates of the feature point are described by the coordinate position before the transformation, and the value of the ORB feature sequence is unchanged. For example, sitting in the original marker image The point marked as (1,1) is transformed into the coordinates (10,10) after the perspective transformation. When the coordinate point is detected as the ORB coordinate point in the transformed image, the ORB feature sequence is calculated and the transformed image is used. Because it is the feature point detected after the transformation), the coordinate position of the point is recorded with the original coordinate (1, 1). In this way, the external parameter matrix 求解 in the solution camera can be calculated by the correspondence between the coordinates of the marker image and the coordinates of the marker image in the camera, which is actually calculated by the correspondence between a series of feature point coordinates. Therefore, the coordinates of the original marker image are used to ensure that the correct result is calculated.

[0118] Step 204: Acquire a current camera image.

[0119] Step 205: Identify the region of interest in the current camera image based on the preset foreign parameter matrix, and go to the region of interest.

[0120] It should be noted that, if the preset external reference matrix is not obtained according to the first frame camera image processing, the preset external reference matrix is the previous frame. The marker-based camera image processing method according to the present invention. The external parameter matrix of the obtained camera. In addition to obtaining the external parameter matrix of the camera in the previous frame, it is stored in the memory and used as a preset external parameter matrix for the next frame camera image processing. If the preset external reference matrix is obtained according to the first frame camera image processing, the first frame camera external reference matrix can be obtained by using an existing image processing scheme and stored in the memory for use as the second frame camera image processing. Preset the external parameter matrix.

[0121] The region of interest is derived from the processing of the previous frame, which can be described by a plurality of vertices (eg, a quadrilateral described by four vertices). According to the preset external parameter matrix, the region of interest corresponding to the camera image of the current frame is obtained.

[0122] As shown in FIG. 5, the white area is the region of interest, and the area outside the white area is the non-interest area ± or, wherein the quadrilateral external image (non-interest area) is replaced by black, The regions of interest are all replaced by black, so that the feature points of the black area can not be extracted, which can greatly improve the calculation speed.

[0123] Step 206: Extract the feature points of interest of the region of interest in the current camera image.

[0124] The feature points of interest of the region of interest in the current camera image are extracted by the feature point extraction algorithm. This embodiment preferably uses an ORB feature point extraction algorithm for extraction.

[0125] Step 207: Obtain a typical image from the sequence of the marker image according to the preset external parameter matrix, and obtain the code. Typical feature points of a type image.

[0126] Step 207 is essentially to select a marker image in the sequence of marker images that is closest to the current camera image state from the sequence of marker images, ie, the aforementioned exemplary image, which is used for the current camera image. Make a match.

[0127] It can be understood that if the previous frame external parameter matrix calculation is not successful (ie, the external parameter matrix of the camera obtained in the previous frame is wrong), the feature points of the original marker image are used, that is, the function of FIG. 3-2. Feature points. Otherwise, a typical image is selected from all the sequence of marker images, and typical feature points of the typical image are used to match the feature points of the camera image acquired by the current frame. The specific options are as follows:

[0128] Calculating the length of four sides of each marker image in the sequence of marker images, and storing them in order, as shown in FIG. 6, the lengths of the four sides are stored as 5 length sequences in the order of 1, 2, 3, and 4. . For each sequence, each side length is divided by the length of the 1st side of the sequence (including the 1st side) and normalized. Then calculate the external parameter matrix of the saved camera according to the previous frame, calculate the coordinate positions of the points A, B, C, and D on the camera image of the previous frame, and calculate the line segments AC, AB, BD on the image of the previous frame camera. The length of the DC, the sequence of lengths, is also normalized. Match all length sequences (5 length sequences above) in the sequence of marker images.

[0129] The specific matching manner is to separately calculate the length sequence of the camera image composition and the Euclidean distance or the Manhattan distance of the length sequence composed of the respective marker images, and select the smallest one from the calculated Euclidean distance or Manhattan distance, wherein A corresponding sequence image with the smallest Euclidean distance or Manhattan distance is a typical image.

[0130] Step 208: Pair the typical feature points of the typical image acquired in step 207 with the feature points of interest in the region of interest to obtain a difference feature point, and remove the difference feature point in the matching result to obtain a matching success. Feature point pairs.

[0131] A typical feature point of a typical image is paired with a feature point of interest of a region of interest using a threshold method.

It is judged whether the pairing value of the typical feature point of the typical image matches the feature point of interest of the region of interest is greater than a threshold, and if so, the feature point of the typical marker image whose pairing value is larger than the threshold is extracted.

[0132] For example, when feature points of a sequence image can be matched to multiple feature points of the region of interest, it is impossible to determine which feature point of the sequence image is correctly paired with which feature point in the region of interest, which is easy to cause Chaos, in order to avoid confusion, interfere with correct pairing, the typical characteristics of the pairing value is greater than the threshold The points are extracted, and a typical feature point whose pairing value is larger than the threshold is used as a difference feature point obtained by the current frame.

[0133] Further, the typical feature points whose pairing value is greater than the threshold value are saved in the difference feature point table obtained in the previous frame. It can be understood that if the current frame is the first frame, the difference feature point table is empty.

[0134] Step 209: Calculate the external parameter matrix of the current frame camera according to the matching feature point pair obtained in step 208, and verify whether the calculated outer frame matrix of the current frame camera is correct.

[0135] In this step, the external parameter matrix of the camera is calculated using the Robust Planar Pose (RPP) algorithm. Obviously, using the RPP algorithm to calculate the external parameter matrix of the camera generally requires the internal reference matrix of the camera and the correspondence between the coordinates of the feature points of at least four pairs of marker images and the coordinates of the feature points of the camera image, that is,

, matching feature point pairs are greater than or equal to four.

[0136] In order to make the matching precision higher, the present invention may perform processing by setting a threshold. Specifically, a threshold may be set, which is assumed to be N (N>4), and is determined before the calculation of the external parameter matrix. Whether the logarithm of the matched feature point pair is greater than N, and if so, the subsequent operation is performed, and if not, the current calculation process fails.

[0137] Further, in the embodiment, the method of error analysis is used to verify whether the calculated outer parameter matrix of the camera of the current frame is correct.

[0138] As shown in the following mathematical model (the mathematical model is a camera perspective projection model), M1 represents the internal parameter matrix of the camera, and M2 represents the calculated external parameter matrix of the camera of the current frame. Let 0XYZ be the world coordinate system and uv be the image coordinate system in pixels. If the coordinates of the object point P in the world coordinate system are (X, Y, Ζ), the coordinates of the corresponding object point Ρ in the image coordinate system are (u, v).

[0139] The method of the present invention directly uses the original marker pixel point coordinates as the X-axis and Y-axis coordinates of the world coordinate system, Z is 0 (photographed with the original marker image position as the Z-axis zero point), and the pixel coordinates of the current camera shot. Is the pixel coordinate system. In this case, the obtained internal reference matrix and the external reference matrix represent the image coordinates of each point of the original marker image 吋 marker image, and then the function relationship of the camera image coordinates is extracted. Using this function relationship, and the coordinates of each matching marker image feature point, an image coordinate is calculated, and the distance between the image coordinate and the matching camera image feature point is obtained, and the distance is set as the feature point of the marker image. The error distance.

[0140] For the calculation result of each frame, the error distances of all the feature points (matching success) of the currently calculated marker image are added, divided by the logarithm of the total matching success, and the average error distance is calculated. Set a threshold Value, if the average error distance is greater than this threshold, the calculation fails, that is, the external parameter matrix of the camera calculated in the current frame is incorrect; if the average error distance is less than the threshold, the calculation is successful, that is, the external parameter matrix of the camera calculated in the current frame correct.

[0141] Step 210: Further, if it is determined in step 209 that the calculated external parameter matrix of the current frame camera is correct, updating the region of interest and the difference feature point table saved in the previous frame, and saving the current frame is calculated. The external parameter matrix of the camera. The saved outer parameter matrix of the current frame camera is used as a preset outer parameter matrix of the next frame camera image.

[0142] Specifically: for the feature point of each marker image, if the error distance is greater than the error distance threshold, the feature point of the marker image is a difference feature point, and the sequence number of the feature point of the marker is recorded and added. The difference feature point table saved in the previous frame. Alternatively, the error distance sum of the feature points of the marker image of the last consecutive frames may be set, and the error distance and the threshold may be set. If the error distance is greater than the error distance and the threshold, the feature point of the marker image may be determined as The difference feature point is added to the feature point of the marker image added to the difference feature point table saved in the previous frame.

[0143] As shown in FIG. 7, the dotted line represents a feature point pair in which the marker image feature point and the camera image feature point match successfully, and the solid line represents the coordinates derived by using the calculated external parameter matrix of the camera after performing the RPP algorithm. Point correspondence. In the left figure, 1, 2, 3, 4 represent the position of the feature point in the marker image, and the right image represents the position of the corresponding feature point in the camera image, where, 2', 3' 4' is a feature point that is successfully paired with 1, 2, 3, and 4 in the image of the marker in the camera image; 2, 3, and 4" are reversed by using the obtained external parameter matrix after using the RPP algorithm The feature points in the camera image calculated by the verification are 1, 2, 3, and 4. According to the verification result, it can be judged whether it is correct. The error distance between the feature points 4, 4' and 4" in Fig. 7 is large. Then, it can be judged that the feature point 4 is a difference feature point and is added to the difference feature point table.

[0144] Further, if the calculated external parameter matrix of the current frame camera is correct, that is, the calculation is successful, and then the A, B, C, and D in FIG. 6 are calculated according to the internal reference matrix and the external parameter matrix. The coordinates of the points in the current camera image, the quadrilateral represented by the vertices of the four coordinates is the region of interest, the coordinates of the four points are saved for the next frame to exclude the non-interest region, and the sequence from the marker image A sequence image (i.e., the aforementioned typical image) of the change relationship described by the camera outer parameter matrix obtained closest to the current frame is selected, and typical feature points of the typical image are selected. [0145] It can be understood that if the outer parameter matrix obtained by the current frame or any subsequent frame is wrong, the difference feature point table of the region of interest is cleared.

[0146] The present invention also provides a method for realizing augmented reality, which uses the above-described marker-based camera image processing method to obtain an external parameter matrix of a camera.

[0147] Further, the method for implementing augmented reality further includes:

[0148] according to the internal reference matrix of the camera and the external parameter matrix, the virtual graphic at the current position of the camera is drawn in the preset model;

[0149] The obtained virtual graphic is combined with the current camera image to obtain a composite image. It can be understood that the synthesized image is an AR image.

[0150] The present invention also provides an apparatus for implementing augmented reality, the apparatus comprising a processor for executing a computer program stored in the memory to implement the steps of the method as described above.

The present invention also provides a computer readable storage medium having stored thereon a computer program executed by a processor to implement the steps of the method as described above.

[0152] The present invention also provides a marker-based camera image processing apparatus, the apparatus comprising:

[0153] a marker image sequence obtaining module 801, configured to select or extract a marker image, and perform perspective transformation on the marker image to obtain a marker image sequence;

[0154] The first feature point extraction module 802 is configured to extract sequence feature points of each sequence image in the sequence of marker images;

[0155] The current camera image acquisition module 803 is configured to acquire a current camera image;

[0156] a feature point pairing module 804, configured to extract an image feature point of the current camera image, and pair an image feature point of the current camera image with a sequence feature point of the sequence image to obtain a feature point pair that is successfully matched;

[0157] The external parameter matrix calculation module 805 is configured to calculate a foreign parameter matrix of the current frame camera according to the matching feature point pair, and the current frame camera external parameter matrix is successfully matched with the camera image by the current frame camera external reference matrix. The coordinate correspondence of the feature points.

[0158] Further, the marker-based camera function relationship obtaining apparatus of the embodiment further includes: [0159] an error verification module 806, configured to perform error calculation on the acquired outer parameter matrix of the current frame camera, and obtain an error. As a result, based on the error result, it is verified whether the outer parameter matrix of the camera of the current frame is correct. The above is only the embodiment of the present invention, and is not intended to limit the scope of the invention, and the equivalent structure or equivalent process transformations made by the specification and the drawings of the present invention may be directly or indirectly applied to other related technologies. The scope of the invention is included in the scope of patent protection of the present invention.

Claims

Claim

A marker-based camera image processing method, characterized in that the method comprises the following steps:

A: selecting or extracting a marker image, performing perspective transformation on the marker image to obtain a sequence of marker images;

B: extracting sequence feature points of each sequence image in the sequence of marker images; C: acquiring a current camera image;

D: extracting image feature points of the current camera image, and pairing image feature points of the current camera image with sequence feature points of the sequence image to obtain a feature point pair with successful matching;

E: calculating an outer parameter matrix of the current frame camera according to the matching feature point pair of the camera, and the outer parameter matrix of the current frame camera is the coordinate of the feature point that the marker image and the camera image match successfully. Correspondence relationship.

The method for processing a camera image based on a marker according to claim 1, wherein the step A further comprises the steps of:

A1: acquiring an internal parameter matrix of the camera, where the internal parameter matrix of the camera includes parameter information of the camera;

A2: Initialize the system environment and configure system parameters.

The method for processing a camera image based on a marker according to claim 1, wherein the step A specifically includes the following steps:

The selected or extracted marker image is subjected to pose transformation using a preset transformation matrix to generate the marker image sequence.

The method of processing a marker-based camera image according to claim 1, wherein the step C further comprises:

B11: Feature point extraction is performed on all sequence images in the marker image sequence by using a feature point extraction algorithm.

The method of processing a marker-based camera image according to claim 4, wherein the step B11 comprises: Feature point extraction is performed on all sequence images in the marker image sequence by using an ORB feature point extraction algorithm.

The method of claim 4, wherein the step C further comprises:

B12: Perform self-matching on sequence feature points of each sequence image extracted in step B11;

B13: Deleting self-matching sequence feature points, retaining sequence feature points from self-matching failure

[Claim 7] The method for processing a camera image based on a marker according to claim 6, wherein the step D specifically includes the following steps:

D11: identifying a region of interest in the current camera image based on the preset foreign parameter matrix, and removing the non-interest region;

D12: extracting a feature point of interest of the region of interest in the current camera image; D13: pairing the feature point of interest of the region of interest with the sequence feature point retained in step B13.

[Claim 8] The method for processing a camera image based on a marker according to claim 7, wherein the step D13 specifically includes the following steps:

D131: Obtain a typical image from the sequence of the marker image according to the preset foreign parameter matrix, and acquire a typical feature point of the typical image;

D132: Pair the feature points of interest of the region of interest with typical feature points of the typical image.

[Claim 9] The method for processing a camera image based on a marker according to claim 8, wherein the step D131 specifically includes the following steps:

D1311: Obtain a sequence vertex coordinate corresponding to each sequence image in the sequence of the marker image;

D1312: Calculate each side length of each sequence image based on the sequence vertex coordinates, and sequentially save to obtain a first side length sequence of each sequence image;

D1313: Normalize the first side length sequence of each sequence image obtained deal with;

D1314: Obtain, according to the preset outer parameter matrix, coordinates of a vertex of interest of the region of interest;

D1315: Calculate a second side length length sequence of the region of interest based on the obtained vertex coordinates of the region of interest, and normalize the calculated second side length sequence of the region of interest;

D1316: respectively calculating a first side length length sequence of all sequence images of the normalized processing in step D1313 and a second side length length sequence of the normalized processing region of interest in step D1315 Euclidean distance or Manhattan distance; D1317: Judging from the obtained Euclidean distance or Manhattan distance, the typical image is obtained.

[Claim 10] The method for processing a camera image based on a marker according to claim 9, wherein the step D132 specifically includes:

D1321: pairing a typical feature point of the typical image with a feature point of interest of the region of interest by using a threshold method;

D1322: determining whether a pairing value of a typical feature point of the typical image and a feature point of interest of the region of interest is greater than a threshold, and if so, extracting a typical feature point of a typical image whose pairing value is greater than a threshold;

D1323: In the matching result, the typical feature points of the typical image whose pairing value is greater than the threshold value are removed, and the feature point pairs with matching success are obtained.

[Claim 11] The method for processing a camera image based on a marker according to claim 1, wherein the step E specifically includes the following steps:

The RPP algorithm is used to calculate the matching feature point pairs obtained in step D, and the external parameter matrix of the current frame camera is calculated according to the internal parameter matrix of the camera.

The method of claim 1, wherein the step E further comprises:

F: performing error calculation on the outer parameter matrix of the current frame camera acquired in step E, and obtaining an error result; G: verifying, according to the error result, whether the external parameter matrix of the current frame camera is correct

The method of claim 12, wherein the step F comprises the following steps:

F1: based on the outer parameter matrix of the current frame camera acquired in the step E, combining the coordinates of the feature points of the sequence image and the current camera image, and calculating the sequence feature point coordinates of the sequence image in the camera coordinate system Calculated coordinates in ;

F2: calculating an error distance between the calculated coordinates obtained in the step F1 and the matching coordinates of the feature points in the current camera image that match the success;

F3: Calculate the average error distance based on all the error distances obtained;

The step G specifically includes the steps of:

01: determining whether the average error distance is greater than an average error distance threshold;

02: If the average error distance is greater than the average error distance threshold, the outer parameter matrix of the current frame camera obtained in step E is wrong. Otherwise, it is determined that the outer frame matrix of the current frame camera obtained in step E is correct.

The method of claim 13 is as follows: The method further includes:

If the outer frame matrix of the current frame camera obtained in the step E is correct, further perform the following steps:

Updating the region of interest and the difference feature point table saved in the previous frame;

The outer parameter matrix of the current frame camera is saved as the preset external reference matrix of the next frame camera image.

[Claim 15] A marker-based camera image processing apparatus, wherein the apparatus includes a marker image sequence acquisition module for selecting or extracting a marker image, and performing perspective on the marker image Transforming, obtaining a sequence of marker images; a first feature point extraction module, configured to extract sequence feature points of each sequence image in the sequence of marker images; a current camera image acquisition module, configured to acquire a current camera image;

a feature point matching module, configured to extract an image feature point of the current camera image, and match an image feature point of the current camera image with a sequence feature point of the sequence image to obtain a feature point pair that is successfully matched;

The outer parameter matrix calculation module is configured to calculate an outer parameter matrix of the current frame camera according to the matching feature point pair of the matching, and the outer parameter matrix of the current frame camera is the target image matching the camera image A marker-based camera image processing device according to claim 15, wherein the device further comprises:

The error verification module is configured to perform error calculation on the acquired outer parameter matrix of the current frame camera, obtain an error result, and verify whether the outer parameter matrix of the current frame camera is correct according to the error result.

A method for realizing augmented reality, characterized in that the method uses the marker-based camera image processing method according to any one of claims 1 to 14 to obtain an external parameter matrix of the camera, and realizes augmented reality according to claim 17. The method, further comprising:

Drawing a virtual graphic at a current position of the camera in a preset model according to the internal reference matrix of the camera and the external parameter matrix;

The obtained virtual graphic is combined with the current camera image to obtain a composite image. An apparatus for implementing augmented reality, characterized in that the apparatus comprises a processor for executing a computer program stored in a memory to implement the steps of the method of any one of claims 1-14.

A computer readable storage medium having stored thereon a computer program, characterized in that the computer program is executed by a processor to perform the steps of the method of any of claims 1-14.