CN110766025A

CN110766025A - Method, device and system for identifying picture book and storage medium

Info

Publication number: CN110766025A
Application number: CN201910953281.5A
Authority: CN
Inventors: 朱铭德; 周志敏; 丛林
Original assignee: Hangzhou Yixian Advanced Technology Co Ltd
Current assignee: Hangzhou Yixian Advanced Technology Co Ltd
Priority date: 2019-10-09
Filing date: 2019-10-09
Publication date: 2020-02-07
Anticipated expiration: 2039-10-09
Also published as: CN110766025B

Abstract

The application relates to the technical field of image processing, and discloses a picture book identification method, a device, a system and a storage medium, wherein the method comprises the following steps: acquiring a shot image acquired by image acquisition equipment; determining a target picture template matched with the shot image from the plurality of picture templates; determining the position information of a picture book in a shot image in an image coordinate system corresponding to image acquisition equipment; and determining the projection position of the position information relative to the projection equipment according to the coordinate mapping relation between the image coordinate system and the projection coordinate system corresponding to the projection equipment, so that the projection equipment projects the projection picture corresponding to the target notebook template onto the notebook according to the projection position. The painted book identification method, the painted book identification device, the painted book identification system and the storage medium can identify the painted book and the identification method of the position of the painted book, so that the painted book can be accurately projected, the effect of drawing sound and color is achieved, and the interest in reading the painted book is improved.

Description

Method, device and system for identifying picture book and storage medium

Technical Field

The present application relates to the field of image processing technologies, and in particular, to a method, an apparatus, a system, and a storage medium for identifying a picture book.

Background

This section is intended to provide a background or context to the embodiments of the application that are recited in the claims. The description herein is not admitted to be prior art by inclusion in this section.

The Picture Book is mainly a Picture and can be connected into a Book with a single complete story. The current picture book is mainly set for infants and children, and takes the content suitable for the infants and children to read as the orientation. The picture book reading can help children to know the world, learn knowledge and more importantly develop the hobbies and interests of the children in reading.

The current mainstream picture book identification method mainly comprises the following steps: the method comprises the steps of carrying out picture recognition through an image recognition method, playing audio set for the picture in advance based on the recognized picture, and improving the interest of children in reading the picture in the process so as to improve the cognitive ability of the children.

However, the existing picture book identification method can only identify the page number corresponding to the picture book and then play the corresponding audio, and only can achieve the sound drawing effect, and the expression form is single.

Disclosure of Invention

In view of the above technical problems, there is a need for a method for identifying a painted book and a position of the painted book, so as to accurately project the painted book, achieve the effect of drawing sound and color, and improve the interest in reading the painted book.

In one aspect, an embodiment of the present application provides a picture book identification method, including:

acquiring a shot image acquired by image acquisition equipment;

determining a target picture template matched with the shot image from a plurality of picture templates;

determining the position information of a picture book in the shot image in an image coordinate system corresponding to the image acquisition equipment;

and determining the projection position of the position information relative to the projection equipment according to the coordinate mapping relation between the image coordinate system and the projection coordinate system corresponding to the projection equipment, so that the projection equipment projects the projection picture corresponding to the eye chart template onto the chart according to the projection position.

In one aspect, an embodiment of the present application provides a picture book recognition apparatus, including:

the image acquisition module is used for acquiring a shot image acquired by the image acquisition equipment;

the picture book matching module is used for determining a target picture book template matched with the shot image from a plurality of picture book templates;

the picture book position determining module is used for determining the position information of the picture book in the shot image in an image coordinate system corresponding to the image acquisition equipment;

the projection module is used for determining the projection position of the position information relative to the projection equipment according to the coordinate mapping relation between the image coordinate system and the projection coordinate system corresponding to the projection equipment, so that the projection equipment projects the projection picture corresponding to the eye chart template onto the chart book according to the projection position.

In one aspect, an embodiment of the present application provides a picture recognition apparatus, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any one of the methods when executing the computer program.

In one aspect, an embodiment of the present application provides a picture book recognition system, including: image acquisition equipment, projection equipment and any one of the picture recognition devices;

the image acquisition equipment is used for acquiring a shot image containing a picture book;

and the projection equipment is used for projecting the projection picture corresponding to the target picture template determined by the picture book recognition device onto the picture book according to the projection position determined by the picture book recognition device.

In one aspect, an embodiment of the present application provides a computer-readable storage medium having stored thereon computer program instructions, which, when executed by a processor, implement the steps of any of the above-described methods.

In one aspect, an embodiment of the present application provides a computer program product comprising a computer program stored on a computer-readable storage medium, the computer program comprising program instructions that, when executed by a processor, implement the steps of any of the methods described above.

According to the technical scheme, the target picture template matched with the shot image is determined from the plurality of picture templates, the position information of the picture in the shot image in the image coordinate system corresponding to the image acquisition equipment is determined, then the projection position of the position information relative to the projection equipment is determined according to the coordinate mapping relation between the image coordinate system and the projection coordinate system corresponding to the projection equipment, therefore, the projection equipment can accurately project the projection picture corresponding to the target picture template onto the picture according to the projection position, and the fun in the process of reading the picture is improved in the mode of projection on the picture.

Drawings

The above and other objects, features and advantages of exemplary embodiments of the present application will become readily apparent from the following detailed description read in conjunction with the accompanying drawings. Several embodiments of the present application are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings and in which:

fig. 1 is a schematic view of an application scenario of a picture book recognition method according to an embodiment of the present application;

fig. 2 is a schematic flowchart illustrating a picture recognition method according to an embodiment of the present application;

FIG. 3 is a diagram illustrating an example of a pattern projected by a projection device when calibrating a coordinate mapping relationship according to an embodiment of the present disclosure;

FIG. 4 is a schematic flow chart of the template for matching purposes provided by an embodiment of the present application;

fig. 5 is a schematic structural diagram of a feature extraction model used in a DeLF method according to an embodiment of the present application;

fig. 6 is a schematic flowchart illustrating a process of obtaining feature points corresponding to each template at each resolution according to an embodiment of the present application;

FIG. 7 is a schematic flow chart of a determining objective plotting template provided by an embodiment of the present application;

FIG. 8 is a flowchart illustrating geometric verification of a template according to an embodiment of the present application;

fig. 9 is a flowchart illustrating a process of determining a matching sequence of the sketch template corresponding to each resolution according to an embodiment of the present application;

fig. 10 is a schematic flowchart illustrating a process of filtering feature points of an extracted captured image according to an embodiment of the present application;

fig. 11 is a schematic diagram illustrating an operation of filtering feature points of an extracted captured image according to an embodiment of the present application;

fig. 12 is a flowchart illustrating a picture recognition method according to an embodiment of the present application;

FIG. 13 is a schematic flow chart illustrating tracking of feature points of the template according to an embodiment of the present disclosure;

fig. 14 is a schematic structural diagram of a picture recognition apparatus according to an embodiment of the present application;

fig. 15 is a schematic structural diagram of a picture recognition apparatus according to an embodiment of the present application;

fig. 16 is a schematic structural diagram of a picture recognition system according to an embodiment of the present application.

Detailed Description

The principles and spirit of the present application will be described with reference to a number of exemplary embodiments. It should be understood that these embodiments are given solely for the purpose of enabling those skilled in the art to better understand and to practice the present application, and are not intended to limit the scope of the present application in any way. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

As will be appreciated by one skilled in the art, embodiments of the present application may be embodied as a system, apparatus, device, method, or computer program product. Accordingly, the present disclosure may be embodied in the form of: entirely hardware, entirely software (including firmware, resident software, micro-code, etc.), or a combination of hardware and software.

In this document, it is to be understood that any number of elements in the figures are provided by way of illustration and not limitation, and any nomenclature is used for differentiation only and not in any limiting sense.

For convenience of understanding, terms referred to in the embodiments of the present application are explained below:

drawing the book: the picture book is a 'drawn book' which is mainly drawn and matched with a small amount of characters, and is often used for telling a story for children and helping the children draw knowledge to form cognition.

AR (Augmented Reality) projection system: through the behaviors of the sensor and the user, the projection light machine plays animation/pictures/sound at the identified object or the specific position, so that the purpose of enhancing reality is achieved, and the projection light machine is very suitable for the education industry.

Homography matrix (homographic): the mapping relation (3 x 3) between two planes has an actual degree of freedom of 8, so at least 4 pairs of points can solve the corresponding relation between the two planes.

The terminal equipment: the electronic device can be mobile or fixed, and can be used for installing various applications and displaying objects provided in the installed applications. For example, a mobile phone, a tablet computer, various wearable devices, a vehicle-mounted device, a Personal Digital Assistant (PDA), a point of sale (POS), or other electronic devices capable of implementing the above functions may be used.

The principles and spirit of the present application are explained in detail below with reference to several representative embodiments of the present application.

Summary of The Invention

The applicant finds that the existing picture book identification method can only identify the page number corresponding to the picture book and then play the corresponding audio, and only can achieve the sound drawing effect, and the expression form is single.

In order to solve the problem, in the implementation of the application, a shot image acquired by an image acquisition device is acquired; determining a target picture template matched with the shot image from a plurality of picture templates; determining the position information of a picture book in the shot image in an image coordinate system corresponding to the image acquisition equipment; and determining the projection position of the position information relative to the projection equipment according to the coordinate mapping relation between the image coordinate system and the projection coordinate system corresponding to the projection equipment, so that the projection equipment projects the projection picture corresponding to the eye plotting book template onto the plotting book according to the projection position, and the pleasure in the process of reading the plotting book is improved in a projection mode on the plotting book.

Having described the basic principles of the present application, various non-limiting embodiments of the present application are described in detail below.

Application scene overview

Fig. 1 is a schematic view of an application scenario of the picture book recognition method according to the embodiment of the present application. The application scenario comprises a picture book 101, an image acquisition device 102, a picture book recognition apparatus 103, and a projection device 104. The image acquisition device 102 is connected with the picture book recognition device 103, the projection device 104 is connected with the picture book recognition device 103, the image acquisition device 102, the picture book recognition device 103 and the projection device 104 can be integrated in the same terminal device, and meanwhile, in order to ensure the projection effect, the shooting area 106 of the image acquisition device 102 and the projection area 107 of the projection device 104 are overlapped as much as possible.

In order to ensure the stability of the image capture device 102, the image capture device 103, and the projection device 104, the terminal device can be fixed, and then the image capture device 101 is placed in the shooting area 106 of the image capture device 102 for image capture. Specifically, the image capturing device 101 captures a captured image including a picture book, and sends the captured image to the picture book recognition device 103; the picture book recognition device 103 determines a target picture book template matched with the shot image from the plurality of picture book templates, determines the position information of the picture book in the shot image in the image coordinate system corresponding to the image acquisition equipment 102, and determines the projection position of the position information relative to the projection equipment 104 according to the coordinate mapping relation between the image coordinate system and the projection coordinate system corresponding to the projection equipment 104; the projection device 104 projects the projection picture corresponding to the eye drawing book template onto the drawing book 101 according to the projection position determined by the drawing book recognition device 103, so that the user can see the projection picture on the drawing book, and the interest in reading the drawing book is improved.

The projection screen in the embodiment of the present application may be an image or video animation or the like created in advance based on the content of each page of the picture book, where the content in the image or video animation is not limited to the form of characters, pictures or the like. For example, if a certain page of picture book contains a process of turning a caterpillar into a butterfly, the projection picture can be a video animation of turning the caterpillar into the butterfly.

The scene shown in fig. 1 further includes an audio playing device 105, and when the projecting device 104 performs projection, the audio corresponding to the rendering may be synchronously played through the audio playing device 105, so as to achieve an effect of drawing sound and drawing color, for example, the audio of a story described in the rendering may be played, and music may also be played. The audio playback device 105 may also be integrated in a terminal device.

It should be noted that the method for recognizing a sketch book provided in the embodiment of the present application may be executed by a terminal device, and accordingly, the device for recognizing a sketch book 104 may be disposed in the terminal device. The method for recognizing a sketch book in the embodiment of the present application may also be executed by a server in communication connection with a terminal device to improve processing efficiency, and accordingly, the sketch recognition apparatus 104 may be disposed in the server.

The server may be hardware or software. When the server is hardware, it may be implemented as a distributed server cluster formed by multiple servers, or may be implemented as a single server. When the server is software, it may be implemented as multiple pieces of software or software modules (e.g., software or software modules used to provide distributed services), or as a single piece of software or software module. And is not particularly limited herein.

Exemplary method

The following describes the technical solution provided in the embodiment of the present application with reference to the application scenario shown in fig. 1. It should be noted that the above application scenarios are only presented to facilitate understanding of the spirit and principles of the present application, and the embodiments of the present application are not limited in this respect. Rather, embodiments of the present application may be applied to any scenario where applicable.

As shown in fig. 2, an implementation flow diagram of the picture book identification method provided in the embodiment of the present application may include the following steps:

s201, acquiring a shot image collected by image collecting equipment.

During specific implementation, a user can place the picture book to be identified in a shooting area of the image acquisition equipment so as to ensure that the shot image contains a complete picture book page.

S202, a target picture template matched with the shot image is determined from the plurality of picture templates.

The picture book template in the embodiment of the application refers to an image obtained by scanning each page in a picture book, or an image without a background obtained by shooting each page in the picture book.

In specific implementation, the characteristics of the shot image can be extracted, the characteristics of the shot image are respectively compared with the characteristics of each picture book template, and the picture book template with the highest characteristic similarity is determined as the template picture book template. The characteristics of each picture template can be extracted in advance, and comparison is convenient. The feature points or feature vectors in the image may be extracted from the captured image, for example, the feature points in the image may be extracted by using a feature point extraction algorithm such as orb (organized FAST and Rotated bright transform), SIFT (Scale-invariant feature transform), and AKAZE (invariant feature transform), or the high-dimensional feature vectors of the image may be extracted through a trained neural network.

And S203, determining the position information of the picture book in the shot image in the image coordinate system corresponding to the image acquisition equipment.

The image coordinate system in the embodiment of the present application is used to describe coordinates of each point in space relative to the image acquisition device.

In specific implementation, the corresponding coordinates of the plurality of key points of the picture book in the image coordinate system can be determined as the position information. Specifically, the vertices of the four corners of the sketch may be selected as key points for determining the position information.

And S204, determining the projection position of the position information relative to the projection equipment in the step S203 according to the coordinate mapping relation between the image coordinate system and the projection coordinate system corresponding to the projection equipment, so that the projection equipment projects the projection picture corresponding to the target book template onto the target book according to the projection position.

The projection coordinate system in the embodiment of the present application is used to describe coordinates of each point in space relative to the projection device.

In specific implementation, a corresponding projection picture needs to be specified for each picture template in advance.

In practical application, after the relative positions of the image acquisition device and the projection device are fixed, the coordinate mapping relation between the image coordinate system and the projection coordinate system can be determined, so that the coordinates under the image coordinate system and the projection coordinate system are converted according to the coordinate mapping relation.

Specifically, after the relative positions of the image acquisition device and the projection device are fixed, the coordinate mapping relationship can be determined by calibrating the pixel conversion relationship between the image acquisition device and the projection device. For example, firstly, a checkerboard pattern or other patterns which facilitate the extraction of accurate corner points can be projected on a desktop through a projection device, and the projected pattern can refer to the pattern shown in fig. 3; then, acquiring a projection image containing a projection pattern through image acquisition equipment, and extracting an angular point of the projection pattern in the projection image; then, matching each corner point labeled in the projection pattern in advance with a corner point extracted from the projection image, adding two matched corner points serving as a corner point pair into a corner point pair set, wherein the number of the corner point pairs added into the corner point pair set is not less than four, and determining a homography matrix based on any four corner point pairs in the corner point pair set; and finally, fitting a homography matrix of the image acquisition equipment and the projection equipment by using a least square method to obtain a coordinate mapping relation from the image coordinate system to the projection coordinate system, and recording the coordinate mapping relation as ColToPro, and correspondingly obtaining a coordinate mapping relation from the projection coordinate system to the image coordinate system, and recording the coordinate mapping relation as ProToCol.

The picture book identification method provided by the embodiment of the application determines the target picture book template matched with the shot image from the plurality of picture book templates, determines the position information of the picture book in the shot image in the image coordinate system corresponding to the image acquisition equipment, and then determines the projection position of the position information relative to the projection equipment according to the coordinate mapping relation between the image coordinate system and the projection coordinate system corresponding to the projection equipment.

Further, in order to improve matching efficiency of the picture book template, the embodiment of the application provides a multi-level resolution matching mode, and the target picture book template matched with the shot image is determined from the multiple picture book templates.

Referring to fig. 4, as a possible implementation manner, step S202 specifically includes:

s401, respectively extracting feature points of the shot image under N resolution and feature vectors of each feature point.

In the embodiment of the present application, N is an integer greater than or equal to 2, and a specific value of N may be determined according to an actual situation, which is not limited in the embodiment of the present application.

For example, it is assumed that the original resolution of the captured image is 1280 × 720, which is defined by the image pickup apparatus. When N is 2, it is possible to extract the feature points and the feature vectors of the captured image of a resolution of 1280 × 720, and extract the feature points and the feature vectors of the captured image of which the original resolution is down-sampled by one time (i.e., a resolution of 640 × 360). When N is 3, the feature points and the feature vectors of the captured images of the three resolutions 1280 × 720, 640 × 360, and 320 × 180 can be extracted, respectively. For the corresponding sampling resolution when N takes other values, no further description is given.

In specific implementation, feature points in the shot image at the resolution of N and feature vectors of each feature point can be extracted by using feature point extraction algorithms such as ORB, SIFT and AKAZE. In order to distinguish from the feature vector of the image, in the embodiment of the present application, the feature vector of the feature point is referred to as a descriptor.

In specific implementation, a delf (deep Local features) method can be used for extracting feature points in the shot image at the resolution of N. It is worth mentioning that the DeLF method can effectively suppress the interference of the projection.

Referring to fig. 5, the feature extraction model used in the DeLF method includes: the Full Convolution Network (FCN) is used for extracting feature vectors from the picture template, and the convolution network is used for screening candidate feature points from feature points corresponding to the feature vectors output by the full convolution network. The full convolution network can be a network such as VGG, ResNet and the like without a full connection layer, the full convolution network outputs a plurality of descriptors (namely feature vectors), and the center of each descriptor receptive field is the position of a feature point. The full convolution network can be connected with a two-layer convolution network so as to screen the feature points corresponding to the feature vectors output by the full convolution network. Specifically, the two layers of convolutional networks output and input parameters with the same dimensionality, play a role in average pooling, calculate the pooling weight corresponding to each feature point, and use the feature points and descriptors with the pooling weight larger than a certain threshold as the extracted feature points and descriptors.

After the feature extraction model shown in fig. 5 is constructed, the feature extraction model is trained based on a drawing dataset, wherein the drawing dataset includes a large number of drawing templates and a large number of projection pictures, and corresponding feature points are marked in the drawing templates and the projection pictures. The feature extraction model shown in fig. 5 is trained through the picture book data set to obtain a model which can be better suitable for extracting the feature points of the picture book, and meanwhile, the model has certain projection inhibition capability. Then, inputting the shot images with the N resolutions into the trained feature extraction model respectively, so as to obtain feature points corresponding to the shot images with the N resolutions and descriptors corresponding to the feature points.

S402, according to the sequence from low resolution to high resolution, matching the feature vector corresponding to the feature point of the shot image extracted under each resolution with the feature vector corresponding to the feature point of each picture template extracted under the corresponding resolution, and if the picture template matched with the shot image is determined, determining the matched picture template as the target picture template.

It should be noted that, after the step S401 is executed, the step S402 is executed, and in the step S402, once the target template is matched at a certain resolution, the matching is stopped immediately, for example, when the feature points extracted at the minimum resolution of 640 × 360 are matched, the feature points extracted at other resolutions higher than the resolution of 640 × 360 are not required to be matched when the target template is matched. Of course, in order to further improve the processing efficiency, step S401 and step S402 may also be executed in parallel, that is, the feature points and the corresponding descriptors of the captured images at N resolutions are sequentially extracted in the order from low resolution to high resolution, each time the feature points and the descriptors of the captured images at one resolution are extracted, the descriptors corresponding to the feature points of the captured images extracted at the resolution are immediately matched with the descriptors corresponding to the feature points of each of the sketch templates extracted at the corresponding resolution, and once an appropriate sketch template is matched, the matched sketch template is determined as the target sketch template, and at this time, the step of extracting the feature points and the matching descriptors may be stopped.

In specific implementation, before the picture book identification, the feature points corresponding to each picture book template at N resolution and the descriptors corresponding to each feature point may be extracted in advance. For example, there are 100 total sketch templates, where N is 2, feature points and descriptors corresponding to two resolutions need to be extracted for each sketch template, and feature points and descriptors extracted by one sketch template at one resolution are stored in one sketch feature point set, so that a total of 200 sketch feature point sets can be obtained.

Specifically, feature points corresponding to each template at N resolutions and descriptors corresponding to each feature point may be respectively obtained by using a feature point extraction algorithm such as ORB, SIFT, akage, or the like, or a feature extraction model shown in fig. 5. When the feature extraction model is adopted to extract candidate feature points and feature vectors corresponding to the candidate feature points from the picture book template, the picture book template is input into the trained feature extraction model to obtain the feature vectors corresponding to the feature points and the feature points.

When more feature points are extracted, the first K feature points with the highest score can be selected for the feature points of each picture book template under each resolution, and the screened K feature points and the corresponding descriptors are stored in the corresponding picture book feature point set so as to be used in the subsequent picture book identification process. The score of the feature point refers to whether the point is a criterion of the feature point by various feature point extraction methods, for example, the score of the feature point may be a harris response value in the ORB algorithm, or a pooling weight in the DeLF method.

According to the picture book identification method, the multi-level resolution matching strategy is adopted in the picture book identification process, the identification efficiency of the matching mode based on the low-resolution images is high, more than 90% of picture books with abundant textures can be rapidly identified, the accuracy of the matching mode based on the high-resolution images is high, 10% of picture books with fewer or smaller textures can be accurately identified, and two indexes of the identification efficiency and the identification accuracy can be considered simultaneously.

Further, in order to reduce the interference caused by projection in the process of drawing book identification, when extracting the feature points of the drawing book template, the feature points in the drawing book template may be extracted by any of the above methods, and then the extracted feature points are screened by the projection picture, and the feature point with the minimum interference from projection is retained, that is, the feature point with a larger degree of distinction from the feature points in the drawing book template and the projection picture is retained.

Firstly, feature points corresponding to each projection picture under N resolution and descriptors corresponding to each feature point need to be extracted. The specific extraction process may refer to a method for extracting feature points and descriptors in the sketch template, and is not described in detail.

Specifically, referring to fig. 6, the feature points corresponding to each template at each resolution may be obtained as follows:

s601, extracting candidate characteristic points and characteristic vectors corresponding to the candidate characteristic points from the picture book template.

The specific process of extracting the feature points and the descriptors has been described in the foregoing embodiments, and is not described herein again.

S602, aiming at each candidate feature point of the picture template, calculating a distance value between a feature vector of the candidate feature point and a feature vector of each feature point in each projection picture, and determining the candidate feature point corresponding to the minimum distance value as a screening feature point.

S603, obtaining K screening feature points in the front of the sequence according to the sequence from large to small of the distance values corresponding to the screening feature points, and determining the K screening feature points as the feature points corresponding to the picture book template.

In particular, it is not assumed that the jth template is at resolution P_iThe candidate feature points extracted below include C_j＝{c₁,c₂,..,c_nThe descriptor of the drawing template comprises D_j＝{d₁,d₂,…,d_nWhere, the candidate feature point p_iThe corresponding descriptor is d_iI is 1,2, … n, and descriptors corresponding to feature points of all projection pictures include T ═ { T ═ T₁,t₂,…,t_mThe detailed steps are as follows:

first, calculate d₁Selecting the minimum distance value from the distance values of all descriptors in T and recording the minimum distance value as dis₁To obtain the characteristic point p₁Corresponding dis₁By analogy, p is calculated₂,..,p_nRespectively corresponding minimum distance values dis₂,..,dis_n。

Second, pair dis in descending order₁,dis₂,..,dis_nSorting, obtaining K minimum distance values at the front of the sorting, wherein the characteristic points corresponding to the K minimum distance values are the resolution P of the drawing template_iAnd (5) next corresponding feature points.

In practical application, K may be determined according to a specific application scenario, and the embodiment of the present application is not limited. For example, K may take on values of 1000, 2000, etc.

In specific implementation, the feature points corresponding to each projection template at each resolution can be screened in the following manner: and aiming at the feature points of each projection picture under each resolution, selecting the first K feature points with the highest score as the feature points of the projection picture under the resolution, and storing the screened feature points and descriptors corresponding to the feature points into a corresponding projection feature point set so as to be used in the subsequent picture book identification process.

For each resolution ratio picture book template, the corresponding characteristic points and descriptors can be extracted and screened out by adopting the method. It should be noted that the value of K for each resolution may be the same, or may vary with the resolution, for example, if the resolution is 1280 × 720 for K, the resolution is 640 × 360 for K/2, the resolution is 320 × 180 for K/4, and so on.

Because the condition of page turning exists in the process of drawing the book is identified, and a previous projection picture can be remained in the shot image during page turning, in order to ensure the high reliability of the screened feature points, descriptors of all projection pictures are adopted to screen the feature points of the drawing template. Of course, if the effect of the drawing template identification is relatively reliable, for each drawing template, the feature points of the drawing template may be screened only by using the drawing template and descriptors in the projection pictures corresponding to the front and back several pages of the drawing template, or even the feature points of the drawing template may be screened only by using descriptors in the projection pictures corresponding to the drawing template.

By the mode shown in fig. 6, the feature points in the picture template which are easily interfered by the projection picture can be removed, so that the interference of the projection picture on the recognition result is reduced in the subsequent picture recognition process, and the recognition accuracy is improved.

On the basis of any of the above embodiments, referring to fig. 7, step S402 may be implemented in the following manner, specifically including:

s701, determining the resolution P of the current processing according to the sequence from low resolution to high resolution_n。

Wherein the resolution P_n-1Specific resolution P_nOne level higher. For example, P₁For the resolution of the original image, P₂Down-sampling the original resolution by one time, P₃Is resolution P₂And (5) sampling the resolution doubled, and so on. P_nWhere N is initialized to the number of resolutions N, i.e. P_NAt the lowest resolution, P₁The highest resolution.

S702, determining the currently processed picture book template T according to the matching sequence of the picture book templates_m。

Wherein M is a positive integer, M is less than or equal to M, and M is the total number of the template. Each time step S702 is executed, a value of m is determined according to the matching sequence of the picture book template, where m may be a page number indicating a position of the picture book template in the whole picture book, for example, T₁For first page drawing template, T₂For first page drawing template, T_mAnd drawing the template for the mth page.

S703, aiming at the shot image at the resolution P_nExtracting each feature point, and respectively calculating the feature vector corresponding to the feature point and the picture book template T_mAt resolution P_nSimilarity between the feature vectors corresponding to each extracted feature point is extracted, and if the minimum similarity meets a preset condition, the feature point i of the shot image corresponding to the minimum similarity and the picture template T are compared_mForms a feature point pair (i, j) and is added to the picture template T_mThe corresponding point pair sets.

In specific implementation, the similarity between two characteristic vectors can be calculated by algorithms such as Euclidean distance, Manhattan distance, Chebyshev distance, included angle cosine distance and the like.

In specific implementation, the preset condition may be the minimum similarity d₁Less than the similarity threshold. The preset condition may also be a minimum similarity d₁≤d₂And d is₁≤thre5*d₂，d₂The second smallest similarity, where thre5 is in the range of (0,1), generally thre5 is 0.9, and the smaller the thre5 is, the stricter the value is. Of course, other preset conditions may also be selected, and the embodiment of the present application is not limited.

S704, judging the picture book template T_mWhether the number of the point pairs in the corresponding point pair set is smaller than a first number threshold value or not, if yes, determining the picture book template T_mIf not, executing step S705, otherwise, determining the picture template T_mMatching with the captured image, step S708 is executed.

S705, judging the resolution P_nAnd (4) whether the matching of the lower picture book templates is finished, if so, executing the step S706, and if not, returning to the step S702.

In step S702, the process returns, that is, the matching of the sketch template of the next process is performed according to the matching sequence of the sketch templates.

S706, judging whether n is equal to 1, if yes, executing step S707, if no, subtracting 1 from the value of n, and returning to step S701.

And S707, determining that the target picture template is not matched.

In specific implementation, if the target picture book template is not matched, the next frame of shot image collected by the image collecting equipment can be obtained for identification.

S708, drawing the template T_mThe template is determined for the eye.

Further, based on the method shown in FIG. 7, in order to improve the accuracy of the matching objective plotting template, if the plotting template T is determined_mMatching the photographed image, the template T may be further processed before the step S708 is executed_mPerforming geometric verification, if the geometric verification is passed, executing step S708 to draw the template T_mDetermining a target template; otherwise, the matching of the picture book template is continued until the target picture book template is matched or the picture book templates with all resolutions are matched, and step S705 is executed.

Specifically, referring to fig. 8, the notebook template T may be plotted in the following manner_mPerforming geometric verification, specifically comprising the following steps:

s801, drawing the template T from the book_mAnd acquiring a preset number of point pairs from the corresponding point pair set.

In specific implementation, a homography matrix can be determined by at least four point pairs, so the minimum value of the preset number is 4.

S802, according to the obtained preset number of point pairs, determining a characteristic shot image and a picture book template T_mA homography matrix of coordinate mapping relations between them.

The specific process of calculating the homography matrix is the prior art and is not described in detail.

S803, aiming at the picture book template T_mDetermining the characteristics of each point pair (i, j) in the corresponding point pair set according to the homography matrixPoint i in the picture book template T_mAnd calculating the expected coordinates of the characteristic point i and the expected coordinates of the characteristic point j in the picture template T_mThe distance value between the coordinates in (1).

S804, determining whether the number of distance values smaller than the error threshold is greater than a second number threshold, if so, performing step S805, and if not, performing step S806.

The error threshold and the second quantity threshold may be determined by those skilled in the art according to the test result and the specific application scenario, and the embodiments of the present application are not limited thereto.

S805, determining the picture book template T_mPassing the geometric verification.

In specific implementation, if the picture book template T is determined_mThrough geometric verification, the template T of the drawing book can be stored_mAnd the homography matrix H passing the geometric verification is used for determining the position information of the sketches in the shot images in the image coordinate system.

S806, judging the template T for the picture book_mIf the number of times of verification exceeds the first threshold, if yes, step S807 is executed, and if not, step S802 is returned to.

S807, determining the picture book template T_mFailed geometric verification.

Wherein, returning to step S802, the template T is drawn_mAnd acquiring a new preset number of point pairs from the corresponding point pair set, and performing geometric verification again, wherein the acquired new preset number of point pairs are not identical to the previously acquired preset number of point pairs.

On the basis of determining the homography matrix H, step S203 specifically includes: and determining the position information of the picture in the shot image in the image coordinate system according to the homography matrix H which enables the target picture template to pass through geometric verification and the target picture template.

In a specific implementation, the position information may be coordinates of vertices of four corners of the target sketch template.

Specifically, the length and width of the chart template are acquired, and it is not assumed that the length of the chart template is width and the width is height.

Firstly, according to homography HAnd calculating the coordinates of the vertexes of the four corners of the target drawing template in the shot image. Taking the vertex at the lower right corner as an example, the coordinate (x) in the captured image can be determined by the following formula₁,y₁,z₁)：

(x₁,y₁,z₁)^T＝H*(width,height,1)^T。

Then, coordinates of vertices of four corners of the subject template in the image coordinate system are calculated. Taking the vertex of the lower right corner as an example, the coordinates in the image coordinate system are:

(x_col,y_col)^T＝(x₁/z₁,y₁/z₁)^T。

the corresponding coordinates of the vertexes of the other three angles of the target plotting template in the image coordinate system are calculated in a similar manner, and are not repeated.

After the coordinates corresponding to the vertexes of the four corners of the target plot template in the image coordinate system are obtained, the coordinates of the vertexes of the four corners of the target plot template in the projection coordinate system can be calculated. Specifically, the projection position of the position information relative to the projection device may be determined according to a coordinate mapping relationship between the image coordinate system and a projection coordinate system corresponding to the projection device. Taking the vertex of the lower right corner as an example, the coordinate (x) of the vertex can be determined in the projection coordinate system by the following formula_pro,y_pro)^T：

(x₂,y₂,z₂)^T＝ColToPro*(x_col,y_col,1)^T

(x_pro,y_pro)^T＝(x₂/z₂,y₂/z₂)^T

The corresponding coordinates of the vertexes of the other three angles of the target plotting template in the projection coordinate system are calculated in a similar manner, and are not repeated.

Like this, projection equipment just can be according to the projection position (being the target and painted the coordinate of summits under the projection coordinate system at four angles of this template), draws the projection picture projection that this template corresponds with the target to draw on this, even if the position of drawing this in the use has taken place to remove, also can guarantee that the projection picture can accurately project to draw on this or draw this peripheral assigned position department for the projection picture can be better with draw this combination of body, improve and read the interest of drawing this.

In specific implementation, the matching sequence of the sketch template can be determined by any one of the following methods:

in the first mode, the matching sequence of the picture book template is determined according to the sequence of the corresponding page numbers of the picture book template from small to large.

Specifically, the matching order may be { T }₁，T₂，……，T_MThat is, first match the first page of this template T₁Then matching with the second page picture book template T₂And so on.

In the second mode, the matching sequence of the picture book template is determined according to the similarity between the coding vector of the picture book template and the coding vector of the shot image.

Specifically, for each resolution, the similarity between the coding vector of each picture template at the resolution and the coding vector of the shot image is calculated respectively, and the matching order of the picture templates is determined according to the sequence from high to low of the similarity of the coding vectors.

For each picture template at each resolution, the corresponding encoding vector is determined according to the descriptor corresponding to the feature point extracted by the picture template at the resolution, for example, the descriptors corresponding to all the feature points extracted by the picture template at the resolution can be encoded by using a coding method such as VLAD, FV, and the like, so as to obtain the encoding vector. The descriptors corresponding to the characteristic points of the shot image can also be encoded by the encoding method to obtain the encoded vectors of the shot image, and the specific encoding process is the prior art and is not repeated.

And in the third mode, the matching sequence of the sketch template is determined according to the target sketch template matched before.

Specifically, determining a matching sequence of the sketch templates according to the target sketch template matched previously includes: determining the page turning direction of the drawing according to the page number of the at least one recently matched target drawing template; and determining the matching sequence of the picture book template according to the page turning direction of the picture book and the page number of the target picture book template matched last time.

The page turning direction comprises a forward page turning and a backward page turning. For example, if the two most recently matched textbook templates are respectively page 5 and page 6, and the page 5 is matched first and then the page 6 is matched, determining that the page turning direction is backward page turning; and if the two most recently matched picture book templates are respectively the 8 th page and the 7 th page, the 8 th page is matched firstly, and the 7 th page is matched later, determining that the page turning direction is forward page turning. For example, if the last matching sketch template is the last page of the whole sketch, the page turning direction is determined as page turning forward. The specific rule for judging the page turning direction can be set according to the actual application scene, and the embodiment of the application is not limited.

Further, determining the matching sequence of the picture book template according to the page turning direction of the picture book and the page number of the target picture book template matched last time, specifically comprising: if the page turning direction is forward page turning, the template of the picture book before the picture book of the current page is preferentially matched, and if the page turning direction is backward page turning, the template of the picture book after the picture book of the current page is preferentially matched.

As a possible implementation, the matching order of the sketch template can be determined by the following formula:

when m is z-1 or m < n, then there are:

when m is more than or equal to n, the following are:

in the above formula, z is the total page number of the picture book template, m is the page number of the target picture book template that has been matched last time, n is the page number of the target picture book template that has been matched last time, and m is the page number of the target picture book template that has been matched last time<n represents that the page turning direction is forward page turning, m is more than or equal to n represents that the page turning direction is backward page turning, and O_iK represents the i +1 th matching page numberIs a target template of k +1, i is an integer, and i is more than or equal to 0 and less than or equal to z-1.

For rounding-down operators, e.g.

% is a remainder operator, e.g., 19% 10-9, 4% 10-4. And if the target picture book template which is matched for the last time is not matched with the target picture book template before, n is 0.

For example, when z is 10, m is 8, and n is 9, m is applicable<In the case of n, then O₀～O₉The values of (a) are 7, 8, 6, 9, 5, 0, 4, 1, 3 and 2 in sequence, that is, the matching sequence of the drawing template is page 8, page 9, page 7, page 10, page 6, page 1, page 5, page 2, page 4 and page 3. When z is 10, m is 3, and n is 2, m ≧ n is applicable, O₀～O₉The values of (a) are 4, 3, 5, 2, 6, 1, 7, 0, 8 and 9 in sequence, that is, the matching sequence of the drawing template is pages 5, 4, 6, 3, 7, 2, 8, 1, 9 and 10.

The above formula is only an example of determining the matching order of the sketch template, and in specific implementation, other formulas may also be used to determine the matching order of the sketch template, and the embodiment of the present application is not limited.

The method of the embodiment of the application utilizes the priori knowledge that pages can be turned in sequence under most conditions in the process of using the picture book to determine the matching sequence of the picture book templates, improves the probability that the previous picture book templates are the target picture book templates in the matching process, and greatly reduces the processing time consumption.

In specific implementation, if the target template is not determined based on the previous F-frame captured image, the matching sequence may be determined in the first manner or the second manner, otherwise, the matching sequence may be determined in the third manner. The value of F may be determined according to an actual application scenario, for example, F may be 30.

Further, in order to improve the processing efficiency of determining the matching order of the sketch templates, the lowest resolution P may be determined by any of the above-mentioned methods_NMatching order of corresponding picture template based on lowest resolution P_NThe corresponding matching order determines the resolution P_N-1The corresponding matching order is based on the resolution P_N-1Determining resolution P according to the matching sequence_N-2And determining the matching sequence corresponding to each resolution in turn according to the corresponding matching sequence.

Specifically, referring to fig. 9, the matching order of the sketch templates corresponding to each resolution may be determined by the following manner:

s901, aiming at each picture template, according to the resolution ratio P_nThe obtained number and resolution P of the point pairs contained in the point pair set corresponding to the sketch template_nAnd determining the ranking score of the drawing template according to the position of the drawing template in the matching sequence of the corresponding drawing template.

For example, at a resolution P_nThe number of the point pairs contained in the point pair set corresponding to the obtained sketch template is y, and the sketch template has a resolution ratio P_nIf the bit in the corresponding matching order is I, the ordering score of the template is y-I × th, where th is a non-negative real number, e.g., th may be 0 or 1.

S902, determining the resolution P according to the sequence of the ranking scores from high to low_n-1Matching sequence of corresponding picture template.

Wherein the resolution P_n-1Specific resolution P_nOne level higher. For example, P_n-1When 1280 × 720, P_n＝640×360。

The mode shown in fig. 9 will be described by taking two resolutions as an example. Firstly, determining the matching sequence of the low-resolution lower picture template through any one of the three modes; then, in the manner shown in fig. 9, for each template, the ranking score of the template is determined according to the number of pairs included in the pair set corresponding to the template obtained at the low resolution and the rank of the template in the matching order at the low resolution, the ranking of each template is determined in the order from high to low of the ranking score, and the ranking is used as the matching order of the template at the high resolution.

On the basis of any of the above embodiments, before matching the feature vector corresponding to the feature point extracted from the captured image with the feature vector corresponding to the feature point of each picture template, the method of the embodiment of the present application further includes the following steps: and respectively filtering the feature points of the shot image extracted under each resolution according to the feature points of the projection picture currently projected by the projection equipment to filter out the feature points which are not interfered by the projection picture, and matching based on the filtered feature points.

Referring to fig. 10, the step of filtering, for the feature points of the captured image extracted at each resolution, according to the feature points of the projection picture currently projected by the projection apparatus includes:

s1001, acquiring a first coordinate of each feature point in a projection coordinate system of the projection picture projected currently.

S1002, determining a second coordinate of each feature point in the projection picture of the current projection in the image coordinate system according to the coordinate mapping relation and each first coordinate.

And S1003, drawing a circle with the second coordinate as the center and r as the radius in a mask with the same size as the shot image according to each second coordinate.

The same size means that the length and the width of the mask and the shot image are the same, and the number of the contained pixel points is the same.

S1004, the mask is overlaid on the captured image, and the feature points in the circle in the mask in the captured image are deleted.

Referring to fig. 11, a mask 1102 having the same size as the captured image 1101 is constructed, and the initial value of each pixel in the mask 1102 is 0. For each feature point (x) in the projection picture of the current projection₁,y₁) Adding the coordinate (x) of the top left corner vertex of the projection picture of the current projection in the projection coordinate system₀,y₀) Obtaining a feature point (x)₁,y₁) First coordinate (x) in a projection coordinate system₂,y₂) I.e. (x)₂,y₂)＝(x₁,y₁)+(x₀,y₀) Then, a first coordinate (x) is calculated₂,y₂) Second coordinate (x) in image coordinate system₃,y₃,z)^T：

(x₃,y₃,z)^T＝ProToCol*(x₂,y₂,1)^T。

Next, for each second coordinate, the coordinate (x) is drawn in the mask 1102₃/z,y₃Z) is a circle 1104 with a center 1103 and a radius r. The value of r is related to the number of pixel points included in the shot image, the unit of r is a pixel point, for example, when the value of r is 2, the coordinate (x) is represented₃/z,y₃/z) are all within a circle, and the value of the pixel in the circle in the mask 1102 is set to 1. For convenience of illustration, in fig. 11, the pixel point with the value set to 1 in the mask is gray, and the value 0 is white. Finally, the mask 1102 is covered on the shot image 1101, and the feature points in the circle in the mask 1102 in the shot image 1101 are filtered, that is, if the pixel points where the feature points in the shot image 1101 are located are covered by the pixel points with the numerical value of 1 in the mask 1102, the feature points are filtered, and if the pixel points where the feature points in the shot image 1101 are located are covered by the pixel points with the numerical value of 0 in the mask 1102, the feature points are reserved. Taking fig. 11 as an example, the feature point Q in the captured image 1101₃And characteristic point Q₄If covered by a pixel with a value of 1 in the mask 1102, Q is filtered₃And Q₄. Keeping only the characteristic points Q₁And Q₂。

First, coordinates of vertices of four corners of the subject template in the captured image are calculated from the homography matrix H. Taking the vertex at the lower right corner as an example, the coordinate (x) in the captured image can be determined by the following formula₁,y₁,z₁)：

(x₁,y₁,z₁)^T＝H*(width,height,1)^T。

(x_col,y_col)^T＝(x₁/z₁,y₁/z₁)^T。

(x₂,y₂,z₂)^T＝ColToPro*(x_col,y_col,1)^T

(x_pro,y_pro)^T＝(x₂/z₂,y₂/z₂)^T

Like this, projection equipment just can be according to the projection position (being the target and drawing the coordinate of summits under the projection coordinate system at four angles of this template), draws the projection picture projection that this template corresponds to the target to drawing originally, even if the position of drawing originally in the use has taken place to remove, also can guarantee that the projection picture can accurately project to drawing originally or draw this peripheral assigned position department for the projection picture can be better with draw this combination of oneself, improve and draw this in-process interest.

Further, after the coordinates of the vertexes of the four corners of the target plotting template in the projection coordinate system are determined, the coordinates of each feature point corresponding to the target plotting template can be converted into the image coordinate system by using the homography matrix H corresponding to the target plotting template. Then, one dimension is added to the two-dimensional coordinates of the feature points of the template of the target picture book, and the numerical values of the added dimensions are all marked as 1, for example, the three-dimensional coordinates of the feature points (x, y) after one dimension is added are (x, y,1), and are converted into an extended matrix [ R | t ] of a PNP problem (3D → 2D) in a SLAM (simultaneous localization and mapping) algorithm to initialize an SVO (sparse direct method) tracker, where R represents a rotation matrix, and t represents a displacement vector, which refers to the position and posture (including two dimensions of rotation and displacement) of the image acquisition device relative to the shot picture book, and can also be converted into the position and posture of the picture book relative to the image acquisition device, and the position information of the picture book in the image coordinate system can be determined based on the position and posture. The SVO tracker is an efficient attitude tracking algorithm in the SLAM algorithm, and is not described in detail for the prior art.

In order to improve the tracking efficiency, the feature points of the target rendering template with the lowest resolution may be used to determine the augmentation matrix [ R | t ] to initialize the SVO tracker.

On the basis of any one of the above embodiments, in order to improve the processing efficiency of the picture book identification method, the picture book identification method in the embodiment of the present application may further incorporate a tracking algorithm, and after the target picture book template is identified by the identification method in any one of the above embodiments, the target picture book template in the captured image acquired by the image acquisition device in real time may be subsequently tracked; if the target picture book template is tracked in the shot image collected in real time, picture book identification is not needed; if the target picture template is not tracked, it indicates that the user has turned the page, and based on the picture recognition method provided in any of the above embodiments, the currently acquired shot image is re-subjected to picture recognition, and a new target picture template is determined, so that the projection device is controlled to project a projection picture corresponding to the new target picture template.

Specifically, referring to fig. 12, an embodiment of the present application provides a picture book identification method, which specifically includes the following steps:

and S1201, acquiring a shot image acquired by the image acquisition equipment.

And S1202, determining a target picture template matched with the shot image from the plurality of picture templates.

And S1203, determining position information of the picture book in the shot image in an image coordinate system corresponding to the image acquisition equipment.

And S1204, determining the projection position of the position information determined in the step S1203 relative to the projection equipment according to the coordinate mapping relation between the image coordinate system and the projection coordinate system corresponding to the projection equipment, so that the projection equipment projects the projection picture corresponding to the target book template onto the target book according to the projection position.

The specific implementation of steps S1201 to S1204 can refer to steps S201 to S204, and will not be described again.

And S1205, acquiring a next frame of shot image acquired by the image acquisition equipment in real time, and tracking the feature points of the target plotting template in the acquired next frame of shot image.

And S1206, judging whether the successfully tracked feature points meet preset requirements, if so, executing the step S1207, otherwise, executing the step S1201 again.

If the successfully tracked feature points meet the preset requirement, the user is indicated to turn pages, and matching of the picture template needs to be performed again. Returning to step S1201 means acquiring a new captured image captured by the image capturing apparatus again, performing picture book recognition based on the new captured image, and determining a new target picture book template.

S1207, judging whether the picture book in the shot image moves, if so, executing a step S1208, otherwise, returning to the step S1205.

If the picture book in the shot image is not moved, it indicates that the user does not move the picture book, and the projection position of the projection picture does not need to be adjusted, at this time, the process may return to step S1205, obtain the next shot image, and continue to track.

And S1208, determining the transformation offset of the picture book in the shot image relative to the image acquisition equipment according to the tracking result, and re-determining the projection position of the picture book in the shot image relative to the projection equipment according to the transformation offset so that the projection equipment adjusts the position of the projection picture corresponding to the picture book template projected onto the picture book according to the re-determined projection position.

When the page is not turned, the tracking algorithm can determine the new position of the picture book in real time, so that the new projection position is determined, and the effect of tracking the projection is achieved.

After step S1208 is executed, the process returns to step S1205, and the next captured image is acquired to continue tracking.

In particular, feature points of the target template may be tracked based on an SVO tracker that tracks key points in the SLAM algorithm.

As a possible implementation, referring to fig. 13, the method for tracking the feature points of the target template specifically includes the following steps:

and S1301, tracking a block corresponding to the feature point of the target picture book template based on the SVO tracker.

In specific implementation, the blocks corresponding to the feature points of the target template can be determined by the following method: and determining the coordinates of each characteristic point of the eye plotting template in the shot image by using the homography matrix H which enables the eye plotting template to pass through geometric verification, and respectively acquiring image blocks in a preset range around each coordinate as the blocks corresponding to each characteristic point. The captured image in this step refers to the captured image acquired in S1201 before tracking is performed.

S1302, obtaining transformation offset [ R ] of position and posture of the image acquisition equipment based on the tracking result₁|t₁]And the number of blocks successfully trackedAmount of the compound (A).

And S1303, judging whether the target plot template moves or not according to the number of the successfully tracked blocks and the number of the successfully tracked blocks based on the last frame of shot image.

Specifically, assuming that the number of blocks successfully tracked based on the previous frame of captured image is n, and the number of blocks successfully tracked based on the current frame of captured image is m, if | n-m |/m > thre, it is considered that the user has performed a page turning operation or moved too strongly, and the target picture template cannot be tracked. The value of the threshold thre may be adjusted by a person skilled in the art according to the actual application scenario and the recognition accuracy, for example, when thre is 0.2, it indicates that if the fluctuation between the two previous and next frames of captured images exceeds 20%, it may be determined that the user has performed a page turning operation or the drawing has moved greatly.

In specific implementation, whether the target plotting template moves or not can be judged in the following modes: according to the shift amount [ R₁|t₁]And initializing an augmentation matrix [ R | t ] of the SVO tracker]Determining the current position and posture of the image acquisition equipment as [ RR ]₁|Rt₁+t]Using [ RR ]₁|Rt₁+t]Calculating the coordinates of the vertexes of the four corners of the picture book in the image coordinate system in the current processed shot image, calculating the coordinates Y of the vertexes of the four corners of the picture book in the projection coordinate system by utilizing the coordinate mapping relation, and assuming that the coordinates of the vertexes of the four corners of the picture book in the projection coordinate system in the shot image determined based on the previous frame of shot image are Y₀If Y and Y are₀If the error is less than thre0, the target template is considered to have not moved, otherwise, the target template is considered to have moved. The value of thre0 may be adjusted according to an actual situation, and the embodiment of the present application is not limited, for example, the value of the threshold thre0 may be 10.

It should be noted that, if the SVO tracker is used for tracking, after returning to step S1201, a new augmentation matrix [ R | t ] needs to be obtained based on the re-determined target picture template to re-initialize the SVO tracker, and then tracking is performed based on the re-initialized SVO tracker.

The method for identifying the painted book can locate the position of the painted book in real time in the handheld painted book moving process, can accurately detect the action of a user for turning the page of the painted book based on the tracking algorithm, can re-match the painted book template only after the page turning is detected, does not need to continuously match the painted book template, only carries out the tracking algorithm in most time, and ensures the low-load operation of equipment. Therefore, the method for recognizing the picture book in the embodiment of the application can well operate in the terminal device without depending on the server, and the problem of low processing efficiency caused by extra network communication between the terminal device and the server is solved. For example, when the graph recognition method shown in fig. 12 is run on a low-end mobile processor such as RK3288, the average response time is 200ms, and the recognition rate can reach more than 99%.

In addition, in the prior art, a frame difference method is usually adopted to determine whether to turn a page, but if the picture book is moved in the reading process or other objects enter the shooting area of the image acquisition device, misdetermination is caused, the accuracy of page turning identification is low, and the projection picture also affects the page turning identification, especially the projection for the animation. The picture book identification method provided by the embodiment of the application detects whether to turn pages or not based on the characteristic points of the tracking target picture book template, and is strong in anti-interference capability and higher in detection accuracy. In addition, thanks to the robustness of the SVO tracker, the vertices of the four corners of the sketch can be tracked when the user holds the sketch by hand to move, and the scene identified by the sketch is perfectly matched.

Exemplary device

Having described the method of the exemplary embodiment of the present application, the present invention recognition apparatus of the exemplary embodiment of the present application will be described next with reference to fig. 14, 15, and 16.

Fig. 14 is a schematic structural diagram of a picture recognition apparatus according to an embodiment of the present application. In one embodiment, the picture recognition device 140 includes: an image acquisition module 1401, a sketch matching module 1402, a sketch position determination module 1403, and a projection module 1404.

An image acquiring module 1401, configured to acquire a captured image acquired by the image acquiring apparatus.

A picture matching module 1402, configured to determine a target picture template matching the captured image from the plurality of picture templates.

A plotter position determining module 1403, configured to determine position information of a plotter in the captured image in an image coordinate system corresponding to the image capturing device.

A projection module 1404, configured to determine, according to a coordinate mapping relationship between the image coordinate system and a projection coordinate system corresponding to a projection device, a projection position of the position information relative to the projection device, so that the projection device projects, according to the projection position, a projection picture corresponding to the target book template onto the book.

Optionally, the picture matching module 1402 is specifically configured to:

respectively extracting the characteristic points of the shot image and the characteristic vector of each characteristic point under N resolution ratios;

sequentially matching the feature vector corresponding to the feature point of the shot image extracted under each resolution with the feature vector corresponding to the feature point of each picture template extracted under the corresponding resolution according to the sequence of the resolution from low to high;

and if the picture book template matched with the shot image is determined, determining the matched picture book template as the eye picture book template.

Optionally, the picture matching module 1402 is specifically configured to:

determining the current processing drawing template T according to the matching sequence of the drawing template_m；

At a resolution P for the captured image_nExtracting each feature point, and respectively calculating the feature vector corresponding to the feature point and the picture book template T_mAt resolution P_nSimilarity between feature vectors corresponding to each extracted feature point is calculated, and if the minimum similarity meets a preset condition, the feature point i of the shot image corresponding to the minimum similarity and the picture template T are compared_mForms a feature point pair (i, j) and is added to the picture template T_mCorresponding point pair sets;

if the picture book template T_mDetermining that the number of the point pairs in the corresponding point pair set is less than a first number threshold value_mIs not matched with the shot image and is drawn according to the drawing template T_mThe matching sequence of (1) matching the sketch template of the next processing, otherwise, determining the sketch template T_mAnd matching with the shot image.

Optionally, the sketch matching module 1402 is further configured to:

determining a matching template from the template T before determining the template as the target template_mAcquiring a preset number of point pairs from the corresponding point pair set;

determining the characterization of the shot image and the picture template T according to the preset number of point pairs_mHomography matrix of coordinate mapping relation;

for the picture book template T_mDetermining the characteristic point i in the picture template T according to the homography matrix for each point pair (i, j) in the corresponding point pair set_mAnd calculating the expected coordinates of the characteristic point i and the expected coordinates of the characteristic point j in the drawing template T_mA distance value between the coordinates in (1);

if the number of distance values smaller than the error threshold is larger than a second number threshold, determining the picture template T_mPassing geometric verification;

the picture matching module 1402 is specifically configured to: if the picture book template T is determined_mThrough geometric verification, the drawing template T is obtained_mThe target template is determined.

Optionally, the sketch matching module 1402 is further configured to:

determining the matching sequence of the picture book template according to the sequence of the page numbers corresponding to the picture book template from small to large; or

Respectively calculating the similarity between the coding vector of each picture template and the coding vector of the shot image, and determining the matching sequence of the picture templates according to the sequence of the similarity of the coding vectors from high to low, wherein the coding vector of each picture template is determined according to the characteristic points of each picture template, and the coding vector of the shot image is determined according to the characteristic points of the shot image; or

And determining the matching sequence of the sketch templates according to the target sketch template matched previously.

Optionally, the picture matching module 1402 is specifically configured to:

determining the page turning direction of the picture book according to the page numbers of at least two recently matched eye picture book templates;

and determining the matching sequence of the picture book template according to the page turning direction of the picture book and the page number of the target picture book template matched last time.

Optionally, the sketch matching module 1402 is further configured to:

for each template, according to the resolution P_nThe obtained number and resolution P of the point pairs contained in the point pair set corresponding to the sketch template_nDetermining the ranking score of the picture template according to the position of the picture template in the matching sequence of the corresponding picture template;

determining the resolution P in the order of the ranking scores from high to low_n-1Matching order of corresponding picture template, wherein resolution P_n-1Specific resolution P_nOne level higher.

Optionally, the atlas location determining module 1403 is specifically configured to determine the location information of the atlas in the captured image in the image coordinate system according to the homography matrix that makes the atlas template pass geometric verification and the atlas template.

Optionally, the sketch matching module 1402 is further configured to:

acquiring a first coordinate of each feature point in a projection coordinate system of the current projection picture before matching the feature vector corresponding to the feature point of the shot image extracted at each resolution with the feature vector corresponding to the feature point of each picture template extracted at the corresponding resolution in sequence from low resolution to high resolution;

determining a second coordinate of each feature point in the image coordinate system according to the coordinate mapping relation and each first coordinate;

drawing a circle with the second coordinate as a circle center and r as a radius in a mask with the same size as the shot image according to each second coordinate;

and covering the mask on the shot image, and deleting the characteristic points in the shot image, which are positioned in the circle in the mask.

Optionally, the chart recognition apparatus 140 further includes a tracking module 1405, configured to acquire the captured image acquired by the image acquisition device in real time after the chart template is determined, and track the feature point of the chart template in the acquired captured image.

Accordingly, projection module 1404 is specifically configured to:

if the successfully tracked feature points meet the preset requirement and the movement of the picture book in the shot image is determined, determining the transformation offset of the picture book in the shot image relative to the image acquisition equipment according to the tracking result;

and according to the transformation offset, re-determining the projection position of the picture book in the shot image relative to the projection equipment, so that the projection equipment adjusts the position of the projection picture corresponding to the eye book template projected on the picture book according to the re-determined projection position.

Optionally, the image obtaining module 1401 is further configured to obtain a new captured image collected by the image collecting device if the successfully tracked feature point does not meet a preset requirement;

the picture book matching module 1402 is further configured to determine a new target picture book template from the picture book templates according to the matching degree between the feature points extracted from the new shot image and the feature points of each picture book template;

a plotter position determining module 1403, further configured to determine position information of a plotter in the new captured image in an image coordinate system corresponding to the image acquisition device;

the projection module 1404 is further configured to determine, according to a coordinate mapping relationship between the image coordinate system and a projection coordinate system corresponding to the projection device, a projection position of the position information corresponding to the new captured image relative to the projection device, so that the projection device projects, according to the projection position corresponding to the new captured image, the projection picture corresponding to the new target picture template onto the picture book.

Optionally, the feature point corresponding to each sketch template is obtained as follows:

extracting candidate characteristic points and characteristic vectors corresponding to the candidate characteristic points from the picture template;

calculating a distance value between a feature vector of the candidate feature point and a feature vector of each feature point in each projection picture aiming at each candidate feature point of the picture template, and determining the candidate feature point corresponding to the minimum distance value as a screening feature point;

and acquiring K screening feature points in the front of the sequence according to the sequence from large to small of the distance values corresponding to the screening feature points, and determining the K screening feature points as the feature points corresponding to the picture template.

Optionally, the extracting the candidate feature points and the feature vectors corresponding to the candidate feature points from the picture book template specifically includes:

inputting the picture template into the trained feature extraction model to obtain candidate feature points and feature vectors corresponding to the candidate feature points;

the feature extraction model comprises a full convolution network and a convolution network, the full convolution network is used for extracting feature vectors from a picture book template, the convolution network is used for screening candidate feature points from feature points corresponding to the feature vectors output by the full convolution network, the feature extraction model is obtained based on a picture book data set in a training mode, and the picture book data set comprises a picture book template and a projection picture.

The device for recognizing the picture book and the method for recognizing the picture book provided by the embodiment of the application adopt the same inventive concept, can obtain the same beneficial effects, and are not repeated herein.

Based on the same inventive concept as the above-mentioned picture book identification method, the embodiment of the present application further provides a picture book identification apparatus, which may include a processor 1501 and a memory 1502, referring to fig. 15.

The Processor 1501 may be a general-purpose Processor, such as a Central Processing Unit (CPU), a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other Programmable logic device, a discrete Gate or transistor logic device, or a discrete hardware component, and may implement or execute the methods, steps, and logic blocks disclosed in the embodiments of the present Application. A general purpose processor may be a microprocessor or any conventional processor or the like. The steps of a method disclosed in connection with the embodiments of the present application may be directly implemented by a hardware processor, or may be implemented by a combination of hardware and software modules in a processor.

The memory 1502, which is a non-volatile computer-readable storage medium, may be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. The Memory may include at least one type of storage medium, and may include, for example, a flash Memory, a hard disk, a multimedia card, a card-type Memory, a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a Programmable Read Only Memory (PROM), a Read Only Memory (ROM), a charged Erasable Programmable Read Only Memory (EEPROM), a magnetic Memory, a magnetic disk, an optical disk, and so on. The memory is any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer, but is not limited to such. The memory 1502 in the embodiments of the present application may also be circuitry or any other device capable of performing a storage function for storing program instructions and/or data.

Based on the same inventive concept as the above-mentioned picture book identification method, an embodiment of the present application further provides a picture book identification system, and referring to fig. 16, the picture book identification system 160 may include: an image acquisition device 1601, a projection device 1602 and a picture book recognition apparatus 1603 provided in any one of the above implementations.

The image capturing apparatus 1601 is configured to capture a captured image including a picture book.

The projection device 1602 is configured to project the projection picture corresponding to the target picture template determined by the picture book recognition device 1603 onto the picture book according to the projection position determined by the picture book recognition device 1603. The projection device 1602 may be an AR projection system.

The embodiment of the present application further provides an audio playing device 1604 that may be further included in the picture book recognition system 160, when the projection device 1602 projects, the audio corresponding to the picture book may be synchronously played through the audio playing device 1604, so as to achieve the effect of drawing sound and drawing color.

In order to ensure the stability of the picture book recognition effect and the projection effect, the picture book recognition system 160 may be a terminal device integrated with the image capture device 1601, the projection device 1602, the picture book recognition apparatus 1603, and the audio playback device 1604, so as to ensure that the shooting area of the image capture device 1601 and the projection area of the projection device 1602 are overlapped as much as possible.

The picture book is placed in the shooting area of the image capturing apparatus 1601 to perform picture book recognition. Specifically, the image capturing apparatus 1601 captures a captured image including a picture book, and sends the captured image to the picture book recognizing device 1603; the sketch identifying device 1603 determines a target sketch template matched with the shot image from the plurality of sketch templates, determines the position information of the sketch in the shot image in an image coordinate system corresponding to the image acquisition device 1601, and determines the projection position of the position information relative to the projection device 1602 according to the coordinate mapping relationship between the image coordinate system and a projection coordinate system corresponding to the projection device 1602; the projection device 1602 projects the projection picture corresponding to the eye-drawing book template onto the drawing book according to the projection position determined by the drawing book identification device 1603, so that the user can see the projection picture on the drawing book, and the interest in reading the drawing book is improved.

Exemplary program product

An embodiment of the present application provides a computer-readable storage medium for storing computer program instructions for the electronic device, which includes a program for executing the above-mentioned picture recognition method.

The computer storage media may be any available media or data storage device that can be accessed by a computer, including but not limited to magnetic memory (e.g., floppy disks, hard disks, magnetic tape, magneto-optical disks (MOs), etc.), optical memory (e.g., CDs, DVDs, BDs, HVDs, etc.), and semiconductor memory (e.g., ROMs, EPROMs, EEPROMs, non-volatile memory (NAND FLASH), Solid State Disks (SSDs)), etc.

The above embodiments are only used to describe the technical solutions of the present application in detail, but the above embodiments are only used to help understanding the method of the embodiments of the present application, and should not be construed as limiting the embodiments of the present application. Modifications and substitutions that may be readily apparent to those skilled in the art are intended to be included within the scope of the embodiments of the present application.

Claims

1. A picture book recognition method is characterized by comprising the following steps:

acquiring a shot image acquired by image acquisition equipment;

2. The method according to claim 1, wherein the determining a target sketch template matching the captured image from the plurality of sketch templates specifically comprises:

3. The method according to claim 2, wherein the feature vectors corresponding to the feature points of the captured image extracted at each resolution are sequentially matched with the feature vectors corresponding to the feature points of the template extracted at the corresponding resolution, and the resolution P is set for each resolution_nThe feature vector corresponding to the feature point of the shot image extracted below and the resolution P_nMatching the feature vectors corresponding to the feature points of each drawing template extracted below, specifically comprising:

4. The method of claim 3, wherein prior to determining the matching cost template as the goal cost template, the method further comprises:

from the picture book template T_mAcquiring a preset number of point pairs from the corresponding point pair set;

the determining the matched picture template as the target picture template specifically includes:

if the picture book template T is determined_mThrough geometric verification, the drawing template T is obtained_mThe target template is determined.

5. The method of claim 3, wherein the matching order of the sketch templates is determined by:

6. The method according to claim 5, wherein determining a matching order of the sketch templates according to the previously matched target sketch templates specifically comprises:

7. The method according to claim 3, wherein the matching sequence of the sketch template corresponding to each resolution is determined by the following method:

8. The method according to claim 4, wherein the determining the position information of the sketch in the captured image in the image coordinate system corresponding to the image capturing device specifically includes:

determining position information of a picture in the captured image in the image coordinate system based on a homography that passes the picture template through geometric verification and the picture template.

9. The method according to any one of claims 2 to 8, wherein before the feature vectors corresponding to the feature points of the captured image extracted at each resolution are sequentially matched with the feature vectors corresponding to the feature points of each picture template extracted at the corresponding resolution in the order from low resolution to high resolution, the method further comprises:

acquiring a first coordinate of each feature point in a projection picture of the current projection in the projection coordinate system;

10. The method of any of claims 1 to 8, further comprising:

after the target plotting template is determined, acquiring a shot image acquired by the image acquisition equipment in real time, and tracking the characteristic points of the target plotting template in the acquired shot image;

11. The method of claim 10, further comprising:

if the successfully tracked feature points do not meet the preset requirements, acquiring a new shot image acquired by the image acquisition equipment;

determining a new target picture template from the picture template according to the matching degree of the feature points extracted from the new shot image and the feature points of each picture template;

determining the position information of the picture in the new shot image in the image coordinate system corresponding to the image acquisition equipment;

determining the projection position of the position information corresponding to the new shot image relative to the projection equipment according to the coordinate mapping relation between the image coordinate system and the projection coordinate system corresponding to the projection equipment;

and projecting the projection picture corresponding to the new target picture drawing template onto the picture drawing according to the projection position corresponding to the new shot image.

12. The method according to any one of claims 1 to 8, wherein the feature points corresponding to each of the sketch templates are obtained by:

13. The method according to claim 12, wherein the extracting the candidate feature points and the feature vectors corresponding to the candidate feature points from the template specifically comprises:

14. A picture recognition apparatus, comprising:

15. A picture recognition apparatus comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the steps of the method according to any one of claims 1 to 13 are implemented when the computer program is executed by the processor.

16. A picture recognition system, comprising: an image acquisition device, a projection device and a picture recognition apparatus as claimed in claim 15;

17. A computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, implement the steps of the method of any one of claims 1 to 13.