WO2022001600A1

WO2022001600A1 - Information analysis method, apparatus, and device, and storage medium

Info

Publication number: WO2022001600A1
Application number: PCT/CN2021/098998
Authority: WO
Inventors: 李亚乾; 侯振靖; 黄超
Original assignee: Oppo广东移动通信有限公司
Priority date: 2020-07-03
Filing date: 2021-06-08
Publication date: 2022-01-06
Also published as: CN111860232A

Abstract

An information analysis method, apparatus, and device, and a storage medium. The method comprises: determining an information type of information comprised in an object to be analyzed (S110), wherein said object is an image related to an application program; under the condition that the information type comprises a preset type, analyzing information corresponding to the preset type in said object, and determining identifier information of the application program (S120); and in response to receiving an instruction of searching the application program, skipping to an application program store according to the identifier information of the application program (S130).

Description

Information analysis method and device, equipment, storage medium

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is based on the Chinese patent application with the application number of 202010637185.2 and the application date of July 3, 2020, and claims the priority of the Chinese patent application. The entire content of the Chinese patent application is hereby incorporated into the application in its entirety. .

technical field

The present application relates to the technical field of image processing, and relates to, but is not limited to, information analysis methods and devices, devices, and storage media.

Background technique

In many user scanning scenarios, there is a need to obtain the relevant information of the object to be scanned immediately for the preview image in the viewfinder. The name or identification number (Identity, ID) of the application. In this case, the user's scanning habit is usually to detect the QR code in the promotional advertisement first. However, if the QR code itself is damaged or the icon is updated, the correct information cannot be parsed. At the same time, there is the limitation of no QR code scene.

SUMMARY OF THE INVENTION

Embodiments of the present application provide an information parsing method, apparatus, device, and storage medium.

The technical solutions of the embodiments of the present application are implemented as follows:

In a first aspect, an embodiment of the present application provides an information parsing method, the method comprising:

Determine the information type of the information included in the object to be parsed; wherein, the object to be parsed is an image related to an application;

In the case that the information type includes a preset type, parse the information corresponding to the preset type in the object to be parsed, and determine the identification information of the application;

In response to receiving the instruction to search for the application, jump to the application store according to the identification information of the application.

In a second aspect, an embodiment of the present application provides an information parsing device, the device includes a determining module, an parsing module and a jumping module, wherein:

The determining module is configured to determine the information type of the information included in the object to be analyzed; wherein the object to be analyzed is an image related to an application program;

The parsing module is configured to parse the information corresponding to the preset type in the object to be parsed, and determine the identification information of the application when the information type includes a preset type;

The jumping module is configured to jump to the application store according to the identification information of the application in response to receiving the instruction to search for the application.

In a third aspect, an embodiment of the present application provides an information parsing device, including a memory and a processor, where the memory stores a computer program that can be run on the processor, and the processor implements the above information parsing method when executing the program steps in .

In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, implements the steps in the above information analysis method.

Description of drawings

In order to illustrate the technical solutions in the embodiments of the present application more clearly, the following briefly introduces the drawings that are used in the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, under the premise of no creative work, other drawings can also be obtained from these drawings, wherein:

FIG. 1 is an optional schematic flowchart of an information analysis method provided by an embodiment of the present application;

2 is a schematic diagram of a barcode and a two-dimensional code provided by an embodiment of the present application;

FIG. 3 is an optional schematic flowchart of the information analysis method provided by the embodiment of the present application;

FIG. 4 is an optional schematic flowchart of the information analysis method provided by the embodiment of the present application;

FIG. 5 is an optional schematic flowchart of the information analysis method provided by the embodiment of the present application;

FIG. 6 is an optional schematic flowchart of the information analysis method provided by the embodiment of the present application;

7 is a schematic diagram of an application poster provided by an embodiment of the present application;

8A is a schematic flowchart of two-dimensional code detection and correction provided by an embodiment of the present application;

8B is a schematic flowchart of poster cropping and correction provided by an embodiment of the present application;

9 is a schematic flowchart of information parsing of a two-dimensional code without an icon according to an embodiment of the present application;

10 is a schematic flowchart of information parsing of a two-dimensional code with an icon provided by an embodiment of the present application;

11 is a schematic flowchart of icon information analysis provided by an embodiment of the present application;

12 is a schematic flowchart of information analysis of publicity content provided by an embodiment of the present application;

FIG. 13 is a schematic flowchart of a solution for single feature comparison based on image search provided by an embodiment of the present application;

14 is a schematic flowchart of a solution for image search-based multi-feature fusion and comparison provided by an embodiment of the present application;

FIG. 15 is a schematic flowchart of a solution for single information recognition based on image recognition provided by an embodiment of the present application;

16 is a schematic flowchart of a fusion solution for image recognition-based multi-information recognition provided by an embodiment of the present application;

FIG. 17 is a schematic diagram of the composition and structure of an information analysis apparatus provided by an embodiment of the present application;

FIG. 18 is a schematic diagram of a hardware entity of an information parsing device provided by an embodiment of the present application.

detailed description

In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments It is a part of the embodiments of the present application, but not all of the embodiments. The following examples are used to illustrate the present application, but are not intended to limit the scope of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

In the following description, reference is made to "some embodiments" which describe a subset of all possible embodiments, but it is understood that "some embodiments" can be the same or a different subset of all possible embodiments, and Can be combined with each other without conflict.

It should be pointed out that the term "first\second\third" involved in the embodiments of the present application is only to distinguish similar objects, and does not represent a specific ordering of objects. It is understandable that "first\second\third" "Where permitted, the specific order or sequence may be interchanged to enable the embodiments of the application described herein to be practiced in sequences other than those illustrated or described herein.

It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the embodiments of this application belong. It should also be understood that terms, such as those defined in a general dictionary, should be understood to have meanings consistent with their meanings in the context of the prior art and, unless specifically defined as herein, should not be interpreted in idealistic or overly formal meaning to explain.

The embodiments of the present application provide an information parsing method, which is applied to a mobile terminal. The mobile terminals include but are not limited to mobile phones, notebook computers, tablet computers and handheld Internet devices, multimedia devices, streaming media devices, mobile Internet devices, wearable devices or other types of terminal devices. The functions implemented by the method can be implemented by calling the program code by the processor in the mobile terminal. Of course, the program code can be stored in a computer storage medium. It can be seen that the mobile terminal includes at least a processor and a storage medium.

FIG. 1 is an optional schematic flowchart of an information analysis method provided by an embodiment of the present application. As shown in FIG. 1 , the method includes:

Step S110, determining the information type of the information included in the object to be parsed.

Here, the object to be parsed is an image related to an application program. Usually, there are different information such as product icons, preset pictures, identification characters, product introductions, and link addresses. The corresponding information types include pictures, texts, and comprehensive products. A barcode/QR code for information such as name, date of manufacture, manufacturer and link address. In an actual scanning scenario, the information in the preview screen may only contain one of the above-mentioned information or a combination of certain information.

Here, the information type of the information represents an information encoding method, and different information encoding methods are adopted according to different information carried, thereby generating different information types such as text, sound, video, and pictures.

Illustratively, as shown in Figure 2, the barcode 21 and the two-dimensional code 22 are both common image carriers used for information encoding and parsing. effective means of collecting data. Compared with the traditional barcode 21, the two-dimensional code 22 can store more information, can also represent more types of data, and has a strong error correction ability. Mainstream QR code applications include information acquisition, website redirection, mobile payment, account login, etc.

Step S120, in the case that the information type includes a preset type, parse the information corresponding to the preset type in the object to be parsed, and determine the identification information of the application program.

Here, the preset type of information is information representing the core content of the application or the corresponding product, and includes at least one of the following preset types of information: a two-dimensional code, an application icon, and text information.

It should be noted that some QR code centers include application icons, while some QR codes only use black and white pixel matrix coding; application icons include application icons and/or application posters at the center of the QR code. The application icon on the layout of the icon; the text information is the preset content keyword, such as the application or product name, slogan, etc., where the application name may appear in three scenarios: in the download address screen, near the icon and in the promotional content .

Here, the identification information of the application program may be the name of the application program or the corresponding product name, or may be an ID number. The information of each preset type is parsed through visual algorithms such as image search technology or image recognition technology, so as to quickly obtain the name/ID of the application or product when scanning the image related to the application.

Step S130, in response to receiving the instruction to search for the application, jump to the application store according to the identification information of the application.

Here, by receiving an operation instructing to search for the application program on a specific interface, generating a link address according to the identification information of the application program, and jumping to the interface of the application store.

It should be noted that the jump from the current scanning program to the application store program is realized by designing the logic of the terminal itself. In the implementation process, the application program name or product name can be used as input, and the corresponding application program software can be searched in the application program store to download and install.

In the embodiment of the present application, first, the information type of the information included in the object to be parsed is determined; wherein, the object to be parsed is an image related to an application program; then, when the information type includes a preset type , parse the information corresponding to the preset type in the object to be parsed, and determine the identification information of the application; finally, in response to receiving an instruction to search for the application, jump according to the identification information of the application Go to the application store; in this way, by parsing the information of the preset type in the object to be parsed, the obtained object to be parsed is quickly mapped to the identification information of the application and jumped to the application store, so as to solve the need for users to quickly obtain the application identification. information, but the QR code is damaged or the icon cannot be parsed correctly when the icon is updated. It also applies to scenarios without QR code.

FIG. 3 is an optional schematic flowchart of the information analysis method provided by the embodiment of the present application. As shown in FIG. 3 , the object to be parsed is an application poster, and the information corresponding to the preset type is a file containing an application icon. Two-dimensional code, the method includes at least the following steps:

Step S310, determining the information type of the information included in the application poster.

Step S320, when the application icon is located in the two-dimensional code, intercept the image area corresponding to the application icon from the two-dimensional code containing the application icon.

Here, for the case where the information type includes a two-dimensional code containing an application icon, that is, a scenario where the application icon and the two-dimensional code appear at the same time, the location and area of the application icon can be located through the location of the two-dimensional code. For example, the location and area of the icon can be obtained through an object detection algorithm, an image segmentation algorithm, and the like.

Step S330, analyze the image area to determine the identification information of the application program.

Here, the identification information of the application program is determined by performing image recognition or image search on the image area corresponding to the icon. Wherein, the identification information of the application program may be the name of the application program or the corresponding product name, or may be an ID.

Step S340, in response to receiving the instruction to search for the application, jump to the application store according to the identification information of the application.

Step S350, in response to the instruction to download the application, download the application through an application store.

Here, the identification information of the application, such as the product name, can be used as input, and the application can be downloaded through the application store.

In the embodiment of the present application, for the scanning scene containing the application icon in the two-dimensional code, the area corresponding to the application icon in the two-dimensional code is intercepted for analysis, so that the information of the two-dimensional code itself can be ignored when scanning, and the direct Detecting and identifying the icon can avoid the situation where the QR code is damaged and cannot obtain the correct analysis result, and improve the user experience. At the same time, the information of the application identification information obtained by the analysis is diverted and directly jumped to the application store, which is convenient for users to download or purchase.

FIG. 4 is an optional schematic flowchart of the information parsing method provided by this embodiment of the present application. As shown in FIG. 4 , the object to be parsed is an application poster, and the information types include two or more preset types. The method includes at least the following steps:

Step S410, determining the information type of the information included in the application poster.

Step S420, in the case that the information type includes two or more preset types, preprocess the application poster, and determine information of each preset type from the application poster.

Here, in the case where the information types include two or more preset types, first preprocess the application poster scanned by the camera, including the alignment and normalization of the acquired poster image information and the standard image, as well as the promotion content and two QR code positioning, cropping, correction, etc., application icon positioning, detection and segmentation, etc., so as to extract information such as QR code, application icon and promotional content from the application poster.

Step S430: Prioritize the information corresponding to each preset type.

In step S440, target information is determined according to the sorting result.

Here, the priority of the target information is higher than the priority of any other information in the information of each preset type.

It should be noted that different preset types of information in the application poster, such as QR code, application icon, text and other information, may contain the required content at the same time. The level ranking determines the most important information at the moment as the target information.

In some possible embodiments, the two or more preset types of information include a two-dimensional code without the application icon, and the method further includes: determining the two-dimensional code without the application icon The priority level is lower than the priority level of other preset types of information.

Here, for the direct scanning of the QR code without the application icon, the QR code may be damaged and the correct information cannot be obtained by parsing. Priority level, so that target information is selected from other preset types of information for priority analysis.

Step S450, analyze the target information to determine the identification information of the application program.

Here, the target information can be input into the trained feature extraction model to extract the features contained in the target information, so as to obtain the identifier of the application program contained in the target information; the target information can also be input into the trained information In the identification model, the identification of the application program contained in the target information is directly obtained.

In some possible embodiments, the above-mentioned step S450 may be implemented through the following steps: “Perform feature extraction on the target information to obtain a target feature point set; perform feature comparison between the target feature point set and a preset sample feature point set. Yes, determine the identification information of the application".

In some possible embodiments, the above step S450 may be implemented through the following steps to "perform image recognition on the target information to obtain the identification information of the application program".

Step S460, in response to receiving the instruction to search for the application, jump to the application store according to the identification information of the application.

The embodiment of the present application determines the target information by prioritizing different preset types of information in the object to be analyzed, so as to quickly analyze the most important information content at present based on the feature extraction and feature comparison algorithm or based on the image recognition algorithm, Map the images related to the application to the introduction information of the corresponding application for further processing such as downloading or purchasing.

FIG. 5 is an optional schematic flowchart of the information analysis method provided by the embodiment of the present application. As shown in FIG. 5 , the object to be parsed is an application poster, and the method includes at least the following steps:

Step S510, determining the information type of the information included in the application poster.

Step S520, in the case that the information type includes two or more preset types, call a parser corresponding to each of the preset types, and perform feature extraction on the information of each of the preset types to obtain at least one feature point collection.

Here, the application poster to be parsed may also include information such as QR code, application icon, text, etc., and feature extraction can be performed on each preset type of information, that is, each preset type can be obtained through single-mode feature extraction. Set the feature point set corresponding to the type of information.

It should be noted that the feature extraction model can be obtained through the manual shooting of poster and icon data of various applications, and joint training with standard application poster and icon data. Therefore, the feature extraction model is used to perform feature extraction on each of the preset types of information. At the same time, new application information or existing application update poster pictures are dynamically added to perform real-time training and update of the feature extraction model.

Step S530: Perform feature fusion on the at least one feature point set to obtain a target feature point set.

Here, the feature fusion (feature fusion) is to generate new features from different extracted features by a certain method. By performing feature fusion on feature point sets corresponding to different preset types of information, the new features can better represent the information contained in the application poster. For example, feature fusion is performed on the feature point set extracted from the two-dimensional code and the feature point set extracted from the application program icon, so as to obtain the comprehensive feature point set included in the application program poster. It is worth noting that the fusion process can be performed at three levels: input, feature, and output

Step S540, compare the features of the target feature point set with a preset sample feature point set to determine the identification information of the application program.

Here, the preset sample feature point set is obtained by performing feature extraction on a sample application poster.

Here, the process of feature comparison between the standard feature point set and the preset sample feature point set is essentially the comparison of the application poster to be scanned and the sample application poster in a high-dimensional feature space. Identifying information for the final application is determined by matching with the characteristics of the sample application poster.

Step S550, in response to receiving the instruction to search for the application, jump to the application store according to the identification information of the application.

In this embodiment of the present application, feature extraction is performed on information of different preset types in the application poster, respectively, to obtain a feature point set corresponding to each preset type of information, and then feature fusion is performed to obtain a target feature point set, and finally the target feature point set is obtained. The feature comparison is performed with the preset sample feature point set corresponding to the sample application poster to determine the identification information of the application.

FIG. 6 is an optional schematic flowchart of the information parsing method provided by the embodiment of the present application. As shown in FIG. 6 , the object to be parsed is an application poster, and the method at least includes the following steps:

Step S610, determining the information type of the information included in the application poster.

Step S620, in the case that the information type includes two or more preset types, call the parser corresponding to each of the preset types, and perform information identification on the information of each of the preset types to obtain at least one set of identification data.

Here, identifying each of the preset types of information refers to performing image recognition on the information area included in the application poster, so as to identify various types of information content. For example, perform image recognition on the QR code in the application poster, and obtain the information content contained in the QR code to obtain a set of identification data; perform image recognition on the icon in the application poster, and obtain the information content contained in the icon, namely Obtain another set of identification data, etc.

It should be noted that a large number of photographed application posters or icon images can be used, and manual annotations can be used as training data to train an information recognition model, and then the trained information recognition model can be used to identify posters, icons and other information. At the same time, new application information or existing application update poster pictures are dynamically added for real-time training and updating of the information recognition model.

Step S630: Perform data fusion on the at least one set of identification data according to the preset identification confidence, and determine the identification information of the application program.

Here, the preset recognition confidence level represents the probability that the difference between the recognized mean value and the overall real situation is smaller than a specific threshold. The so-called confidence level, also known as the confidence level, refers to the degree to which a specific individual believes in the truth of a specific proposition.

Here, according to the preset identification confidence, data fusion is performed on the identified identification data, that is, the content contained in the information of each preset type of information collected by the application poster is fused, so as to determine the identification information and Download address information.

Step S640, in response to receiving the instruction to search for the application, jump to the application store according to the identification information of the application.

In this embodiment of the present application, image recognition is performed on different preset types of information in the application poster, to obtain a plurality of information related to the application, and then the information output level is fused according to the recognition confidence, and finally the identification information and Download address information.

Below, an exemplary application of the embodiments of the present application in a practical application scenario will be described.

The embodiment of the present application proposes a technical solution capable of parsing poster information through a visual algorithm, so that the application poster image corresponds to the name/ID of the application or product in one-to-one correspondence. In this way, when the user scans the poster with the smart scan function of the machine, the corresponding application/product name can be quickly obtained, which can be used as the input for jumping to the application store or purchasing address in the next step. The technical process of the information analysis method provided by the embodiment of the present application mainly includes two parts: one part is the extraction and preprocessing of different types of information in the application poster; the other part is to identify and analyze different types of information respectively.

1. Extraction and Preprocessing of Different Types of Information in Application Posters

The core information that may be included in the application poster, such as QR code, application icon, promotional content, and text information, which are four categories of application-related information, basically cover the information that users may encounter in various scanning scenes. Type of information. In actual use, the scene in the scanned image may contain only one of the above-mentioned information or a combination of certain information.

FIG. 7 is a schematic diagram of an application poster provided by an embodiment of the present application. As shown in FIG. 7 , the application poster includes a two-dimensional code 71 , promotional content 72 , an icon 73 at the center of the two-dimensional code 71 , and the upper left corner of the promotional content 72 73 , the application name 74 and the slogan 75 at the center of the promotional content 72 . The icon 73 in the center of the two-dimensional code 71 and the icon 73 in the upper left corner of the promotional content 72 are both application icons, and the application name 74 and the slogan 75 in the center of the promotional content 72 are text information.

The two-dimensional code 71 is the key information scanned by the camera. Most of the two-dimensional code 71 of the application program contains the icon 73 in the center, but there are cases where the icon is not included. There are many technical means to detect two-dimensional code 71. You can choose special algorithm for two-dimensional code detection, target detection and image post-processing, semantic segmentation, key point detection, text area detection (such as regression irregular quadrilateral idea) and other methods to obtain two-dimensional code. Dimension code area.

In a real scene, whether it is a paper poster or a poster image on the device screen, the information contained in the QR code, icon, and promotional content can basically be regarded as being distributed on a two-dimensional plane. When a user scans the corresponding information with a mobile phone, due to the position and posture of the camera, the photographed picture is often subjected to inverse affine transformation, and the conventional rectangle/square becomes an irregular quadrilateral in the picture.

Before processing each piece of information individually, document clipping and correction may be performed on the promotional content 72 or the two-dimensional code 71, and the specific technology may use key point detection or straight line detection. FIG. 8A is a schematic flowchart of two-dimensional code detection and correction provided by an embodiment of the application. As shown in FIG. 8A , the two-dimensional code area 81 detected in the viewfinder is generally an irregular quadrilateral, and the two-dimensional code area 81 can be After correction, a conventional rectangular two-dimensional code area 82 is obtained. FIG. 8B is a schematic flowchart of poster clipping and correction provided by an embodiment of the present application. As shown in FIG. 8B , the clipped promotional content is a trapezoid 83 , and the affine transformation is calculated through four vertices A, B, C, and D. matrix, and affine transformation is performed on the trapezoid 83 to obtain a regular rectangle 84.

The icon 73 will appear in two different scenarios: the first one is at the same time as the two-dimensional code 71 and is located in the center of the two-dimensional code 71; the second is independent of the two-dimensional code 71, for example, it may appear in the application The top of the program poster may also appear on the desktop interface of the terminal device, or on the screen of the download address.

Before the icon 73 is detected, the correction of the promotional content 72 or the two-dimensional code 71 has been completed by default. For the scene where the icon 73 and the two-dimensional code 71 appear at the same time, the position and area of the icon 73 can be located by the position of the two-dimensional code 71, or a separate algorithm can be designed to detect the two-dimensional code 71 and the icon 73 at the same time. This multi-task model that combines two different features can often improve the detection accuracy of each. For the scene in which the icon 73 appears independently of the two-dimensional code 71, the position and area of the icon 73 can be obtained through a target detection algorithm, an image segmentation algorithm, and the like.

For the application name 74, the text can be extracted by a common Optical Character Recognition (Optical Character Recognition, OCR) technology, and matched with the existing application name in the database to determine the final application name.

2. Identify and analyze different types of information separately

Judging from the user's current cognition and usage habits, in the poster scene, the user's priority for scanning after turning on the scan function is: first scan the QR code 71, then scan the application icon 73, and finally scan the promotional content 72, Based on this priority, the following application scenarios are summarized:

1. Scan the QR code without icon 71

Decoding of application information can be achieved by taking screenshots.

FIG. 9 is a schematic flowchart of information parsing of a two-dimensional code without an icon provided by an embodiment of the present application. As shown in FIG. 8 , it at least includes: step S901 two-dimensional code detection; step S902 information decoding; Step S903 screenshots; step S904 icon identification and name identification; finally obtain the application name/ID.

2. Scan the QR code with the icon71

Because the QR code contains the application icon, the information of the QR code itself can be ignored when scanning, and the icon can be detected and recognized directly.

FIG. 10 is a schematic flowchart of information extraction of a two-dimensional code with an icon provided by an embodiment of the present application, as shown in FIG. 10 , including solution 1 and solution 2. For scheme 1, it includes step S1001 two-dimensional code detection; step S1002 icon positioning; for scheme 2, it includes step S1003 two-dimensional code and icon multi-task positioning detection; finally scheme 1 and scheme 2 go through step S1004 icon recognition, and obtain the final application name/ID.

3. Scan icon 73

There are two cases for scanning icons. The first is that there is only an icon in the screen, and the second is that the name of the application or the slogan is also included near the icon.

FIG. 11 is a schematic flowchart of icon information analysis provided by the embodiment of the present application. As shown in FIG. 11 , for the first case, it includes step S1101 icon positioning and step S1102 icon identification, and finally obtains the application name/ID; The two cases include step S1103 text detection, step S1104 OCR character recognition and step S1105 character matching, and finally obtain the application name or ID.

4. Scan promotional content 72

When the user scans the application poster, there may only be promotional content 72 in the screen, and may also include a two-dimensional code 71, an icon 73, an application name 74 or a slogan 75, and so on. Therefore, the identification of the propaganda content 72 can be performed independently, or the identification of multi-dimensional information fusion can be performed. Single-task and multi-task models can be established, such as feature fusion or cross-modal feature fusion for the identified icon 73 features and the identified promotional content 72 features.

Fig. 12 is a schematic flowchart of the information analysis of publicity content provided by the embodiment of the present application, as shown in Fig. 12, including step S1201 cropping and correction of publicity content, step S1202 publicity content identification; step S1203 icon positioning, step S1204 icon identification; step S1203 S1205 text detection, step S1206 OCR recognition and text matching, and finally obtain the application name/ID.

3. Realization of the overall technical scheme

The first two steps are to realize information extraction preprocessing and information identification and analysis, and then build the overall technical framework of poster information analysis. Four different schemes will be introduced here, as shown in Figures 13 to 16. The first two schemes are based on image search technology and are suitable for scenarios with a large number of application/product categories, frequent data updates, and a small number of samples; the latter two This solution is based on image recognition technology and is suitable for scenarios with a small number of application/product categories, slow data updates, and many standard and natural environment shooting samples.

The solution of single feature comparison based on image search provided by the embodiments of the present application, after information extraction and preprocessing, the most important information at present is determined through information priority, and then feature extraction and comparison are performed.

FIG. 13 is a schematic flowchart of a solution for single-feature comparison based on image search provided by an embodiment of the present application. As shown in FIG. 13 , it includes the following steps: Step S1301 obtains a poster picture; Step S1302 Information extraction and preprocessing; Step S1303 Priority sorting Step S1304 feature extraction; Step S1305 online real-time training feature extraction model; Step S1306 real-time update standard poster image and APP information database; Step S1307 feature extraction; Step S1308 update feature database; Step S1309 feature comparison; Step S1310 output application name /ID. Among them, step S1304 is to perform feature extraction on the information with the highest priority, and step S1307 is to perform feature extraction on the corresponding information in the standard poster image.

The image search-based multi-feature fusion solution provided by the embodiment of the present application is to perform feature extraction on all the extracted information, and then perform feature fusion and comparison.

Fig. 14 is a schematic flowchart of a solution for multi-feature fusion and comparison based on image search provided by an embodiment of the application, as shown in Fig. 14 , including the following steps: step S1401 obtaining a poster picture; step S1402 information extraction and preprocessing; step S1403 feature Extraction; step S1404 feature fusion; step S1405 online real-time training feature extraction model; step S1406 real-time update standard poster image and application information database; step S1407 feature extraction; step S1408 update fusion feature database; step S1409 feature comparison; step S1410 Output application name/ID. Wherein, step S1404 is to perform feature extraction on different information in the acquired poster picture respectively, and step S1407 is to perform feature extraction on corresponding information in the standard poster picture.

The solution based on single-information image recognition provided by the embodiments of the present application determines the most important information through information priority after information extraction and preprocessing, and then directly outputs the parsed application name or ID information through the recognition model.

Fig. 15 is a schematic flowchart of a solution for single information recognition based on image recognition provided by an embodiment of the present application, as shown in Fig. 15 , including the following steps: step S1501 obtaining a poster picture; step S1502 information extraction and preprocessing; step S1503 information identification; Step S1504 update the standard and the photographed application poster in real time; step S1505 update the application information database; step S1506 online real-time training information recognition model; step S1507 information fusion; step S1508 output the application name/ID. Wherein, steps S1504 to S1506 are the process of training the information recognition model.

The fusion scheme based on multi-information image recognition provided by the embodiment of the present application identifies all the extracted information separately, obtains multiple application program names and ID information, and then performs fusion at the information output level according to the recognition confidence.

16 is a schematic flowchart of a fusion solution for image recognition-based multi-information recognition provided by an embodiment of the present application, as shown in FIG. 16 , including the following steps: step S1601 obtaining a poster picture; step S1602 information extraction and preprocessing; step S1603 information recognition Step S1604 real-time update standard and photographed application poster; step S1605 update application information database; step S1606 online real-time training information recognition model; step S1607 information fusion; step S1608 output application name/ID. Among them, steps S1604 to S1606 are the process of training the information recognition model.

There are several modules in the four schemes that need to be highlighted:

1. Information extraction and preprocessing module

If the image obtained by the camera of the smart terminal device is directly compared with the standard icon and standard poster image, the accuracy will be greatly affected. In this embodiment of the present application, the images scanned by the camera are first preprocessed, including the positioning, cropping, and correction of posters and QR codes, the positioning, detection, and segmentation of icons, and the alignment and normalization of the extracted poster image information and standard images. change, etc.

2. Training modules for feature extraction models or recognition models

The first two schemes are based on image search, which is essentially the comparison of images in high-dimensional feature space. The search-based scheme is obtained through manual shooting of posters and icon data from various applications, and joint training with standard poster and icon data. Feature extraction model.

The latter two schemes are based on image recognition, using a large number of photographed poster and icon images, and using manual annotation as training data to train an information recognition model to identify posters, icons and other information.

The feature extraction model and recognition model in the four schemes can be single-task or multi-task learning models.

3. Online real-time update database and feature extraction/recognition model module

Dynamically add new application information, or update existing application standard icons, poster pictures, manually captured pictures, update information database and feature database (or fusion feature database), and real-time feature extraction model and information recognition model Training update.

4. Information prioritization module

Different information in the application poster can be prioritized according to the importance of the information contained, such as the icon in the QR code, the icon in the promotional content, the promotional content, the application name/product name/slogan in descending order of priority .

5. Single information processing and feature/information fusion module

The scanned image may contain QR code, icon, text and other information at the same time, which can be extracted and compared through single-mode feature extraction, or can be compared through multi-mode feature fusion. It should be noted that the fusion process can be performed at three levels: input, feature, and output.

This embodiment of the present application can implement the resolution of the application or product name/ID of the poster. Posters/icons are no longer simple pictures, but can be used as carriers for small information storage, such as storing application or product names and corresponding IDs, etc. Through the embodiments of this application, the corresponding information can be parsed from the input poster pictures, Can be used as input for application downloads, product purchases. It can also perform terminal scanning based on name and ID to realize augmented reality advertisements and enrich poster content.

The embodiment of the present application can expand the scanning application scenario, and improve the download probability of the application store application. For example, users can easily and quickly use the local smart scan function, which expands the application scenarios of the scan function, whether it is for a QR code containing an icon, an application icon or a poster, in various user scanning scenarios. You can jump to the local app store to download the app. This convenience triggers the opportunity for more app downloads, increasing the probability that users will download apps. At the same time, the embodiment of the present application maps various scanned information to the corresponding application program name/ID, and directly jumps to the download address page of the application store.

The embodiments of the present application first propose a method for extracting and preprocessing information such as icons, QR codes, promotional content, and text that may exist in posters, and then build an overall technical architecture based on traditional and deep learning visual algorithms. The core innovations are mainly reflected in the technical architecture of four different poster comprehensive information analysis and the application of image search/recognition technology in poster information analysis scenarios.

An embodiment of the present application provides an information parsing device, the device includes each module included and each submodule included in each module, which can be implemented by a processor in a terminal; of course, it can also be implemented by a specific logic circuit; In the implementation process, the processor can be a central processing unit (Central Rrocessing Unit, CPU), a microprocessor (Micro Rrocessing Unit, MPU), a digital signal processor (Digital Signal Processor, DSP) or a field programmable gate array ( Field Programmable Gate Array, FPGA), etc.

FIG. 17 is a schematic structural diagram of an information parsing apparatus provided by an embodiment of the present application. As shown in FIG. 17 , the apparatus 1700 includes a determination module 1701, an analysis module 1702, and a jump module 1703, wherein:

The determining module 1701 is configured to determine the information type of the information included in the object to be parsed; wherein the object to be parsed is an image related to an application;

The parsing module 1702 is configured to parse the information corresponding to the preset type in the object to be parsed, and determine the identification information of the application when the information type includes a preset type;

The jumping module 1703 is configured to jump to the application store according to the identification information of the application in response to receiving the instruction to search for the application.

In some possible embodiments, the object to be parsed is an application poster, and the preset type of information in the application poster includes at least one of the following: a two-dimensional code, an application icon, and text information; wherein the An application icon is located on the layout of the application poster, and/or, on the QR code.

In some possible embodiments, the apparatus 1700 further includes a download module configured to download the application program through an application store in response to the instruction to download the application program.

In some possible embodiments, the parsing module 1702 includes an intercepting sub-module and a first parsing sub-module, wherein: the intercepting sub-module is configured to be configured when the application icon is located in the two-dimensional code , intercept the image area corresponding to the application icon from the two-dimensional code containing the application icon; the first parsing submodule is configured to analyze the image area and determine the identification information of the application .

In some possible embodiments, the information types include two or more preset types, the parsing module 1702 includes a priority sorting sub-module, a determination sub-module and a second parsing sub-module, wherein: the priority sorting sub-module a module, configured to prioritize the information corresponding to the two or more preset types when the information types include more than two preset types; the determining sub-module is configured to determine according to the sorting result target information; wherein, the priority of the target information is higher than the priority of other preset types of information in the object to be parsed; the second parsing sub-module is configured to parse the target information and determine the target information. the identification information of the application.

In some possible embodiments, the two or more preset types of information include a two-dimensional code without the application icon, and the priority sorting sub-module is further configured to determine that the application icon does not exist The priority level of the QR code is lower than the priority level of other preset types of information.

In some possible embodiments, the second parsing sub-module is further configured to perform feature extraction on the target information to obtain a target feature point set; perform feature extraction on the target feature point set and a preset sample feature point set By comparison, the identification information of the application is determined.

In some possible embodiments, the second parsing submodule is further configured to perform image recognition on the target information to obtain identification information of the application.

In some possible embodiments, the information types include two or more preset types, and the parsing module 1702 includes a third parsing sub-module and a feature fusion sub-module, wherein: the third parsing sub-module is configured to be respectively Analyze the information of each preset type to obtain the analysis result of each of the information; the feature fusion sub-module is configured to perform feature fusion on the analysis result of each of the information to obtain the application program identification information.

It should be pointed out here that the descriptions of the above apparatus embodiments are similar to the descriptions of the above method embodiments, and have similar beneficial effects to the method embodiments. For technical details not disclosed in the device embodiments of the present application, please refer to the descriptions of the method embodiments of the present application for understanding.

It should be noted that, in the embodiments of the present application, if the above-mentioned information analysis method is implemented in the form of a software function module and sold or used as an independent product, it may also be stored in a computer-readable storage medium. Based on such understanding, the technical solutions of the embodiments of the present application may be embodied in the form of software products in essence or the parts that contribute to related technologies. The computer software products are stored in a storage medium and include several instructions to make The automatic test line of the device including the storage medium executes all or part of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read only memory (Read Only Memory, ROM), magnetic disk or optical disk and other media that can store program codes.

Correspondingly, an embodiment of the present application provides an information parsing device, and FIG. 18 is a schematic diagram of a hardware entity of the information parsing device provided by an embodiment of the present application. As shown in FIG. 18 , the hardware entity of the device 1800 includes: a processor 1801, a communication interface 1802 and memory 1803, where

The processor 1801 generally controls the overall operation of the device 1800.

The communication interface 1802 may enable the device 1800 to communicate with other terminals or servers over a network.

The memory 1803 is configured to store instructions and applications executable by the processor 1801, and can also cache data (eg, image data) to be processed or processed by the processor 1801 and each module in the device 1800, which can be stored in flash memory (FLASH) or random Access memory (Random Access Memory, RAM) implementation.

Correspondingly, the embodiments of the present application provide a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, implements the steps in the information analysis method provided in the foregoing embodiments.

It should be pointed out here that the descriptions of the above storage medium and device embodiments are similar to the descriptions of the above method embodiments, and have similar beneficial effects to the method embodiments. For technical details not disclosed in the embodiments of the storage medium and device of the present application, please refer to the description of the method embodiments of the present application to understand.

It is to be understood that reference throughout the specification to "one embodiment" or "an embodiment" means that a particular feature, structure or characteristic associated with the embodiment is included in at least one embodiment of the present application. Thus, appearances of "in one embodiment" or "in an embodiment" in various places throughout this specification are not necessarily necessarily referring to the same embodiment. Furthermore, the particular features, structures or characteristics may be combined in any suitable manner in one or more embodiments. It should be understood that, in various embodiments of the present application, the size of the sequence numbers of the above-mentioned processes does not mean the sequence of execution, and the execution sequence of each process should be determined by its functions and internal logic, and should not be dealt with in the embodiments of the present application. implementation constitutes any limitation. The above-mentioned serial numbers of the embodiments of the present application are only for description, and do not represent the advantages or disadvantages of the embodiments.

It should be noted that, herein, the terms "comprising", "comprising" or any other variation thereof are intended to encompass non-exclusive inclusion, such that a process, method, article or device comprising a series of elements includes not only those elements, It also includes other elements not expressly listed or inherent to such a process, method, article or apparatus. Without further limitation, an element qualified by the phrase "comprising a..." does not preclude the presence of additional identical elements in a process, method, article or apparatus that includes the element.

In the several embodiments provided in this application, it should be understood that the disclosed apparatus and method may be implemented in other manners. The device embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components may be combined, or Can be integrated into another system, or some features can be ignored, or not implemented. In addition, the coupling, or direct coupling, or communication connection between the components shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be electrical, mechanical or other forms. of.

The unit described above as a separate component may or may not be physically separated, and the component displayed as a unit may or may not be a physical unit; it may be located in one place or distributed to multiple network units; Some or all of the units may be selected according to actual needs to achieve the purpose of the solutions of the embodiments of the present application.

In addition, each functional unit in each embodiment of the present application may all be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above integration The unit can be implemented either in the form of hardware or in the form of hardware plus software functional units.

Alternatively, if the above-mentioned integrated units of the present application are implemented in the form of software function modules and sold or used as independent products, they may also be stored in a computer-readable storage medium. Based on such understanding, the technical solutions of the embodiments of the present application may be embodied in the form of software products in essence or the parts that contribute to related technologies. The computer software products are stored in a storage medium and include several instructions to make The automatic test line of the device performs all or part of the methods described in the various embodiments of the present application. The aforementioned storage medium includes various media that can store program codes, such as a removable storage device, a ROM, a magnetic disk, or an optical disk.

The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined under the condition of no conflict to obtain new method embodiments.

The features disclosed in several method or device embodiments provided in this application can be combined arbitrarily without conflict to obtain new method embodiments or device embodiments.

The above is only the embodiment of the present application, but the protection scope of the present application is not limited to this. Covered within the scope of protection of this application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Industrial Applicability

The embodiment of the present application parses the information of the preset type in the object to be parsed, quickly maps the acquired object to be parsed to the identification information of the application program and jumps to the application store, so as to solve the problem that the user needs to quickly obtain the identification information of the application program, but The problem that the QR code is damaged or the icon cannot be correctly parsed when the icon is updated is also applicable to scenarios without QR code.

Claims

An information parsing method, the method comprising:

Determine the information type of the information included in the object to be parsed; wherein, the object to be parsed is an image related to an application;

In the case that the information type includes a preset type, parse the information corresponding to the preset type in the object to be parsed, and determine the identification information of the application;

In response to receiving the instruction to search for the application, jump to the application store according to the identification information of the application.
The method of claim 1, wherein the image related to the application is an application poster, and the preset type of information in the application poster includes at least one of the following: a two-dimensional code, an application icon and text messages;

Wherein, the application icon is located on the layout of the application poster, and/or is located in the two-dimensional code.
The method according to claim 2, wherein, when the information type includes a preset type, parse the information corresponding to the preset type in the object to be parsed, and determine the identification information of the application program ,include:

In the case that the application program icon is located in the two-dimensional code, intercept the image area corresponding to the application program icon from the two-dimensional code;

The image area is analyzed to determine the identification information of the application.
The method according to claim 1 or 2, wherein the information types include two or more preset types, and when the information types include preset types, the preset types in the object to be parsed are determined. The corresponding information is parsed to determine the identification information of the application, including:

prioritizing the information corresponding to the two or more preset types;

Determine target information according to the sorting result; wherein, the priority of the target information is higher than the priority of other preset types of information in the object to be analyzed;

The target information is parsed to determine the identification information of the application.
The method of claim 4, wherein the information of the two or more preset types includes a two-dimensional code without the application icon, and the information corresponding to the two or more preset types is prioritized level sorting, including:

It is determined that the priority level of the two-dimensional code without the application program icon is lower than the priority level of other preset types of information.
The method according to claim 4, wherein the analyzing the target information to determine the identification information of the application program comprises:

Perform feature extraction on the target information to obtain a target feature point set;

The feature comparison of the target feature point set and the preset sample feature point set is performed to determine the identification information of the application program.
The method according to claim 4, wherein the analyzing the target information to determine the identification information of the application program comprises:

Perform image recognition on the target information to obtain identification information of the application.
The method according to claim 1 or 2, wherein the information types include two or more preset types, and when the information types include preset types, the preset types in the object to be parsed are determined. The corresponding information is parsed to determine the identification information of the application, including:

Respectively analyze each preset type of information to obtain an analysis result of each of the information;

Feature fusion is performed on the analysis results of each of the information to obtain the identification information of the application.
The method according to any one of claims 1 to 3, wherein after jumping to the application store according to the identification information of the application, the method further comprises:

In response to the instruction to download the application, the application is downloaded through the application store.
An information analysis device, the device includes a determination module, an analysis module and a jump module, wherein:

The determining module is configured to determine the information type of the information included in the object to be analyzed; wherein the object to be analyzed is an image related to an application program;

The parsing module is configured to parse the information corresponding to the preset type in the object to be parsed, and determine the identification information of the application when the information type includes a preset type;

The jumping module is configured to jump to the application store according to the identification information of the application in response to receiving an instruction to search for the application.
The apparatus of claim 10, wherein the image related to the application is an application poster, and the preset type of information in the application poster includes at least one of the following: a two-dimensional code, an application icon and text messages;

Wherein, the application icon is located on the layout of the application poster, and/or is located in the two-dimensional code.
The apparatus of claim 11, wherein the parsing module comprises an interception sub-module and a first parsing sub-module, wherein:

The interception submodule is configured to intercept the image area corresponding to the application icon from the two-dimensional code containing the application icon when the application icon is located in the two-dimensional code;

The first parsing submodule is configured to parse the image area to determine the identification information of the application.
The apparatus according to claim 10 or 11, wherein the information types include two or more preset types, and the parsing module includes a priority sorting sub-module, a determination sub-module and a second parsing sub-module, wherein:

The priority sorting sub-module is configured to, when the information types include two or more preset types, prioritize the information corresponding to the two or more preset types;

The determining submodule is configured to determine target information according to the sorting result; wherein, the priority of the target information is higher than the priority of other preset types of information in the object to be analyzed;

The second parsing submodule is configured to parse the target information to determine the identification information of the application.
The apparatus of claim 13, wherein the two or more preset types of information include a two-dimensional code without the application icon, and the priority sorting sub-module is further configured to determine that the application has no icon. The priority level of the QR code of the application icon is lower than the priority level of other preset types of information.
The apparatus according to claim 13, wherein the second parsing sub-module is further configured to perform feature extraction on the target information to obtain a target feature point set; and combine the target feature point set with a preset sample feature point The set performs feature comparison to determine the identification information of the application program.
The apparatus of claim 13, wherein the second parsing submodule is further configured to perform image recognition on the target information to obtain identification information of the application program.
The apparatus according to claim 10 or 11, wherein the information types include two or more preset types, and the parsing module includes a third parsing sub-module and a feature fusion sub-module, wherein:

The third parsing sub-module is configured to parse the information of each preset type respectively to obtain the parsing result of each of the information;

The feature fusion sub-module is configured to perform feature fusion on the analysis results of each of the information to obtain the identification information of the application.
The apparatus of any one of claims 10 to 12, wherein the apparatus further comprises a download module configured to download the application program through the application store in response to an instruction to download the application program.
An information analysis device, comprising a memory and a processor, the memory stores a computer program that can be run on the processor, and the processor executes the computer program to perform the following operations:

Determine the information type of the information included in the object to be parsed; wherein, the object to be parsed is an image related to the application; in the case where the information type includes a preset type, the preset type in the object to be parsed is determined The corresponding information is parsed to determine the identification information of the application; in response to receiving an instruction to search for the application, jump to the application store according to the identification information of the application.
A computer-readable storage medium having a computer program stored thereon, the computer program implementing the steps in the method of any one of claims 1 to 9 when executed by a processor.