CN113420099B

CN113420099B - Buried point data access control method and device, computer equipment and storage medium

Info

Publication number: CN113420099B
Application number: CN202110764232.4A
Authority: CN
Inventors: 林佳铖
Original assignee: Guangzhou Cubesili Information Technology Co Ltd
Current assignee: Guangzhou Cubesili Information Technology Co Ltd
Priority date: 2021-07-06
Filing date: 2021-07-06
Publication date: 2022-11-04
Anticipated expiration: 2041-07-06
Also published as: CN113420099A

Abstract

The application discloses a buried point data access control method and device, computer equipment and a storage medium, and relates to the field of Internet live broadcast; the method comprises the following steps: responding to a buried point data submission event, analyzing structured data and unstructured data, and adding the structured data and the unstructured data into a buried point data table of a relational data structure; enabling, by the data warehouse service engine, a data view, the data view being structured as a relational data structure, data of which references a self-buried data table, fields of which are fields belonging to structured data and/or fields belonging to unstructured data in the buried data table, the fields being preset in a field dictionary; querying an expression from a data warehouse service engine, and querying and acquiring result data from a data view specified by the query expression; and pushing the result data to the client for display. The method and the device can perform unified and standardized processing on multi-source data of the embedded points, and can improve the acquisition and access efficiency of the data of the embedded points of the Internet live broadcast application program product.

Description

Buried point data access control method and device, computer equipment and storage medium

Technical Field

The embodiment of the application relates to the field of internet live broadcast, in particular to a buried point data access control method and device, computer equipment and a storage medium.

Background

In order to control quality of various online services of the internet, embedded data collection codes are generally embedded in an application program (e.g., a webcast application program) or an access page corresponding to the online service, and when the relevant collection codes are run, relevant embedded data is collected and submitted to a cloud server for storage so as to perform data analysis. For such internet online service products, the data volume and reporting dimension of the embedded point basically occupy the largest proportion of the business data, and statistics, monitoring and analysis of the embedded point become one of indispensable works in each development stage of the internet products.

With the continuous development and enrichment of services provided by an internet platform, internet products are diversified, product versions are diversified, and the content reported by a buried point is continuously changed. The method has the advantages that massive buried point data with different types and different contents are concentrated in a data warehouse, how to optimize a buried point event management scheme is to ensure that the management is standard and clear, related data results can be efficiently output, the use difficulty of various users is reduced, and the method becomes a common concern in the field of internet data.

At present, enterprises with a certain scale in the internet industry start to build one or more systems in buried point management, data warehouse management and visual self-service query. However, these systems generally take the form of independent systems, and the three systems cannot be efficiently and organically integrated, which is not beneficial to improving the efficiency of collecting and analyzing buried point data while reducing the threshold of user usage.

Disclosure of Invention

The present application is directed to overcome at least some of the deficiencies in the prior art and provide a buried point data access control method, and a corresponding apparatus, computer device and storage medium.

In order to solve the technical problem, one technical scheme implemented by the application is as follows:

the application discloses a buried point data access control method, which comprises the following steps:

responding to a buried point data submission event, acquiring a data message and a buried point type carried in the data message, analyzing structured data and unstructured data contained in the data message, adding the structured data and the unstructured data into a buried point data table of a relational data structure corresponding to the buried point type, storing the structured data according to a field structure of the structured data, and integrally storing the unstructured data in a single field;

enabling, by the data warehouse services engine, one or more data views, each data view being structured as a relational data structure, data of which is referenced from a buried point data table of at least one buried point type, fields of which are fields belonging to structured data and/or fields belonging to unstructured data in the referenced buried point data table, the fields being preset in a field dictionary;

responding to a buried point data query request triggered by a client from a visual query template editing page, submitting a query expression constructed by the query template to a data warehouse service engine, and executing the query expression by the service engine to query and acquire result data from a data view specified by the query expression;

and pushing result data acquired by the query to the client for display so as to answer the query request.

In another embodiment of the present application, the buried point data access control method further includes the following steps:

responding to an application program access event for acquiring a buried point data acquisition code, and acquiring a field corresponding to the buried point data anchored for the application program;

verifying whether the corresponding field of the buried point data is a newly added field or not, and enabling the newly added field to become a referenceable field of the buried point data table, the data view and the field dictionary;

receiving an expression for acquiring the embedded point data corresponding to the newly added field to complete the configuration of the association relationship between the newly added field and the embedded point data corresponding to the newly added field;

and generating a buried point data acquisition code for the application program to reference according to the field corresponding to the buried point data anchored for the application program.

In a further specific embodiment, verifying whether a field corresponding to the buried point data is a newly added field, and enabling the newly added field to be a referenceable field of the buried point data table, the data view and the field dictionary includes the following steps:

judging whether a field corresponding to the buried point data is a referenceable field of the buried point data table, if not, configuring the newly-added field into the referenceable field of the buried point data table, and if so, continuing to execute the subsequent steps;

judging whether the field corresponding to the buried point data is a referenceable field of the data view, if not, configuring the newly-added field as the referenceable field of the data view, and if so, continuing to execute the subsequent steps;

judging whether the field corresponding to the buried point data is a referenceable field of a field dictionary, if not, configuring the newly added field as the referenceable field of the field dictionary, and if so, continuing to execute the subsequent steps;

configuring a referenceable field in the field dictionary into a referenceable field of the data view;

the referenceable fields in the data view are configured as referenceable fields of the buried point data table.

In a further extended embodiment, after generating the embedded point data acquisition code for the application to refer to according to the field corresponding to the embedded point data anchored for the application, the method further includes the following steps:

carrying out structure matching on a field corresponding to the embedded point data anchored for the application program and a data structure model of each data view, configuring an embedded point data table quoted by the data structure model and the data view matched with the data structure model as the embedded point data in a data message carried by an embedded point data submission event triggered after the embedded point data acquisition code is operated,

and otherwise, receiving a newly-built matched data structure model and a new buried point data table used for storing buried point data in a data message carried by a buried point data submission event triggered after the buried point data acquisition code is operated, and generating a new data view according to the data structure model.

In a preferred embodiment, the data warehouse service engine performs the following steps for a data view to reference data:

acquiring the structured data of each buried point data table quoted by the data view, and quoting the content of each field in the structured data in a one-to-one correspondence manner;

acquiring unstructured data of each buried point data table quoted by the data view, and analyzing the contents of a plurality of fields contained in the unstructured data;

and correspondingly referencing the contents of a plurality of fields contained in the unstructured data to each corresponding field preset in the data view.

In a further extended embodiment, the method for acquiring unstructured data of each buried point data table referred by the data view and analyzing the contents of a plurality of fields contained in the unstructured data includes the following steps:

determining each buried point data table quoted by the data view, and acquiring unstructured data in each buried point data table, wherein the unstructured data is packaged into a key-value-pair-based implementation format;

analyzing the acquired unstructured data according to the corresponding format specification, and extracting each key value pair in the unstructured data;

and resolving the key domain in each key value pair into a corresponding field, and resolving the value domain corresponding to the key domain into the content of the field.

In a preferred embodiment, the data warehouse service engine, when executing the query expression, includes the following steps:

analyzing the query expression, wherein the query expression comprises a statistical expression;

acquiring related data records from the specified data view according to the query expression;

performing statistical operation on the related data records according to a statistical expression in the query expression to obtain a statistical operation result;

and formatting the statistical operation result into result data.

the utility model provides a buried point data access control device, it includes:

the data acquisition module is used for responding to a buried point data submission event, acquiring a data message and a buried point type carried in the data message, analyzing structured data and unstructured data contained in the data message, adding the structured data and the unstructured data into a buried point data table of a relational data structure corresponding to the buried point type, storing the structured data according to a field structure of the structured data, and integrally storing the unstructured data in a single field;

a data mapping module configured to enable one or more data views by the data warehouse service engine, each data view being structured as a relational data structure, data of which is referenced from a buried point data table of at least one buried point type, fields of which are fields belonging to structured data and/or fields belonging to unstructured data in the referenced buried point data table, the fields being preset in a field dictionary;

the query request module is used for responding to a buried point data query request triggered by a client from a visual query template editing page, submitting a query expression constructed by the query template to a data warehouse service engine, and executing the query expression by the service engine so as to query and acquire result data from a data view specified by the query expression;

and the result feedback module is used for pushing the result data acquired by the query to the client for display so as to respond to the query request.

In another embodiment of the present application, the buried point data access control apparatus further includes an authentication synchronization module for performing the following functions, including: the access response submodule is configured to respond to an application program access event for acquiring the buried point data acquisition code and acquire a field corresponding to the buried point data anchored for the application program; the verification newly-added sub-module is used for verifying whether the fields corresponding to the buried point data are newly-added fields or not so that the newly-added fields become referenceable fields of the buried point data table, the data view and the field dictionary; the receiving configuration submodule is configured to receive an expression for acquiring the buried point data corresponding to the new added field so as to complete the configuration of the association relationship between the new added field and the buried point data corresponding to the new added field; and the code generation submodule is configured to generate the embedded point data acquisition code for the application program to reference according to the field corresponding to the embedded point data anchored for the application program.

In a further embodied embodiment, the verification newly added module is configured to implement the following specific functions: judging whether the field corresponding to the buried point data is a referenceable field of the buried point data table, if not, configuring the newly-added field as the referenceable field of the buried point data table, and if so, continuing to execute the subsequent steps; judging whether the field corresponding to the buried point data is a referenceable field of the data view, if not, configuring the newly-added field as the referenceable field of the data view, and if so, continuing to execute the subsequent steps; judging whether the field corresponding to the buried point data is a referenceable field of a field dictionary, if not, configuring the newly added field as the referenceable field of the field dictionary, and if so, continuing to execute the subsequent steps; configuring a referenceable field in the field dictionary as a referenceable field of the data view; the referenceable fields in the data view are configured as referenceable fields of the buried point data table.

In a further extended embodiment, the verification synchronization module further includes a structure matching sub-module configured to perform structure matching between a field corresponding to the embedded point data anchored for the application and the data structure model of each data view, configure the embedded point data table referred by the data view in which the data structure model is matched with the data view as a data structure table for storing embedded point data in a data message carried by an embedded point data submission event triggered after the embedded point data acquisition code is executed, and otherwise, receive a newly-created matched data structure model and a new embedded point data table for storing new embedded point data in a data message carried by an embedded point data submission event triggered after the embedded point data acquisition code is executed, and generate a new data view according to the data structure model.

In a preferred embodiment, the data warehouse service engine is configured to implement the following modules for referencing data for a data view: the structural reference submodule is used for acquiring the structural data of each buried point data table referenced by the data view and referencing the content of each field in the structural data in a one-to-one correspondence manner; the unstructured parsing submodule is used for acquiring unstructured data of each buried point data table quoted by the data view and parsing the contents of a plurality of fields contained in the unstructured data; and the unstructured reference sub-module is used for correspondingly referencing the contents of a plurality of fields contained in the unstructured data to each preset corresponding field of the data view.

In a further expanded embodiment, the unstructured parsing submodule implements the following functions: determining each buried point data table quoted by the data view, and acquiring unstructured data in each buried point data table, wherein the unstructured data is packaged into a format based on key-value pair implementation; analyzing the acquired unstructured data according to the corresponding format specification, and extracting each key value pair in the unstructured data; and resolving the key domain in each key value pair into a corresponding field, and resolving the value domain corresponding to the key domain into the content of the field.

In a preferred embodiment, the data warehouse service engine, when executing the query expression, operates to: analyzing the query expression, wherein the query expression comprises a statistical expression; acquiring related data records from the specified data view according to the query expression; performing statistical operation on the related data records according to a statistical expression in the query expression to obtain a statistical operation result; and formatting the statistical operation result into result data.

In order to solve the above technical problem, an embodiment of the present invention further provides a computer device, which includes a memory and a processor, where the memory stores computer readable instructions, and the computer readable instructions, when executed by the processor, cause the processor to execute the steps of the buried point data access control method.

In order to solve the above technical problem, an embodiment of the present invention further provides a storage medium storing computer readable instructions, which when executed by one or more processors, cause the one or more processors to execute the steps of the above-mentioned buried point data access control method.

Compared with the prior art, the method has the following advantages:

the method includes the steps that fields adopted by data messages, buried point data tables, data views and the like of buried point data submitting events are standardized and unified through a field dictionary, centralized and unified storage is conducted on unstructured data in the data messages and the buried point data tables through special fields, structured data are stored correspondingly one by one according to the fields, then in a data view for data presentation, the unstructured data are extracted from the buried point data tables in a mode that the fields serve as units and correspond to the corresponding fields of the data view, the data view establishes the fields based on a relational data structure, and the fields cover specific fields in cited structured data and specific fields in the cited unstructured data. Thus, the fields in the structured data can be planned as the basic data required for the buried point, and the unstructured data can be planned as the extended data required for the buried point, wherein the former provides a standardized framework and the latter provides extended performance, and is organically unified into a standardized information unit in the application. Therefore, data query is conveniently carried out on buried point data quoted by the data view by using a highly uniform query template, the whole query process is efficient and rapid, the format of the obtained result data can be highly uniform, the high coordination and uniformity of data storage, data management and data analysis are realized, the uniform systematic deployment of the originally discrete data processing modules is convenient, and the method is suitable for the requirement of the buried point data under the condition of serving internet online service products under more conditions.

In summary, through unified meta-information management, structured and unstructured data are introduced during buried point design, the characteristic that the data transmission protocol design and a main stream data warehouse service engine are compatible with structured and unstructured data is utilized, and through data warehouse design and development, reported data are successfully accessed to a visual self-service query platform. Therefore, in consideration of monitoring and analyzing user behaviors through massive buried point data, the extensible buried point element information is designed and managed by the internet product (application), and is connected to a visual self-service query platform in combination with a data warehouse technology, so that the function of reporting data to a statistical result full flow, namely, immediate semi-automatic or automatic output can be realized, the management pressure of the element data is reduced, the use threshold of reporting data by a user query multi-structure is reduced, and the management and data analysis/monitoring efficiency of the buried point data of the internet product is effectively improved.

Drawings

The foregoing and/or additional aspects and advantages of the present application will become apparent and readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings of which:

FIG. 1 is a schematic flow chart diagram of a buried point data access control method embodied in an exemplary embodiment of the present application;

FIG. 2 is a schematic diagram of a data structure mapping relationship of the present application, which is used to represent a data structure corresponding relationship between a data packet, a buried data table, a data view, and a field dictionary;

FIG. 3 is a diagram of one embodiment of an edit page for editing a query template according to the present application;

FIG. 4 is a schematic flow chart diagram illustrating an embodiment of obtaining a buried point data acquisition code according to the present application;

FIG. 5 is a flowchart of a reference nature programmed according to one embodiment of the present application from application access event triggering to the full flow of self-query;

FIG. 6 is a flowchart illustrating an embodiment of processing an added field;

FIG. 7 is a schematic diagram illustrating a determination process implemented by association between embedded data, an embedded data table, and a data view for an accessed application program product in an embodiment of the present application;

FIG. 8 is a schematic flow chart illustrating implementation of data referencing of a data view by a data warehouse service engine in an embodiment of the present application;

FIG. 9 is a schematic flow chart illustrating an embodiment of a process for parsing unstructured data by the data warehouse service engine;

FIG. 10 is a schematic flow chart diagram illustrating a process of parsing an execution query expression by the data warehouse service engine in an embodiment of the present application;

fig. 11 is a schematic diagram of a basic structure of a buried point data access control device according to the present application;

fig. 12 is a block diagram of a basic structure of a computer device according to an embodiment of the present application.

Detailed Description

Reference will now be made in detail to the embodiments of the present application, examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the drawings are exemplary only for the purpose of explaining the present application and are not to be construed as limiting the present application.

As used herein, the singular forms "a", "an", "the" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and/or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof. It will be understood that when an element is referred to as being "connected" or "coupled" to another element, it can be directly connected or coupled to the other element or intervening elements may also be present. Further, "connected" or "coupled" as used herein may include wirelessly connected or wirelessly coupled. As used herein, the term "and/or" includes all or any element and all combinations of one or more of the associated listed items.

It will be understood by those within the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the prior art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

As will be appreciated by those skilled in the art, "client," "terminal," and "terminal device" as used herein include both devices that are wireless signal receivers, which are devices having only wireless signal receivers without transmit capability, and devices that are receive and transmit hardware, which have receive and transmit hardware capable of two-way communication over a two-way communication link. Such a device may include: cellular or other communication devices such as personal computers, tablets, etc. having single or multi-line displays or cellular or other communication devices without multi-line displays; PCS (Personal Communications Service), which may combine voice, data processing, facsimile and/or data communication capabilities; a PDA (Personal Digital Assistant), which may include a radio frequency receiver, a pager, internet/intranet access, a web browser, a notepad, a calendar and/or a GPS (Global Positioning System) receiver; a conventional laptop and/or palmtop computer or other device having and/or including a radio frequency receiver. As used herein, a "client," "terminal device" can be portable, transportable, installed in a vehicle (aeronautical, maritime, and/or land-based), or situated and/or configured to operate locally and/or in a distributed fashion at any other location(s) on earth and/or in space. The "client", "terminal Device" used herein may also be a communication terminal, a Internet access terminal, and a music/video playing terminal, and may be, for example, a PDA, an MID (Mobile Internet Device), and/or a Mobile phone with music/video playing function, and may also be a smart television, a set-top box, and other devices.

The hardware referred to by the names "server", "client", "service node", etc. in the present application is essentially an electronic device with the performance of a personal computer, and is a hardware device having necessary components disclosed by the von neumann principles such as a central processing unit (including an arithmetic unit and a controller), a memory, an input device, and an output device, in which a computer program is stored in the memory, and the central processing unit loads a program stored in an external memory into the internal memory to run, executes instructions in the program, and interacts with the input and output devices, thereby accomplishing specific functions.

It should be noted that the concept of "server" as referred to in this application can be extended to the case of a server cluster. According to the network deployment principle understood by those skilled in the art, the servers should be logically divided, and in physical space, the servers may be independent from each other but can be called through an interface, or may be integrated into one physical computer or a set of computer clusters. Those skilled in the art will appreciate this variation and should not be so limited as to restrict the implementation of the network deployment of the present application.

The related technical scheme can be deployed in a cloud server, can realize data communication connection with a server related to business to coordinate online service, and can form a logically related server cluster with other related servers to provide service for related terminal equipment such as a smart phone, a personal computer, a third-party server and the like. The smart phone and the personal computer can both access the internet through a known network access mode, and establish a data communication link with the server of the application so as to access and use the service provided by the server.

For the server, a corresponding program interface is opened by a service engine providing an online service for remote invocation by various terminal devices, and the related technical solution applicable to be deployed in the server in the present application can be implemented in the server in this way.

The person skilled in the art will know this: although the various methods of the present application are described based on the same concept so as to be common to each other, they may be independently performed unless otherwise specified. In the same way, for each embodiment disclosed in the present application, it is proposed based on the same inventive concept, and therefore, concepts of the same expression and concepts of which expressions are different but are appropriately changed only for convenience should be equally understood.

Referring to fig. 1, a buried point data access control method of the present application, which is programmed to be implemented as an application program, is deployed in one or more servers of a distributed server cluster, and in an exemplary embodiment, includes the following steps:

step S1100, responding to a buried point data submission event, acquiring a data message and a buried point type carried in the data message, analyzing structured data and unstructured data contained in the data message, adding the data message and the unstructured data into a buried point data table of a relational data structure corresponding to the buried point type, storing the structured data according to a field structure of the structured data, and integrally storing the unstructured data in a single field:

for ease of understanding, please first refer to fig. 2 to understand the data structure of the present application. Fig. 2 is a diagram illustrating a data structure and a cooperative corresponding relationship between fields of a data structure and a data message carrying data of buried points in a data warehouse, which are implemented based on relational data structures such as SQL, oracle, foxpro, and the like.

In fig. 2, the data message is generally triggered by a buried point data acquisition code pre-embedded in a client application program or an access page, and the buried point data submission event is carried by the submission event and is obtained by analyzing by a server. The data message is standardized according to a pre-protocol format, so that the server can accurately analyze various specific meta information carried by the data message. The embedded point data mainly comprises data with two properties, namely basic data and extended data, wherein the basic data are standardized and are provided with a plurality of commonly specified fields, the content to be acquired is preset according to the structure of the standardized basic data when the embedded point data acquisition code is constructed, and the embedded point data acquisition code can be commonly used in a plurality of acquisition codes; the extension data can be customized to suit each internet product and each buried data acquisition code, and the field of the extension data is usually only specified by the internet product or the acquisition code. It can be understood that the extension data has extensibility with respect to the base data, and addition and deletion of the content included in the extension data is facilitated by adding or deleting fields in advance.

The basic data are represented by adopting structured data, the structured data are mainly based on the characteristics of a relational data structure, and tabular description is carried out according to the corresponding relation between the fields and the contents of the fields, so that the relational data structure is convenient to analyze. On the contrary, the extended data is characterized by adopting unstructured data, for example, map, JSON and other data structure formats based on key-value pairs are characterized as a character string, wherein the key domain of each key-value pair corresponds to a field in the relational data structure, and the value domain is the data content of the field, so that the data packet can treat the whole unstructured data as an integral field. Therefore, in the data message and the buried data table, for the structured data, the related data can be correspondingly stored according to each field contained in the structured data; and establishing a single field specially for the unstructured data, and storing the character string of the unstructured data in a centralized manner as the corresponding content of the single field. That is, for the data packet and the buried point data table, the unstructured data, that is, the data content of the extended data, does not need to be analyzed, so that various buried point data acquisition schemes can be processed in a compatible and unified manner, and a standardized technical architecture is laid for realizing unified data management.

The data message can also carry a buried point type attribute, so that the association between the data message and a buried point data table for storing the buried point data carried by the data message is established through the buried point type attribute, and the specific buried point data table which is stored by the buried point data in the data message can be quickly positioned according to the buried point data type. Generally speaking, the embedded point data acquisition code establishes the association in advance, that is, before the acquisition code is generated, the corresponding storage relationship between the data structure of the embedded point data acquired by the acquisition code and one or more embedded point data tables is matched in advance, so that after the embedded point data submission event submits the data message, the corresponding embedded point data table can be determined according to the embedded point type indicated in the data message, and the embedded point data carried in the data message is correspondingly stored in the embedded point data table.

The buried point data table, which is typically pre-designed, is used to store the original form of the buried point data, and in practice, in some embodiments, is implemented based on a relational data structure, and thus includes a plurality of fields, one of which is used to store the unstructured data exclusively, and the remaining fields are used to store the data content of the corresponding field in the structured data.

The data embedding table is generally matched with the data message according to the data structure of the data embedding table, so that each specific data in the data message can be correspondingly stored in the data embedding table, and theoretically, as long as the data structures are compatible, the data message can be stored in the same data embedding table no matter whether the data message is from the same internet product or the same data embedding collecting code. In general, since different acquisition codes may have different data structures of the acquired buried point data, the buried point data table is suitably designed by dividing according to different acquisition code types. Of course, in some embodiments, a buried point data table may be matched for different internet products separately, which may be implemented flexibly by those skilled in the art.

Step S1200, one or more data views are started by the data warehouse service engine, each data view is constructed into a relational data structure, data of the data views are referred from a buried point data table of at least one buried point type, fields of the data views are fields belonging to structured data and/or fields belonging to unstructured data in the referred buried point data table, and the fields are preset in a field dictionary:

with continued reference to fig. 2, in the data warehouse service engine implementing the relational data structure deployed in the present application, a data view function is provided accordingly, in which case various information of one or more buried point data tables may be presented by means of the data view, including original information and/or statistical information of the buried point data in the buried point data tables.

The data view is similarly constructed as a relational data structure, and comprises a plurality of fields, and each field is used for correspondingly storing the specific data content of each data record. Different from the buried point data table, in the data structure, the fields of the data view not only include the fields in the structured data, but also restore the fields corresponding to the key value pairs in the character string of the unstructured data into the data structure of the data view, so that each key value pair has a field corresponding to it, and the data content of the key value pair in the key value field can be stored as the data content of the field.

The data view may be a mapping to one or more pre-selected fields of the buried point data table, and this mapping relationship may be such that the data records in the data view may not only be in one-to-one correspondence with the data records in the buried point data table, but also, in some embodiments, may be in one-to-many mapping of one data record in the data view formed after counting the data records in the buried point data table. Depending on how a person skilled in the art predefines a specific mapping algorithm between the data view and the one or more buried data tables.

It can be seen that the data contents of the fields in the data view are actually referred to from one or more pre-specified buried point data tables, which may be pointed to by the same buried point type or by a plurality of buried point types. The data view can only comprise the structured data in the buried data table or only comprise the unstructured data in the data table, but the unstructured data is not stored in a single field in the data view in a whole, but is analyzed into a plurality of fields belonging to a relational data structure and is stored correspondingly. Under the condition, the buried point data acquired by the buried point data acquisition code belongs to the extended data part, so that the buried point data can be more visually presented in a data view, and query and reference are facilitated.

As shown in fig. 2, in order to implement the standardization, the present application uses a field dictionary to unify fields presented in a data packet, a buried data table, and a data view, specifically, the present application restricts that fields used in each link must be fields preset in the field dictionary, otherwise, the fields are not accepted by the corresponding link. Thus, in some embodiments, in business logic involving specific segments, such uniformity may be verified to ensure compliance with the standardized specifications.

The data warehouse service engine of the application is responsible for automatically enabling a data view preconfigured by a network management user, so that various data including the structured data and the unstructured data are automatically referred to from the buried point data table according to the preconfigured mapping relation, and the buried point data in the buried point data table is provided by a buried point data submission event as described above, so that it can be understood that the buried point data stored in the buried point data table can be indirectly accessed by accessing the data view.

Step 1300, responding to a buried point data query request triggered by a client from a visual query template editing page, submitting a query expression constructed by the query template to a data warehouse service engine, executing the query expression by the service engine, and querying and acquiring result data from a data view specified by the query expression:

in order to realize the calling of the data in the data view, the built query system provides open service for accessing the buried point data for the client, the client can edit a page through a visual query template provided by the query system, and after the query template is called, newly built and modified to construct a query expression, the query expression is submitted to a server where the query system is located, so that the purpose of data query is realized.

Referring to the example in fig. 3, the edit page constructs a query template edit area located on the left side of the graph and a query expression edit area located on the right side of the graph, and a chart area for graphically presenting result data returned after the query expression is executed is further configured below the query template edit area. For example, the client user may call the query template used in history through the query template editing area to modify the query template, or create and edit a new query template by itself, so as to implement partial editing of the query template. The editing operation corresponding to the editing area of the query template can automatically generate a corresponding query expression in the editing area of the query expression, and a client user can naturally edit the query expression more specifically in the editing area of the query template. And finally, the client user can store the editing result for later use. The client user can construct the buried data query request to submit a query expression to a remote server deployed in the application in a mode of automatically triggering submission by an editing page or through a submission control provided by the editing page, so that the remote server drives a data warehouse service engine to execute a query in a data view according to the query expression.

The query expression generally includes specification of a data view to be queried, a specific field in the data view, and may also be generally expressed in combination with time information, or further combined with some statistical operation expressions such as sorting summary, averaging, and the like, so as to instruct the data warehouse service engine to query data in some data views within a certain time period for the data warehouse service engine, and return corresponding query result data. Regarding the structure of the query expression, those skilled in the art are familiar with the principle of the relational database, and both can be flexibly implemented according to actual requirements, and are not repeated.

And after receiving the query expression, the data warehouse service engine executes corresponding query retrieval according to the data view specified by the query expression, acquires result data required by the specification conforming to the query expression, and returns the result data to the remote server in the original way.

Step S1400, pushing the result data obtained by the query to the client for display so as to respond to the query request:

after the remote server acquires the result data returned by the query executed by the remote server from the data warehouse service engine, the result data can be further formatted according to needs and then pushed to the client to complete the response to the query request so as to trigger the client to display the result data.

The method comprises the steps that data collection, data storage and query are carried out on data, the data are reported and submitted in a data message after being collected, the data are stored in a data-embedded data table, the data are converted into a data view display form, and then the data are queried and obtained through a query expression.

In an embodiment for facilitating implementation of buried point data collection, in comparison with the exemplary embodiment of the present application, please refer to the flowchart shown in fig. 4 and the program flowchart shown in fig. 5, where the buried point data access control method further includes the following steps:

step S2100, in response to the application access event for obtaining the buried point data acquisition code, obtains a field corresponding to the buried point data anchored for the application:

on the basis of constructing a unified service framework, the access of a newly added internet product can be opened, and a buried point data acquisition code is provided for the newly added internet product so as to provide service for the newly added internet product for acquisition and access of the buried point data.

For this purpose, the service framework may receive an access event of an application program that needs to obtain the buried point data acquisition code, and obtain a field corresponding to the buried point data anchored for the application program from the access event.

In order to anchor the field corresponding to the buried point data required to be collected by the application program, a configuration page can be opened for a corresponding management user to configure. The management user specifies specific fields of the buried point data required to be collected in the configuration page to realize the anchoring, and then when the access request is initiated, the fields are packaged together and contained in the access event to be transmitted to the server providing the service.

Step S2200, verifying whether the corresponding field of the buried point data is a new field, and making the new field become a referenceable field of the buried point data table, the data view and the field dictionary:

generally, the anchored field is typically an existing field in the field dictionary. In order to ensure that the definition of the anchored field is unified in the buried point data table, the data view and the field dictionary, whether the anchored field is a new field needs to be judged, and if the anchored field is not the new field, the step is skipped; otherwise, the newly added field needs to be ensured to exist in all the three, so that the field can be referred to in the buried data table, the data view and the field dictionary.

Step S2300, receiving an expression for obtaining the buried point data corresponding to the newly added field to complete the association configuration between the newly added field and the buried point data corresponding thereto:

when the anchored field is a new field, a developer may need to write a corresponding code expression for the anchored field so as to be referred to in the subsequently generated buried point data acquisition code, and when the acquisition code is executed, the buried point data of the corresponding field can be acquired by means of the referred expression. The association relationship between the newly added fields and the expressions can be submitted to the server of the application after being configured by developers, the server of the application can master the association relationship, and a follow-up party can serve for generation of the collected codes.

Step S2400, generating a buried point data acquisition code for the application program to reference according to the field corresponding to the buried point data anchored for the application program:

after the preparation work is completed, the server can generate corresponding embedded data acquisition codes according to the anchored fields, and the embedded data acquisition codes are used for acquiring data contents of the fields when the embedded data acquisition codes are operated. It will be appreciated that the generated buried point data collection code is suitable for reference by its corresponding application program, which the developer may embed into the development code of the application program.

Because the implementation of the embodiment is based on the exemplary embodiment of the application, the advantages of the service framework constructed by the application can be fully utilized, an efficient code acquisition mode is provided for program development, and the development efficiency of software engineering is improved.

Referring to fig. 5 and fig. 6, in a more specific embodiment of the present application, the step S2200 is implemented to include the following specific steps:

step S2211, determining whether the field corresponding to the buried point data is a referenceable field of the buried point data table, if not, configuring the newly added field as the referenceable field of the buried point data table, if so, continuing to execute the following steps:

and verifying the anchored fields one by one, judging whether each anchored field is a referenceable field of the buried point data table, and judging whether each anchored field belongs to the referenceable field of the buried point data table. If so, continuing to perform subsequent steps, and if not, indicating that the anchored field is at least a newly added field in the pool of referenceable fields, and therefore, can be added to the pool of referenceable fields as a referenceable field of the buried point data table.

Step S2212, whether the field corresponding to the buried point data is a referenceable field of the data view is judged, if not, the newly added field is configured to be the referenceable field of the data view, and if yes, the following steps are continuously executed:

it is further determined whether the same anchored field is a referenceable field of the data view. Similarly, the manner of determining whether the referenceable field belongs to the data view can be implemented by pre-constructing a referenceable field pool for the data view, and querying whether the anchored field exists in the referenceable field pool. If so, continuing to perform subsequent steps, and if not, indicating that the anchored field is at least a newly added field in the pool of referenceable fields, and therefore, can be added to the pool of referenceable fields as a referenceable field of the view data.

Step S2213, determining whether the field corresponding to the buried point data is a referenceable field of the field dictionary, if not, configuring the newly added field as a referenceable field of the field dictionary, if yes, continuing to execute the following steps:

and further judging whether the same anchored field is a referenceable field in the field dictionary, namely whether the same anchored field exists in the field dictionary, if so, continuing to execute the subsequent steps, otherwise, indicating that the anchored field is at least a newly added field in the field dictionary, and therefore, adding the anchored field into the field dictionary to be used as the referenceable field in the field dictionary.

Step S2214, configuring the referenceable field in the field dictionary as the referenceable field of the data view:

to ensure that the newly added field in the field dictionary is synchronized to the referenceable field pool of the data view, a further query may be made in this step to confirm whether the newly added field exists in the referenceable field pool of the data view, ensuring that it is synchronized to the referenceable field pool.

Step S2215, the referenceable field in the data view is configured to be the referenceable field of the buried point data table:

similarly, in order to ensure that the newly added field in the field dictionary and the referenceable field pool of the data view is synchronized to the referenceable field pool of the buried-point data table, it may be further queried in this step to confirm whether the newly added field exists in the referenceable field pool of the buried-point data table, ensuring that it is synchronized to the referenceable field pool.

The embodiment provides a specific program implementation business logic, and the business logic can ensure that the same anchored field can obtain standardized specifications in a field dictionary, a buried point data table, a data view and even the data message, and the implementation is simple and efficient.

Referring to fig. 5 and fig. 7, in a further expanded embodiment of the present application, the step S2400 is implemented to include the following steps:

step S2411, carrying out structure matching on the field corresponding to the buried point data anchored for the application program and the data structure model of each data view, configuring the buried point data table quoted by the data structure model and the data view matched with the data structure model as the buried point data in the data message carried by the buried point data submission event triggered after the buried point data acquisition code is operated,

step S2412, otherwise, receiving the newly-built matched data structure model and a new buried point data table used for storing buried point data in a data message carried by a buried point data submission event triggered after the buried point data acquisition code is operated, and generating a new data view according to the data structure model.

It is understood that the two steps in this embodiment may be actually implemented by the same conditional statement, which adapts whether the relational data structure formed by the anchored field can be matched with the data structure model of any existing data view, so as to determine whether to use the data structure model of the existing data view, or receive a new data structure model to implement a new data view. Generally, when a data structure model of a data view is newly added, the data structure of the corresponding buried point data table is also changed, so that not only the newly-added data structure model for generating the data view but also the newly-added data structure model for generating the buried point data table are required to be received, and then the new data view and the new buried point data table are generated according to the data structure models. Of course, if there is a matching buried data table, the service logic required by the new data view is considered mainly, and those skilled in the art can implement this flexibly.

Referring to fig. 1 and 8, in a further embodiment of the present application, the data warehouse service engine performs the following steps for referencing data for a data view:

step S1210, obtaining the structured data of each buried point data table referenced by the data view, and referencing the content of each field in the structured data in a one-to-one correspondence manner:

for structured data, each of the buried point data tables, which belongs to a corresponding field of the structured data portion, is mapped into the data view, and the data content is referenced with the fields corresponding to each other, i.e., each field in the data view correspondingly references each corresponding field belonging to the structured data from its associated buried point data table.

Step S1220, obtaining the unstructured data of each buried point data table referred by the data view, and analyzing the content of a plurality of fields included in the unstructured data:

for unstructured data, since it is stored in an independent field in the buried data table, when the data structure refers to this independent field, the data content in the independent field needs to be extracted and analyzed first, and each key-value pair is analyzed as correspondence data between one field and its corresponding data content in units of key-value pairs.

Step S1230, correspondingly referencing the contents of the fields included in the unstructured data to corresponding fields preset in the data view:

as described above, the data view is organized according to a relational data structure, and therefore, fields corresponding to the key value pairs are configured together, so that the fields and data contents in the key value pairs analyzed from the buried point data table can be referred to by the corresponding fields in the data view, thereby converting unstructured data in the buried point data table into structured data in the data view.

The embodiment further realizes the conversion service logic for converting the unstructured data in the buried point data table into the structured data in the data view, realizes the logic, enables the reference of the unstructured data to be more uniform, can organize and present various unstructured data by means of the flexibility of the data view, and enables the realization of a uniform service framework to be standardized.

Referring to fig. 8 and fig. 9, in a further embodiment, the step S1220 includes the following specific steps:

step S1221, determining each buried point data table referred by the data view, and acquiring unstructured data in each buried point data table, where the unstructured data is encapsulated into a format implemented based on key-value pairs:

as mentioned above, each data view may be associated with one or more buried point data tables, and the association mapping is configured in advance mainly when the data view is constructed, so that when the data view is called, the respective buried point data tables referred to by the data view are naturally determined.

Similarly, as mentioned above, the unstructured data is suitably packaged in a data format such as Map, JSON, or the like, in which case the unstructured data is represented as a character string, the character string includes a plurality of key-value pairs, and a key field in each key-value pair represents a field, and a value field of the key-value pair is used for storing data content corresponding to the field. Therefore, when unstructured data in each buried point data table is acquired, a character string corresponding to the unstructured data is actually acquired.

Step S1222, parsing the obtained unstructured data according to the corresponding format specification, and extracting each key value pair:

it can be understood that the acquired unstructured data can be analyzed to extract each key value pair therein by adapting to the corresponding format specifications of Map or JSON and the like.

Step S1223, parsing the key domain in each key value pair into a corresponding field, and parsing the value domain corresponding to the key domain into the content of the field:

naturally, it can also be understood that the key field in each key value pair is analyzed as a corresponding field, and the value field corresponding to the key field is analyzed as the data content of the field, so that when the key value pair is referenced to the data view in the following, the data content of the key value pair can be correspondingly filled into the same field, and the conversion is completed.

Because the unstructured data storage mode based on the key value pair has the characteristic of platform independence, the universal adaptability of the unified service framework realized by the application can be further enriched.

Referring to fig. 1 and 10, in an embodiment of the present application for enhancing the data query function, when the data warehouse service engine executes the query expression, the method includes the following steps:

step S1311, analyzing the query expression, wherein the query expression comprises a statistical expression:

as mentioned above, the present application is suitably implemented by using SQL, oracle, foxpro, and other relational database service engines, in which case, these service engines themselves support parsing of query expressions written according to their syntax specifications, and also support adding some statistical expressions to the query expressions, so that the service engines perform corresponding statistical operations during query execution. Therefore, the query expression of the process submitted by the client can be analyzed by the data warehouse service engine of the application.

Step S1312, obtaining relevant data records from the data view specified by the query expression:

therefore, the data warehouse service engine can acquire relevant data records meeting the requirements of other constraint conditions of the query expression from the specified data views according to the data view specified information analyzed by the query expression.

Step S1313, performing statistical operation on the relevant data records according to the statistical expression in the query expression, to obtain a statistical operation result:

as described above, if the query expression carries a statistical expression, the data warehouse service engine may perform statistical operations, such as sorting, summarizing, etc., on the relevant data records, and finally obtain a statistical operation result. It should be understood that the result of the operation may still be expressed as a number of data records, depending on the specific content of the query expression.

Step S1314, formatting the statistical operation result into result data:

in order to facilitate presentation of the client, statistical operation results obtained through statistics can be further formatted, the statistical operation results are formatted to accord with result data which can be analyzed and presented in a form such as a chart on a client presentation page, and therefore the client is triggered to analyze and display the corresponding chart after the result data are pushed to the client.

Through the embodiment, the technical preparation for enabling the buried point data query result to have the visualization capacity is perfected, the efficiency of interaction based on query is improved, and the user interaction experience is improved.

Referring to fig. 11, in an exemplary embodiment, an embedded data access control apparatus includes a data acquisition module 1100, a data mapping module 1200, a query request module 1300, and a result feedback module 1400, where: the data acquisition module 1100 is configured to respond to a buried point data submission event, acquire a data packet and a buried point type carried in the data packet, analyze structured data and unstructured data included in the data packet, add the data packet to a buried point data table of a relational data structure corresponding to the buried point type, store the structured data according to a field structure of the structured data, and store the unstructured data in a single field as a whole; the data mapping module 1200 is configured to enable one or more data views by the data warehouse service engine, wherein each data view is constructed as a relational data structure, data of the data view is referenced from a buried point data table of at least one buried point type, and fields of the referenced buried point data table belong to fields of structured data and/or fields of unstructured data, and the fields are preset in a field dictionary; the query request module 1300 is configured to respond to a buried point data query request triggered by a client from a visual query template editing page, submit a query expression constructed by the query template to a data warehouse service engine, and execute the query expression by the service engine to query and obtain result data from a data view specified by the query expression; the result feedback module 1400 is configured to push the result data obtained by the query to the client for displaying so as to respond to the query request.

In another embodiment of the present application, the buried point data access control apparatus further includes an authentication synchronization module for performing the following functions, including: the access response submodule is configured to respond to an application program access event for acquiring the buried point data acquisition code and acquire a field corresponding to the buried point data anchored for the application program; the verification newly-added sub-module is used for verifying whether the fields corresponding to the buried point data are newly-added fields or not, so that the newly-added fields become the referenceable fields of the buried point data table, the data view and the field dictionary; the receiving configuration submodule is configured to receive an expression for acquiring the buried point data corresponding to the new added field so as to complete the configuration of the association relationship between the new added field and the buried point data corresponding to the new added field; and the code generation submodule is configured to generate the embedded point data acquisition code for the application program to reference according to the field corresponding to the embedded point data anchored for the application program.

In a further embodiment, the verification newly added module is configured to implement the following specific functions: judging whether a field corresponding to the buried point data is a referenceable field of the buried point data table, if not, configuring the newly-added field into the referenceable field of the buried point data table, and if so, continuing to execute the subsequent steps; judging whether the field corresponding to the buried point data is a referenceable field of the data view, if not, configuring the newly-added field as the referenceable field of the data view, and if so, continuing to execute the subsequent steps; judging whether a field corresponding to the buried point data is a referenceable field of a field dictionary, if not, configuring the newly added field as the referenceable field of the field dictionary, and if so, continuing to execute the subsequent steps; configuring a referenceable field in the field dictionary into a referenceable field of the data view; the referenceable fields in the data view are configured as referenceable fields of the buried point data table.

In a preferred embodiment, the data warehouse service engine is configured to implement the following modules for referencing data for a data view: the structural reference submodule is used for acquiring the structural data of each buried point data table referenced by the data view and referencing the contents of each field in the structural data in a one-to-one correspondence manner; the unstructured parsing submodule is used for acquiring unstructured data of each buried point data table quoted by the data view and parsing the contents of a plurality of fields contained in the unstructured data; and the unstructured reference sub-module is used for correspondingly referencing the contents of a plurality of fields contained in the unstructured data to each preset corresponding field of the data view.

In a further expanded embodiment, the unstructured parsing submodule implements the following functions: determining each buried point data table quoted by the data view, and acquiring unstructured data in each buried point data table, wherein the unstructured data is packaged into a key-value-pair-based implementation format; analyzing the acquired unstructured data according to the corresponding format specification, and extracting each key value pair in the unstructured data; and resolving the key domain in each key value pair into a corresponding field, and resolving the value domain corresponding to the key domain into the content of the field.

In a preferred embodiment, the data warehouse service engine, when executing the query expression, performs the following functions: analyzing the query expression, wherein the query expression comprises a statistical expression; acquiring related data records from the specified data view according to the query expression; performing statistical operation on the related data records according to a statistical expression in the query expression to obtain a statistical operation result; and formatting the statistical operation result into result data.

In order to solve the technical problem, an embodiment of the present application further provides a computer device. As shown in fig. 12, the internal structure of the computer device is schematic. The computer device includes a processor, a non-volatile storage medium, a memory, and a network interface connected by a system bus. The non-volatile storage medium of the computer device stores an operating system, a database and computer readable instructions, the database can store control information sequences, and when the computer readable instructions are executed by a processor, the processor can realize a buried point data access control method. The processor of the computer device is used for providing calculation and control capability and supporting the operation of the whole computer device. The memory of the computer device may have stored therein computer readable instructions, which, when executed by the processor, may cause the processor to perform the buried point data access control method of the present application. The network interface of the computer device is used for connecting and communicating with the terminal. It will be appreciated by those skilled in the art that the configuration shown in fig. 12 is a block diagram of only a portion of the configuration associated with the present application, and is not intended to limit the computing device to which the present application may be applied, and that a particular computing device may include more or fewer components than shown, or may combine certain components, or have a different arrangement of components.

In this embodiment, the processor is configured to execute specific functions of each module and its sub-module in fig. 11, and the memory stores program codes and various data required for executing the modules or the sub-modules. The network interface is used for data transmission to and from a user terminal or a server. The memory in this embodiment stores program codes and data necessary for executing all modules/submodules in the embedded data access control device of the present application, and the server can call the program codes and data of the server to execute the functions of all the submodules.

The present application further provides a storage medium storing computer readable instructions which, when executed by one or more processors, cause the one or more processors to perform the steps of the buried data access control method of any of the embodiments of the present application.

It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments of the present application can be implemented by a computer program, which can be stored in a computer-readable storage medium, and when the computer program is executed, the processes of the embodiments of the methods can be included. The storage medium may be a non-volatile storage medium such as a magnetic disk, an optical disk, a Read-Only Memory (ROM), or a Random Access Memory (RAM).

In summary, the data acquisition, data storage and data analysis are uniformly and standardizedly processed on multi-source data of the embedded point, the acquisition and access efficiency of the data of the embedded point of the internet application program product can be improved, and the capability of serving massive internet application program products with a uniform service framework is achieved.

Those of skill in the art will appreciate that the various operations, methods, steps in the processes, acts, or solutions discussed in this application can be interchanged, modified, combined, or eliminated. Further, other steps, measures, or schemes in various operations, methods, or flows that have been discussed in this application can be alternated, altered, rearranged, broken down, combined, or deleted. Further, steps, measures, schemes in the prior art having various operations, methods, procedures disclosed in the present application may also be alternated, modified, rearranged, decomposed, combined, or deleted.

The foregoing is only a few embodiments of the present application and it should be noted that those skilled in the art can make various improvements and modifications without departing from the principle of the present application, and that these improvements and modifications should also be considered as the protection scope of the present application.

Claims

1. A buried point data access control method is characterized by comprising the following steps:

responding to a buried point data submission event, acquiring a data message and a buried point type carried in the data message, analyzing structured data and unstructured data contained in the data message, adding the data message and the unstructured data into a buried point data table of a relational data structure corresponding to the buried point type, storing the structured data according to a field structure of the structured data, and integrally storing the unstructured data in a single field;

2. The buried point data access control method according to claim 1, further comprising the steps of:

3. The buried point data access control method of claim 2, wherein verifying whether the field corresponding to the buried point data is a new field, and making the new field a referenceable field of the buried point data table, the data view and the field dictionary comprises the following steps:

judging whether the field corresponding to the buried point data is a referenceable field of the buried point data table, if not, configuring the newly-added field as the referenceable field of the buried point data table, and if so, continuing to execute the subsequent steps;

judging whether a field corresponding to the buried point data is a referenceable field of a field dictionary, if not, configuring the newly added field as the referenceable field of the field dictionary, and if so, continuing to execute the subsequent steps;

configuring a referenceable field in the field dictionary as a referenceable field of the data view;

4. The embedded point data access control method as claimed in claim 2, further comprising the following steps after generating the embedded point data acquisition code for the application program to refer to according to the field corresponding to the embedded point data anchored for the application program:

5. The buried point data access control method of any one of claims 1 to 4, wherein the data warehouse service engine performs the following steps for a data view to reference data:

and correspondingly referencing the contents of a plurality of fields contained in the unstructured data to each preset corresponding field of the data view.

6. The buried point data access control method of claim 5, wherein the step of obtaining the unstructured data of each buried point data table referenced by the data view and analyzing the contents of a plurality of fields contained in the unstructured data comprises:

determining each buried point data table quoted by the data view, and acquiring unstructured data in each buried point data table, wherein the unstructured data is packaged into a format based on key-value pair implementation;

7. The buried point data access control method of any one of claims 1 to 4, wherein the data warehouse service engine, when executing the query expression, comprises the following steps:

and formatting the statistical operation result into result data.

8. A buried point data access control apparatus, comprising:

9. A computer device comprising a memory and a processor, the memory having stored therein computer readable instructions which, when executed by the processor, cause the processor to carry out the steps of the buried data access control method according to any one of claims 1 to 7.

10. A storage medium having computer-readable instructions stored thereon which, when executed by one or more processors, cause the one or more processors to perform the steps of the buried data access control method of any one of claims 1 to 7.