CN113704486A

CN113704486A - Map data construction method and device and map data query method and device

Info

Publication number: CN113704486A
Application number: CN202110845314.1A
Authority: CN
Inventors: 金贺
Original assignee: Qingdao Haier Technology Co Ltd; Haier Smart Home Co Ltd
Current assignee: Qingdao Haier Technology Co Ltd; Haier Smart Home Co Ltd
Priority date: 2021-07-26
Filing date: 2021-07-26
Publication date: 2021-11-26

Abstract

The invention provides a map data construction method and device and a map data query method and device, wherein the map data construction method comprises the following steps: receiving log data sent by data acquisition equipment, wherein the log data is acquired by the data acquisition equipment from big data cluster equipment; determining relationship information between a plurality of target data table identifications and a plurality of target data table identifications based on the log data; determining target map data based on the plurality of target data table identifiers and the relationship information; and sending the target spectrum data to the spectrum database equipment so that the spectrum database equipment can construct a spectrum data network. The map data construction method and device and the map data query method and device provided by the invention can show the complete dependency relationship of the data table, so that the dependency relationship query is more convenient, the accuracy is higher, and the management cost is reduced.

Description

Map data construction method and device and map data query method and device

Technical Field

The invention relates to the technical field of big data, in particular to a map data construction method and device and a map data query method and device.

Background

At present, in the process of digital construction of various units, a data warehouse becomes an important link, a great deal of support is provided for data analysis in the processes of marketing, operation and the like of the units, the management difficulty of the data warehouse is correspondingly increased along with the increase of business requirements, the increase of the management difficulty is mainly reflected in the increase of the number of data tables and the complexity of the dependency relationship among the data tables, and the development and management cost of data developers is increased.

In the existing method for constructing and querying the data warehouse, the relationships among the tables are stored in a relational database, only one-degree dependency relationship can be expressed, complete dependency relationship cannot be expressed, repeated data tables cannot be searched, the accuracy is low, and the management cost is high.

Disclosure of Invention

The invention provides a map data construction method and device and a map data query method and device, which are used for solving the defects that the complete dependency relationship cannot be expressed, a repeated data table cannot be searched, the accuracy is low and the management cost is high in the prior art, realizing the representation of the complete dependency relationship of the data table, enabling the dependency relationship query to be more convenient, the accuracy to be high and reducing the management cost.

The invention provides a map data construction method, which comprises the following steps: receiving log data sent by data acquisition equipment, wherein the log data is acquired by the data acquisition equipment from big data cluster equipment; determining a plurality of target data table identifications and relationship information between the plurality of target data table identifications based on the log data; determining target atlas data based on the plurality of target data table identifications and the relationship information; and sending the target spectrum data to spectrum database equipment so that the spectrum database equipment can construct a spectrum data network.

According to the method for constructing the atlas data, the determining of the relationship information between a plurality of target data table identifications and a plurality of target data table identifications based on the journal data comprises the following steps: and under the condition that the log data is confirmed to have keywords, analyzing the log data to obtain a plurality of target data table identifications and relationship information among the target data table identifications.

According to the method for constructing the map data, the determining of the target map data based on the plurality of target data table identifications and the relationship information includes: screening out a dependent data table identifier with a dependent relation from the target data table identifiers based on the relation information; and determining target map data based on the dependency data table identification and the corresponding relationship information.

The invention also provides a map data query method, which comprises the following steps: receiving a relation query request; determining a reference data table identifier based on the relational query request; based on the reference data table identification, searching a corresponding associated data table identification from the map data network, wherein map network data is constructed by data processing equipment based on a plurality of target data table identifications and relation information between the target data table identifications, the target data table identifications and the corresponding relation information are obtained by the data processing equipment based on log data, and the log data is obtained by the data processing equipment from big data cluster equipment through data acquisition equipment; and determining a query result based on the reference data table identification and the associated data table identification, and sending the query result.

According to the method for querying the atlas data provided by the invention, the step of searching the corresponding associated data table identifier from the atlas data network based on the reference data table identifier comprises the following steps: and calculating the node relation in the graph data network based on the reference data table identification to obtain the associated data table identification corresponding to the reference data table identification.

According to the map data query method provided by the invention, the map data query method further comprises the following steps: receiving a similarity query request; determining similarity information between each two target data table identifications in the graph network data in response to the similarity query request; and sending the similarity information.

The present invention also provides a map data construction apparatus, including: the system comprises a first receiving module, a second receiving module and a third receiving module, wherein the first receiving module is used for receiving log data sent by data acquisition equipment, and the log data is acquired by the data acquisition equipment from big data cluster equipment; a first determining module, configured to determine, based on the log data, a plurality of target data table identifiers and relationship information between the plurality of target data table identifiers; a second determination module for determining target atlas data based on the plurality of target data table identifications and the relationship information; the first sending module is used for sending the target spectrum data to spectrum database equipment so that the spectrum database equipment can construct a spectrum data network.

The present invention also provides a map data query apparatus, including: the second receiving module is used for receiving the relation query request; a third determining module, configured to determine, based on the relationship query request, a reference data table identifier; the searching module is used for searching corresponding associated data table identifications from the map data network based on the reference data table identifications, the map network data are constructed by data processing equipment based on a plurality of target data table identifications and relationship information among the target data table identifications, the target data table identifications and the corresponding relationship information are obtained by the data processing equipment based on log data, and the log data are obtained by the data processing equipment from big data cluster equipment through data acquisition equipment; and the fourth determining module is used for determining a query result based on the reference data table identifier and the associated data table identifier and sending the query result.

The invention further provides an electronic device, which includes a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the map data constructing method or the map data querying method as described in any one of the above methods when executing the program.

The present invention also provides a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the map data construction method or the map data query method as any one of the above.

According to the map data construction method and device and the map data query method and device, the target data table identification and the corresponding relation information are obtained from the log data, and are stored in the form of the map data network, so that the complete dependency relationship of the data table can be expressed, the dependency relationship query is more convenient, the accuracy is higher, and the management cost is reduced.

Drawings

In order to more clearly illustrate the technical solutions of the present invention or the prior art, the drawings needed for the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and those skilled in the art can also obtain other drawings according to the drawings without creative efforts.

FIG. 1 is a schematic flow diagram of a method for constructing profile data according to the present invention;

FIG. 2 is a schematic diagram of the graph data construction and query process provided by the present invention;

FIG. 3 is a schematic structural diagram of an atlas data constructing apparatus provided in the invention;

FIG. 4 is a schematic flow chart diagram of a method for querying profile data provided by the present invention;

FIG. 5 is a schematic structural diagram of a map data query device provided by the present invention;

FIG. 6 is a schematic flow chart diagram of a method for processing map data provided by the present invention;

FIG. 7 is a schematic structural diagram of an atlas data processing apparatus provided in the invention;

FIG. 8 is a schematic diagram of the structure of an atlas data system provided by the invention;

fig. 9 is a schematic structural diagram of an electronic device provided by the present invention.

Detailed Description

In order to make the objects, technical solutions and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings, and it is obvious that the described embodiments are some, but not all embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

The method and apparatus for constructing map data and the method and apparatus for querying map data of the present invention are described below with reference to fig. 1, fig. 2, fig. 3, fig. 4, and fig. 5, the method and apparatus for processing map data of the present invention are described with reference to fig. 6 and fig. 7, the map data system of the present invention is described with reference to fig. 8, and the electronic device of the present invention is described with reference to fig. 9.

At present, the method for constructing and inquiring the dependency relationship among the data tables mainly obtains the relationship among the data tables by analyzing and processing logic, and stores the relationship among the tables in a relational database, but the method can only represent the dependency relationship once and cannot represent the complete dependency relationship, and the inquiry condition is mechanically single and cannot inquire the repeated data tables.

As shown in fig. 1 and 2, the present invention provides an atlas data constructing method that includes the following steps 110 to 140.

In step 110, log data sent by the data acquisition device is received, where the log data is obtained by the data acquisition device from the big data cluster device.

It can be understood that the big data cluster device may be an existing big data platform, the big data platform executes related programs in real time to obtain log data, the data acquisition device is in communication connection with the big data cluster device and can acquire the log data from the big data cluster device, and the data acquisition device can send the log data to the data processing device.

The data processing device may receive log data from the data collection device.

Step 120, determining a plurality of target data table identifications and relationship information between the plurality of target data table identifications based on the log data.

It can be understood that the data processing device may process the log data, and parse the plurality of target data table identifiers and the relationship information between the plurality of target data table identifiers from the log data, where a certain relationship exists between the target data tables, where the relationship between the plurality of target data table identifiers is found, and the relationship may be a relationship between two target data table identifiers, that is, a certain relationship exists between every two target data table identifiers, and is represented by the relationship information.

The target data table identifier may be a name of the target data table or a header of the target data table, or may also be a number of the target data table, which is not specifically limited herein, as long as the target data table can be accurately located, the target data table records various information in a table form, and the target data table identifier is not presented in a table form, but is presented only in a character string form, and is only used for representing the target data table, so as to facilitate retrieval and lookup of the target data table.

Here, the relationship information between the analyzed identifiers of the target data tables represents the relationship information between the target data tables.

Step 130, determining target atlas data based on the plurality of target data table identifications and the relationship information.

It can be understood that the relationship information corresponding to the target data table identifier is not one-dimensional but multi-dimensional, each data table identifier is not limited to have a relationship with another data table identifier, but has a certain relationship with a plurality of data table identifiers, and according to the relationship information corresponding to the target data table identifier, a mesh-like relationship graph can be constructed substantially.

And step 140, sending the target spectrum data to spectrum database equipment so that the spectrum database equipment can construct a spectrum data network.

It is to be understood that the data processing apparatus may transmit the target spectrum data to the spectrum database apparatus, where the target spectrum data is stored, and the spectrum database apparatus may have a storage function, where data, which is not in a single text or table form, is stored in the spectrum database apparatus, but is stored in a spectrum form, and the spectrum database apparatus may construct a spectrum data network based on the target spectrum data.

The graph data network can be a mesh node network and can be formed by a plurality of nodes and connecting lines among the nodes, the connecting lines among the nodes can have directions, and as target graph data are continuously written into the graph database equipment from the data processing equipment, the number of the nodes in the graph data network and the connecting lines among the nodes are increased, so that a relatively complete graph data network can be constructed.

According to the graph data network, the complete dependency relationship of each node can be conveniently found, namely the complete dependency relationship of the data table identification can be determined.

According to the map data construction method provided by the invention, the target data table identification and the corresponding relation information are obtained from the log data, and the target data table identification and the corresponding relation information are stored in the form of the map data network, so that the complete dependency relationship of the data table can be expressed, the dependency relationship query is more convenient, the accuracy rate is higher, and the management cost is reduced.

In some embodiments, the atlas data construction method may be applied in some scenarios where it is desirable to determine relationships between objects.

For example, the present home appliance may be applied to the field of home appliances, and the home appliance uses a physical network technology to access the home appliance to the internet, and a user may give an instruction for controlling the home appliance on a user terminal, so that a certain access right needs to be given to various user terminals and home appliances, and the user terminal or the user terminal and the home appliances are bound, and therefore, a relationship between the user terminal and the user terminal, a relationship between the user terminal and the home appliances, a relationship between the home appliances and the home appliances, and a relationship between the user terminal and a home group need to be considered.

The data table identifier here may be a user terminal identifier, a home appliance identifier, or a home group identifier, and the target map data stored in the corresponding map data network is the user terminal identifier, the home appliance identifier, or the home group identifier and the relationship therebetween.

For example, the method can be applied to the field of financial anti-fraud, and currently, in the field of financial, a unit or an individual may need to establish a guarantee relationship when loaning to a financial institution, so that the relationship between transaction subjects needs to be inquired, and whether relatives, friends or classmates exist between different transaction subjects is judged.

The data table identifier can be a transaction subject identifier, the target map data stored in the corresponding map data network is the relationship between the transaction subject identifier and the transaction subject identifier, and when the map data network is used for query, the queried relationship is the relationship between the transaction subject identifier and the transaction subject identifier.

For example, the method can be applied to the fields of enterprise background investigation and enterprise personnel recruitment, and when the background investigation is performed on the engaging personnel, the previous job units and positions of the personnel need to be known, and the relationships between the personnel need to be known, such as the superior-inferior relationship, the partner relationship or the relative relationship. This requires querying the person-to-person or person-to-institution relationships.

The data table identifier here may be a person identifier or an organization identifier, the target map data stored in the corresponding map data network is the relationship between the person identifier and the organization identifier, and when the map data network is used for query, the queried relationship is the relationship between the person identifier or the organization identifier.

In some embodiments, the determining 120 relationship information between the plurality of target data table identifications and the plurality of target data table identifications based on the log data includes: and under the condition that the key words exist in the log data, analyzing the log data to obtain a plurality of target data table identifications and relationship information among the plurality of target data table identifications.

It will be appreciated that the key is a value used to represent a particular data item for each record in the log data, and is defined in computer language, having a particular identifier, sometimes referred to as a reserved word.

After receiving the log data, the data processing device judges the content of the log data, judges whether a keyword exists in the log data, analyzes the log data if the keyword exists, and obtains a plurality of target data table identifications and relationship information between the plurality of target data table identifications

If the log data does not have the keywords, the log data without the keywords can be generated into message flow information, the message flow information is only used for recording the data processing process and does not participate in map data construction, and the data processing equipment can directly output the message flow information only for recording the data processing process.

In some embodiments, the determining 130 target atlas data based on the plurality of target data table identifications and the relationship information includes: screening out the dependent data table identifications with the dependent relations from the target data table identifications based on the relation information; and determining target map data based on the dependent data table identification and the corresponding relation information.

It can be understood that the target data table identifications can be judged through the relationship information, whether the target data table identifications have a dependency relationship with each other is judged according to the relationship information, and the target data table identifications having the dependency relationship are screened out to be used as the dependent data table identifications.

The object considered here is only the target data table identifier with dependency relationship, so the target map data can be obtained according to the dependency data table identifier and the relationship information between the dependency data table identifiers.

The target data table identification without dependency relationship can be used as an independent data table identification, and message flow information can be generated according to the independent data table identification, wherein the message flow information is only used for recording the data processing process and does not participate in the construction of map data, and the data processing equipment can directly output the message flow information only for recording the data processing process.

As shown in fig. 3, the map data constructing apparatus provided by the present invention will be described below, and the map data constructing apparatus described below and the map data constructing method described above may be referred to in correspondence with each other.

The invention provides a map data construction device, comprising: a first receiving module 310, a first determining module 320, a second determining module 330, and a first transmitting module 340.

The first receiving module 310 is configured to receive log data sent by a data acquisition device, where the log data is obtained by the data acquisition device from a big data cluster device;

a first determining module 320, configured to determine, based on the log data, relationship information between a plurality of target data table identifiers and a plurality of target data table identifiers;

a second determining module 330, configured to determine target atlas data based on the plurality of target data table identifications and the relationship information;

and the first sending module 340 is configured to send the target atlas data to the atlas database apparatus, so that the atlas database apparatus constructs an atlas data network.

The map data construction device provided by the embodiment of the application is used for executing the map data construction method, the specific implementation mode of the map data construction device is consistent with the method implementation mode, the same beneficial effects can be achieved, and details are not repeated here.

As shown in fig. 4, the present invention provides a map data query method, including: step 410 to step 440 are as follows.

Step 410, a relationship query request is received.

It can be understood that after the map data network in the map database device is constructed, the map data network can be used in a map data query process, the map database device can be connected with a server side, a user terminal can be in communication connection with the server side through internet WEB, a user can give a relationship query request in the user terminal, the user terminal can send the relationship query request to the server side through the internet WEB, and the server side can transmit the relationship query request to the map database device for query by the map database device.

Step 420, determining a reference data table identifier based on the relational query request.

It will be appreciated that the relational query request may comprise a node ID or a node name, and the graph database apparatus may find the corresponding reference data table identity in the graph data network in accordance with the relational query request.

Step 430, based on the reference data table identifier, searching a corresponding associated data table identifier from the graph data network, where the graph network data is constructed by the data processing device based on the multiple target data table identifiers and the relationship information between the multiple target data table identifiers, the multiple target data table identifiers and the corresponding relationship information are obtained by the data processing device based on log data, and the log data is obtained by the data processing device from the big data cluster device through the data acquisition device.

It is understood that the graph data network may be a mesh node network, and may be formed by a plurality of nodes and connecting lines between the nodes, and the connecting lines between the nodes may have directions, and as the target graph data is continuously written from the data processing apparatus to the graph database apparatus, the number of nodes in the graph data network and the connecting lines between the nodes may increase, so that a relatively complete graph data network may be constructed.

In the process of constructing the map data network, the data processing device may process log data, and analyze a plurality of target data table identifiers and relationship information between the plurality of target data table identifiers from the log data, where a certain relationship exists between the target data tables, where the relationship between the plurality of target data table identifiers is found, and the relationship may be a relationship between every two target data table identifiers, that is, a certain relationship exists between every two target data table identifiers, and is expressed by the relationship information.

The target data table identification is corresponding to relation information, the relation information is not single-dimensional, but multi-dimensional, each data table identification is not limited to be in relation with another data table identification, but has a certain relation with a plurality of data table identifications, a mesh-shaped relation graph can be constructed substantially according to the relation information corresponding to the target data table identification, target graph data can be generated according to the target data table identifications and the relation information, the target graph data can use the target data table identifications as nodes, the relation information is used as node connecting lines between the target data identifications, and the node connecting lines can have directions and represent certain directivity.

After the reference data table identifier is determined, the reference data table identifier can be searched in the atlas data network to find the associated data table identifier corresponding to the reference data table identifier, the associated data table identifier and the reference data table identifier have a dependency relationship, and the dependency relationship is embodied in the atlas data network constructed in advance.

Step 440, determining the query result based on the reference data table identifier and the associated data table identifier, and sending the query result.

It can be understood that, after the reference data table identifier and the associated data table identifier are determined, the query result corresponding to the relational query request is obtained, that is, a plurality of associated data table identifiers corresponding to the reference data table identifier and the relationship between each associated data table identifier and the reference data table identifier can be found.

The query result can be obtained according to the reference data table identification and the associated data table identification, the query result is fed back to the terminal, and the user can check the query result on the terminal, so that the reference data table identification and the corresponding associated data table identification which the user wants to check can be visually displayed to the user, and the query efficiency and accuracy are improved.

According to the atlas data query method provided by the invention, the associated data table identification corresponding to the reference data table identification is searched through the atlas data network, and the complete dependency relationship among the data table identifications can be queried, so that the query result is clearer, the query efficiency and accuracy are improved, and the user experience is improved.

In some embodiments, the graph data query method may be applied in some scenarios where it is desirable to determine relationships between objects.

The data table identifier here may be a user terminal identifier, a home appliance identifier, or a home group identifier, and the relationship between the user terminal identifier, the home appliance identifier, or the home group identifier is stored in the corresponding map data network, and when the map data network is used for query, the user terminal identifier, the home appliance identifier, or the home group identifier is queried.

The data table identifier here may be a person identifier or an organization identifier, and the corresponding graph data network stores the relationship between the person identifier and the organization identifier of the target graph data.

In some embodiments, the step 430 of finding a corresponding associated data table identifier from the graph data network based on the reference data table identifier includes: and calculating the node relation in the map data network based on the reference data table identification to obtain an associated data table identification corresponding to the reference data table identification.

It is understood that after the reference data table identifier is determined, node relationship calculation may be performed according to the graph data network, that is, other nodes having a dependency relationship with the reference data table identifier are calculated, and the other nodes may be used as the associated data table identifier.

Through the node relation calculation, the associated data table identification corresponding to the reference data table identification can be conveniently and quickly found.

In some embodiments, the profile data query method further comprises: receiving a similarity query request; determining similarity information between every two target data table identifications in the graph data network in response to the similarity query request; and sending the similarity information.

It can be understood that, while performing relationship query on the graph data network, similarity query may also be performed, which is different from the relationship query request, where the similarity query request is to calculate the similarity between every two nodes in the whole graph data network, and the relationship query request queries other nodes having a dependency relationship with a specific node.

After receiving the similarity query request, the graph database device may respond to the similarity query request, and may calculate similarity between every two nodes in the graph data network, where the similarity is not the determined similarity between some two nodes, and the similarity between all nodes in the graph data network may include the similarity between any one data table identifier and other data table identifiers.

The similarity information may be calculated using the following formula:

j (a, B) may represent the similarity between any two data table identifiers, where a and B are each any one data table identifier in the graph data network.

The following describes the map data query apparatus provided by the present invention, and the map data query apparatus described below and the map data query method described above may be referred to in correspondence with each other.

As shown in fig. 5, the present invention provides a map data query apparatus, including: a second receiving module 510, a third determining module 520, a lookup module 530, and a fourth determining module 540.

A second receiving module 510, configured to receive a relationship query request;

a third determining module 520, configured to determine, based on the relationship query request, a reference data table identifier;

a searching module 530, configured to search, based on the reference data table identifier, a corresponding associated data table identifier from an atlas data network, where atlas network data is constructed by the data processing apparatus based on relationship information between a plurality of target data table identifiers and the plurality of target data table identifiers, the plurality of target data table identifiers and the corresponding relationship information are obtained by the data processing apparatus based on log data, and the log data is obtained by the data processing apparatus from the big data cluster apparatus through the data acquisition apparatus;

and a fourth determining module 540, configured to determine a query result based on the reference data table identifier and the associated data table identifier, and send the query result.

The map data query device provided in the embodiment of the present application is used for executing the map data query method, and the specific implementation manner is consistent with the method implementation manner, and the same beneficial effects can be achieved, which is not described herein again.

As shown in fig. 6, the present invention provides a map data processing method, and the map data query includes the following steps 610 to 650.

In step 610, log data is obtained.

It is understood that the executing body of the map data processing method may be an electronic device with a logic computing function, such as a desktop computer, a cloud server, a notebook computer, a mobile phone, a tablet computer, or an industrial personal computer, and such an electronic device may be referred to as a data processing device herein.

The log data can be acquired from big data cluster equipment, the big data cluster equipment can be an existing big data platform, the big data platform executes related programs in real time to acquire the log data, the data acquisition equipment is in communication connection with the big data cluster equipment and can acquire the log data from the big data cluster equipment, and the data acquisition equipment can send the log data to the data processing equipment.

And step 620, under the condition that the keywords exist in the log data, analyzing the log data to obtain a plurality of target data table identifications and relationship information among the plurality of target data table identifications.

The data processing equipment can process the log data, analyze a plurality of target data table identifications and relationship information among the plurality of target data table identifications from the log data, wherein a certain relationship exists among the target data tables, and the relationship among the plurality of target data table identifications is found here, and the relationship can be a relationship between every two target data table identifications, namely, a certain relationship exists between every two target data table identifications, and is represented through the relationship information.

Step 630, based on the relationship information, a dependent data table identifier having a dependent relationship is screened out from the plurality of target data table identifiers.

And step 640, generating target map data based on the dependency data table identification and the corresponding relation information.

And step 650, outputting the target map data.

It is to be understood that the data processing device may output the target spectrum data, for example, the target spectrum data may be transmitted to a spectrum database device, the target spectrum data may be stored in the spectrum database device, the spectrum database device may have a storage function, the data stored in the spectrum database device is not in a single text or table form, but is stored in a spectrum form, and a spectrum data network may be constructed from the target spectrum data in the spectrum database device.

According to the map data processing method provided by the invention, whether the log data has the keywords or not is judged, the log data is analyzed under the condition that the keywords exist, a plurality of target data table identifications and corresponding relation information are obtained, the dependent data table identifications with the dependent relation are screened out according to the relation information, the target map data is generated according to the dependent data table identifications and the corresponding relation information, irrelevant data can be eliminated, the data without the dependent relation can be eliminated, and the effectiveness of the target map data can be improved.

In some embodiments, the atlas data processing method may be applied in some scenarios where it is desirable to determine relationships between objects.

The present invention provides an atlas data processing apparatus, and the atlas data processing apparatus described below and the atlas data processing method described above may be referred to in correspondence with each other.

As shown in fig. 7, the present invention provides an atlas data processing apparatus including: an acquisition module 710, a parsing module 720, a screening module 730, a generation module 740, and an output module 750.

The obtaining module 710 is configured to obtain log data.

And the analyzing module 720 is configured to analyze the log data to obtain a plurality of target data table identifiers and relationship information between the plurality of target data table identifiers when the log data is confirmed to have the keyword.

And the screening module 730 is configured to screen out the dependent relationship from the multiple target data table identifiers as a dependent data table identifier based on the relationship information.

A generating module 740, configured to generate target map data based on the dependency data table identifier and the corresponding relationship information.

And an output module 750 for outputting the target map data.

In some embodiments, the atlas data processing apparatus further comprises: the first message flow information generation module is used for generating message flow information based on the log data under the condition that no keyword exists in the log data, and the message flow information is used for recording the data processing process and does not participate in map data construction; and outputting the message flow information.

In some embodiments, the atlas data processing apparatus further comprises: the second message flow information generation module is used for screening out independent data table identifications without dependency relationship from the multiple target data table identifications based on the relationship information; generating message flow information based on the independent data table identification, wherein the message flow information is used for recording the data processing process and does not participate in map data construction; and outputting the message flow information.

In some embodiments, the output module is further to: and writing the target spectrum data into the spectrum database equipment so that the spectrum database equipment can construct a spectrum data network.

The map data processing device provided by the embodiment of the invention is used for executing the map data processing method, the specific implementation mode of the map data processing device is consistent with the method implementation mode, and the same beneficial effects can be achieved, and the detailed description is omitted here.

As shown in fig. 8, the present invention provides an atlas data system, comprising: data acquisition equipment, data processing equipment and map database equipment.

The data acquisition equipment, the data processing equipment and the map database equipment can be all electronic equipment with a logic operation function, and can be a computer, a cloud server or an industrial personal computer.

The data acquisition device is provided with a data input end which is used for being in communication connection with the big data cluster device to obtain log data.

It can be understood that the data acquisition device has a data input end, the data input end can be in communication connection with the big data cluster device, the big data cluster device can be an existing big data platform, the big data platform executes related programs in real time to obtain log data, the data acquisition device is in communication connection with the big data cluster device and can acquire the log data from the big data cluster device, and the data acquisition device can send the log data to the data processing device.

The input end of the data processing device is in communication connection with the output end of the data acquisition device, and the data processing device is configured to receive the log data and output target map data based on the log data.

It will be appreciated that the data processing device may receive log data from the data collection device.

The data processing device can process the log data to obtain target map data.

The target map data may use the target data table identifier as a node, use the relationship information as a node connection line between the target data identifiers, and the node connection line may have a direction and may represent a certain directivity.

An input of the atlas database apparatus is communicatively connected with an output of the data processing apparatus, the atlas database apparatus being arranged to build an atlas data network based on the target atlas data.

According to the map data system provided by the invention, the data acquisition equipment, the data processing equipment and the map data database equipment which are sequentially in communication connection are utilized to respectively complete the data acquisition, data processing and map data network construction, so that the calculated amount can be reasonably distributed, and the efficiency of map database construction is improved.

In some embodiments, the atlas data system may be applied in some scenarios where it is desirable to determine relationships between objects.

In some embodiments, the output of the data collection device sends the log data to the input of the data processing device by means of a message queue.

It is understood that a message queue is a container that holds messages during their transmission. The message queue manager acts as a man-in-the-middle in relaying a message from its source to its destination. The main purpose of the queues is to provide routing and guarantee delivery of messages; if the recipient is not available when the message is sent, the message queue will hold the message until it can be successfully delivered.

The log data can be sent to the data processing equipment in a message queue mode, the log data can be ensured to be continuously transmitted to the data processing equipment, and the stability and the integrity of the log data in the transmission process can be improved.

In some embodiments, the atlas data system further comprises: a message queue storage device.

The input end of the message queue storage device is in communication connection with the output end of the data acquisition device, the output end of the message queue storage device is in communication connection with the input end of the data processing device, and the message queue storage device is used for caching log data.

It will be appreciated that the message queue storage device may be used to buffer the message queue where the log data may be sent by the data collection device to the message queue storage device in the form of a message queue.

The data processing device can consume the log data from the message queue storage device in real time, and the log data can be transmitted more stably and reliably through the message queue storage device.

In some embodiments, the data processing apparatus is further arranged to: and obtaining relation information between the plurality of target data table identifications and the plurality of target data table identifications based on the log data, and outputting the target map data obtained based on the plurality of target data table identifications and the relation information.

The data processing apparatus may transmit the target spectrum data to the spectrum database apparatus, where the target spectrum data is stored, and the spectrum database apparatus may have a storage function, where data in a form of a spectrum, rather than a single text or a table, is stored, and the spectrum database apparatus may construct a spectrum data network based on the target spectrum data.

Fig. 9 illustrates a physical structure diagram of an electronic device, and as shown in fig. 9, the electronic device may include: a processor (processor)910, a communication Interface (Communications Interface)920, a memory (memory)930, and a communication bus 940, wherein the processor 910, the communication Interface 920, and the memory 930 communicate with each other via the communication bus 940. Processor 910 may invoke logic instructions in memory 930 to perform a method of graph data construction, the method comprising: receiving log data sent by data acquisition equipment, wherein the log data is acquired by the data acquisition equipment from big data cluster equipment; determining a plurality of target data table identifications and relationship information between the plurality of target data table identifications based on the log data; determining target atlas data based on the plurality of target data table identifications and the relationship information; and sending the target spectrum data to spectrum database equipment so that the spectrum database equipment can construct a spectrum data network.

Meanwhile, a profile data query method may be further performed, the method including: receiving a relation query request; determining a reference data table identifier based on the relational query request; based on the reference data table identification, searching a corresponding associated data table identification from the map data network, wherein map network data is constructed by data processing equipment based on a plurality of target data table identifications and relation information between the target data table identifications, the target data table identifications and the corresponding relation information are obtained by the data processing equipment based on log data, and the log data is obtained by the data processing equipment from big data cluster equipment through data acquisition equipment; and determining a query result based on the reference data table identification and the associated data table identification, and sending the query result.

Meanwhile, an atlas data processing method may also be performed, the atlas data processing method including: acquiring log data; under the condition that the log data is confirmed to have keywords, analyzing the log data to obtain a plurality of target data table identifications and relationship information among the target data table identifications; screening out the dependent data table identifications with the dependent relations from the target data table identifications based on the relation information; generating target map data based on the dependency data table identification and the corresponding relationship information; and outputting the target map data.

Furthermore, the logic instructions in the memory 930 may be implemented in software functional units and stored in a computer readable storage medium when the logic instructions are sold or used as independent products. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.

In another aspect, the present invention also provides a computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions which, when executed by a computer, enable the computer to perform the method for constructing atlas data provided by the above methods, the method comprising: receiving log data sent by data acquisition equipment, wherein the log data is acquired by the data acquisition equipment from big data cluster equipment; determining a plurality of target data table identifications and relationship information between the plurality of target data table identifications based on the log data; determining target atlas data based on the plurality of target data table identifications and the relationship information; and sending the target spectrum data to spectrum database equipment so that the spectrum database equipment can construct a spectrum data network.

In yet another aspect, the present invention also provides a non-transitory computer-readable storage medium having stored thereon a computer program which, when executed by a processor, is implemented to perform the above-provided atlas data construction method, the method comprising: receiving log data sent by data acquisition equipment, wherein the log data is acquired by the data acquisition equipment from big data cluster equipment; determining a plurality of target data table identifications and relationship information between the plurality of target data table identifications based on the log data; determining target atlas data based on the plurality of target data table identifications and the relationship information; and sending the target spectrum data to spectrum database equipment so that the spectrum database equipment can construct a spectrum data network.

The above-described embodiments of the apparatus are merely illustrative, and the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.

Claims

1. A map data construction method is characterized by comprising the following steps:

receiving log data sent by data acquisition equipment, wherein the log data is acquired by the data acquisition equipment from big data cluster equipment;

determining a plurality of target data table identifications and relationship information between the plurality of target data table identifications based on the log data;

determining target atlas data based on the plurality of target data table identifications and the relationship information;

and sending the target spectrum data to spectrum database equipment so that the spectrum database equipment can construct a spectrum data network.

2. The atlas data construction method of claim 1, wherein the determining, based on the log data, relationship information between a plurality of target data table identifications and a plurality of the target data table identifications comprises:

and under the condition that the log data is confirmed to have keywords, analyzing the log data to obtain a plurality of target data table identifications and relationship information among the target data table identifications.

3. The atlas data construction method of claim 1, wherein the determining target atlas data based on the plurality of target data table identifications and the relationship information comprises:

screening out a dependent data table identifier with a dependent relation from the target data table identifiers based on the relation information;

and determining target map data based on the dependency data table identification and the corresponding relationship information.

4. A method for querying spectrum data is characterized by comprising the following steps:

receiving a relation query request;

determining a reference data table identifier based on the relational query request;

based on the reference data table identification, searching a corresponding associated data table identification from the map data network, wherein map network data is constructed by data processing equipment based on a plurality of target data table identifications and relation information between the target data table identifications, the target data table identifications and the corresponding relation information are obtained by the data processing equipment based on log data, and the log data is obtained by the data processing equipment from big data cluster equipment through data acquisition equipment;

and determining a query result based on the reference data table identification and the associated data table identification, and sending the query result.

5. The graph data query method according to claim 4, wherein the searching for a corresponding associated data table identifier from the graph data network based on the reference data table identifier comprises:

and calculating the node relation in the graph data network based on the reference data table identification to obtain the associated data table identification corresponding to the reference data table identification.

6. The profile data query method according to claim 4, further comprising:

receiving a similarity query request;

determining similarity information between each two target data table identifications in the graph network data in response to the similarity query request;

and sending the similarity information.

7. An atlas data constructing apparatus, comprising:

the system comprises a first receiving module, a second receiving module and a third receiving module, wherein the first receiving module is used for receiving log data sent by data acquisition equipment, and the log data is acquired by the data acquisition equipment from big data cluster equipment;

a first determining module, configured to determine, based on the log data, a plurality of target data table identifiers and relationship information between the plurality of target data table identifiers;

a second determination module for determining target atlas data based on the plurality of target data table identifications and the relationship information;

the first sending module is used for sending the target spectrum data to spectrum database equipment so that the spectrum database equipment can construct a spectrum data network.

8. A map data query apparatus, comprising:

the second receiving module is used for receiving the relation query request;

a third determining module, configured to determine, based on the relationship query request, a reference data table identifier;

the searching module is used for searching corresponding associated data table identifications from the map data network based on the reference data table identifications, the map network data are constructed by data processing equipment based on a plurality of target data table identifications and relationship information among the target data table identifications, the target data table identifications and the corresponding relationship information are obtained by the data processing equipment based on log data, and the log data are obtained by the data processing equipment from big data cluster equipment through data acquisition equipment;

and the fourth determining module is used for determining a query result based on the reference data table identifier and the associated data table identifier and sending the query result.

9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the steps of the graph data constructing method according to any one of claims 1 to 3 or the graph data querying method according to any one of claims 4 to 6 are implemented when the processor executes the program.

10. A non-transitory computer readable storage medium having stored thereon a computer program, which when executed by a processor implements the steps of the graph data constructing method according to any one of claims 1 to 3 or the graph data querying method according to any one of claims 4 to 6.