CN116450890A

CN116450890A - Graph data processing method, device and system, electronic equipment and storage medium

Info

Publication number: CN116450890A
Application number: CN202310293916.XA
Authority: CN
Inventors: 张敬; 肖勃飞; 贾栩杰; 罗晶; 陈建君; 李宾
Original assignee: Zhongdian Jinxin Software Co Ltd
Current assignee: Zhongdian Jinxin Software Co Ltd
Priority date: 2023-03-23
Filing date: 2023-03-23
Publication date: 2023-07-18

Abstract

The application discloses a graph data processing system, which belongs to the technical field of data processing. The system comprises: the system manages processing configuration information such as data source information, a graph data structure, a constructed graph instance, field mapping configuration information of the graph data structure of an input data source and the graph instance and the like through the data source management engine, so that the graph database processing engine can acquire the data source information according to the processing configuration information, acquire offline data and real-time data, combine the offline data and the real-time data into one data stream, convert the offline data and the real-time data into graph data according to the field mapping configuration information and store the graph data into the graph data storage rear end, and the query application can query the graph data storage rear end according to a received query statement.

Description

Graph data processing method, device and system, electronic equipment and storage medium

Technical Field

The present invention relates to the field of data processing technologies, and in particular, to a graph data processing method, device, system, electronic device, and computer readable storage medium.

Background

Graph data processing refers to processing a data source and outputting the data in the form of a graph. In the prior art, when the graph data is processed, the data is stored and used by adopting a classical lambda architecture (a commonly used big data architecture), and offline data and real-time data are respectively processed and used. The method mainly uses a mode of loading data files in batches for offline data processing, and then the offline data in batches are imported into an offline library at a batch time point, so that the original graph data is expanded and analyzed in the next step to provide graph data service. When the graph data retrieval analysis is performed, all data can not be used in real time no matter the graph data retrieval analysis is performed on offline data or real-time data, so that the real-time performance and accuracy of the data are not perfect. In addition, in the prior art, two sets of data processing logic are used for processing the graph data, so that the maintenance cost is high.

Disclosure of Invention

The embodiment of the application provides a graph data processing method, a device, a system, electronic equipment and a computer readable storage medium, which are beneficial to improving the comprehensiveness and the integrity of an output graph data analysis result and can reduce the maintenance cost of a graph data processing system.

In a first aspect, an embodiment of the present application discloses a graph data processing method, including:

acquiring data source information of an offline data source and data source information of a real-time data source, which are used as input data sources when performing graph data processing, a graph data structure of a graph example, and processing configuration information related to the input data source and the graph data structure according to configuration operation of configuration personnel;

reading the offline data source according to the data source information of the offline data source to obtain a first data stream, and reading the real-time data source according to the data source information of the real-time data source to obtain a second data stream;

carrying out association combination on the first data stream and the second data stream according to the association relation between each field in the first data stream and the second data stream to obtain a combination processing result;

Performing disassembly processing on the merging processing result according to the processing configuration information to generate graph data matched with the graph instance;

storing the graph data and indexes corresponding to the graph examples;

and responding to a user-triggered query request, and querying the stored graph data based on the index to acquire a query result corresponding to the query request.

In a second aspect, an embodiment of the present application discloses a graph data processing apparatus, the apparatus including:

the configuration management module is used for acquiring data source information of an offline data source and data source information of a real-time data source, a graph data structure of a graph instance and processing configuration information related to the input data source and the graph data structure when performing graph data processing according to configuration operation of configuration personnel;

the data stream acquisition module is used for reading the offline data source according to the data source information of the offline data source to obtain a first data stream, and reading the real-time data source according to the data source information of the real-time data source to obtain a second data stream;

the data stream merging module is used for carrying out association merging on the first data stream and the second data stream according to the association relation between the fields in the first data stream and the second data stream to obtain a merging processing result;

The diagram data generating module is used for carrying out disassembly processing on the merging processing result according to the processing configuration information to generate diagram data matched with the diagram instance;

the map data and index storage module is used for storing the map data and indexes corresponding to the map examples;

and the query output module is used for responding to a user-triggered query request and querying the stored graph data based on the index so as to acquire a query result corresponding to the query request.

In a third aspect, embodiments of the present application disclose a graph data processing system, the system comprising: a data source management engine, a graph database processing engine, a graph data store backend, an index store backend, and a query application, wherein,

the data source management engine is used for managing data source information of an offline data source serving as an input data source and data source information of a real-time data source when performing graph data processing, a graph data structure of a pre-configured graph instance, and processing configuration information related to the input data source and the graph data structure;

the map database processing engine is used for reading the offline data source according to the data source information of the offline data source to obtain a first data stream, and reading the real-time data source according to the data source information of the real-time data source to obtain a second data stream;

The graph database processing engine is further used for carrying out combination processing on the first data stream and the second data stream to obtain a combination processing result;

the map database processing engine is further used for carrying out disassembly processing on the merging processing result according to the processing configuration information to generate map data matched with the map instance;

the graph data storage back end is used for storing the graph data;

the index storage back end is used for storing the index corresponding to the graph instance;

the query application is used for querying the rear end of the graph data storage based on a query request triggered by a user so as to acquire a query result corresponding to the query request;

the map data storage back end is further configured to, in response to the query of the query application, retrieve the query result satisfying the query request from the locally stored map data according to the index stored in the index storage back end.

In a fourth aspect, the embodiment of the application further discloses an electronic device, which includes a memory, a processor, and a computer program stored in the memory and capable of running on the processor, where the processor implements the graph data processing method described in the embodiment of the application when executing the computer program.

In a fifth aspect, embodiments of the present application disclose a computer readable storage medium having stored thereon a computer program which, when executed by a processor, performs the steps of the graph data processing method disclosed in embodiments of the present application.

The graph data processing system disclosed in the embodiment of the application comprises: the system manages configuration information through the data source management engine, and comprises: the method comprises the steps that data source information, a graph data structure, a constructed graph instance, and field mapping configuration information of the graph data structure of an input data source and the graph instance are obtained, then, a graph database processing engine can collect offline data and real-time data according to the data source information and combine the offline data and the real-time data into one data stream, and then, the data stream is converted into graph data to be stored at the back end of a graph data storage, so that a query application can query the back end of the graph data storage according to a received query statement, the back end of the graph data storage can return a query result based on the graph data generated by combining the offline data and the real-time data, and the comprehensiveness and the completeness of the output graph data query result are improved. Because the offline data and the real-time data use the common storage position, complete data can be always obtained when graph analysis and traversal are carried out, and global traversal inquiry is facilitated.

The foregoing description is only an overview of the technical solutions of the present application, and may be implemented according to the content of the specification in order to make the technical means of the present application more clearly understood, and in order to make the above-mentioned and other objects, features and advantages of the present application more clearly understood, the following detailed description of the present application will be given.

Drawings

For the purposes of making the objects, technical solutions and advantages of the embodiments of the present application more clear, the technical solutions of the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is apparent that the described embodiments are some embodiments of the present application, but not all embodiments. All other embodiments, which can be made by one of ordinary skill in the art based on the embodiments herein without making any inventive effort, are intended to be within the scope of the present application.

FIG. 1 is a schematic diagram of a data processing system of the type disclosed in an embodiment of the present application;

FIG. 2 is a schematic flow chart of a method for processing data according to an embodiment of the present application;

FIG. 3 is a diagram of data flow in the data processing system of the diagram disclosed in an embodiment of the present application;

FIG. 4 is a schematic diagram of a data processing apparatus according to an embodiment of the present disclosure;

FIG. 5 schematically shows a block diagram of an electronic device for performing a method according to the present application; and

FIG. 6 schematically shows a memory unit for holding or carrying program code implementing a method according to the present application _。

Detailed Description

The following description of the embodiments of the present application will be made clearly and fully with reference to the accompanying drawings, in which it is evident that the embodiments described are some, but not all, of the embodiments of the present application. All other embodiments, which can be made by one of ordinary skill in the art based on the embodiments herein without making any inventive effort, are intended to be within the scope of the present application.

A graph data processing system disclosed in an embodiment of the present application, as shown in fig. 1, the system includes: a data source management engine 110, a graph database processing engine 120, a graph data store back end 130, an index store back end 140, and a query application 150.

The following detailed description of the various components of the data processing system is provided as a further example.

The data source management engine 110 is configured to manage data source information of an offline data source and data source information of a real-time data source, which are input data sources, a graph data structure of a pre-configured graph instance, and processing configuration information associated with the graph data structure by the input data source when performing graph data processing.

In some embodiments of the present application, the data source management engine 110 is configured to manage information of data input/output targets of data processing of the whole graph, including, but not limited to, basic metadata such as data format, data type, schema (structure), data source connection address, driving class, constraint condition, table association relationship, field mapping relationship, and the like. The graph data source used as the input data source can be a relational database, a data file or other graph database. Metadata in the data source management engine is provided for the query and use of the graph database processing engine and is used for the operations of data source information acquisition, data constraint and mapping relation acquisition, data field standardization and the like. Optionally, the data source management engine 110 may include: and the offline data management module and the real-time data management module.

In some embodiments of the present application, the data source management engine 110 is further configured to: obtaining a graph data structure input by a configurator through a configuration interface of the graph data processing system, wherein the graph data structure comprises: nodes, edges, and attributes of the nodes and the edges; and constructing the graph instance according to the graph data structure. For example, when defining a graph model of a device fraud network graph, attributes such as a MAC address, an IP address, a GPS location, an IMEI number, an identification card number, and an enterprise-identical social credit code of the device may be defined as nodes, company-to-MAC addresses, company-to-IP addresses, company-to-IMEI numbers, company-to-GPS locations, person-to-IMEI numbers, person-to-MAC addresses, person-to-GPS locations, and the like may be defined as edges, and start points, end points, and directions of the edges may be defined, and graph data structures such as data types, attribute names, and the like of the attributes may be defined.

According to the defined graph data structure, the graph data processing system can create graph instances.

In some embodiments of the present application, the data source management engine 110 is further configured to: and storing the data source information of the input data source according to the input data source configuration operation of the target graph instance, which is executed by a configurator through a configuration interface of the graph data processing system. For example, a configurator may configure information such as a database type, a data source connection address, a table association relationship, etc. of an input data source of a certain graph instance at a configuration interface of the graph data processing system. Accordingly, the data source management engine 110 stores information about the input data source configured by the configurator after the configurator confirms the configuration.

Correspondingly, the data source management engine 110 is further configured to, in a processing information configuration stage, respond to a selection operation of a configurator on an input data source to determine the input data source corresponding to the target graph instance; storing field mapping configuration information according to the mapping relation between the data fields of the input data source configured by the configurator and the graph data structure of the target graph instance; and generating processing configuration information according to the input data source corresponding to the target graph instance and the storage field mapping configuration information.

For example, in the process information configuration stage, the data source management engine 110 may display an input data source selection interface, a configurator may select an input data source of a currently configured graph instance through the selection interface, and select a field name of the input data source through the configuration interface, map a field of the name into a corresponding graph data structure (i.e. Schema), thereby determining a mapping relationship of attributes of a field- > point or edge, and may configure whether the field needs type conversion, whether the field creates an index, whether ordering, and so on. The data source management engine 110 stores the mapping relation of the map data structure configured by the configurator as field mapping configuration information, and stores the field mapping configuration information as processing configuration information according to the input data source selected by the configurator.

Alternatively, the graph database processing engine 120 may read the data source information and the processing configuration information through the data source management engine 110. The graph database processing engine 120 then performs merging processing on the data read from the input data source, maps and converts the merged data into a data format adapted to the graph model, and loads and writes the data into a graph data storage backend (such as a graph database). The function and implementation of the graph database processing engine 120 is illustrated below.

The graph database processing engine 120 is configured to read the offline data source according to the data source information of the offline data source to obtain a first data stream, and read the real-time data source according to the data source information of the real-time data source to obtain a second data stream.

Optionally, the offline data source includes: the first database reads the offline data source according to the data source information of the offline data source to obtain a first data stream, and the first data stream comprises: writing the total data in the first database corresponding to the data source information of the offline data source into a first message queue through a change data acquisition tool to obtain a first data stream; or, reading the total data in the first database corresponding to the data source information of the offline data source through database connection, and writing the read total data into a first message queue to obtain a first data stream.

In this embodiment of the present application, the graph database processing engine 120 may establish a database connection with the offline data source according to the data source connection address of the offline data source described by the data source information, and then write the offline data in the offline data source into the message queue of the graph data processing system in full quantity through the CDC (Change Data Capture, change data acquisition) tool, and use the data in the message queue as the batch-written data stream. For example, the FLINK-CDC tool may be used to obtain the full offline data in the offline data source corresponding to the data source information, and write the obtained data into a designated message queue (referred to herein as a "first message queue").

Alternatively, for database types that do not support the Flink CDC tool, the database may be read by JDBC (Java Database Connect ivity, java database connection) and the read data written into the message queue (referred to herein as the "first message queue") by invoking the message queue application interface.

Optionally, the first message queue is a message queue corresponding to offline data, that is, a message queue corresponding to a batch data flow.

Optionally, the real-time data source includes: the second database and the message queue, the reading the real-time data source according to the data source information of the real-time data source, and obtaining a second data stream, including: acquiring the change data in the second database corresponding to the data source information of the real-time data source through a change data acquisition tool to obtain a second data stream; or, reading the real-time data in the message queue appointed by the data source information of the real-time data source to obtain a second data stream.

In some embodiments of the present application, there are a variety of sources of real-time data. For example, the graph database processing engine 120 may obtain real-time data written by an external system as a second data stream through a message queue (e.g., kafka) channel. For another example, some real-time data is not written in real-time to the message queue, but is dropped directly into the database, at which point the graph database processing engine 120 may capture the changed data in the database via the CDC data collection tool, resulting in a second data stream.

CDC (change data capture), i.e. change data capture, is one way of backing up a database, often used for backup of large amounts of data. The Mysql database log-based CDC is to open Mysql binary log. Whereas the acquisition of changes in the source database for the data source of the graph in real time may be accomplished through CDC techniques. Currently, various types of databases in the prior art have many well-established CDC technologies and corresponding components. In the embodiment of the application, a CDC data acquisition tool is adopted to acquire the data change of a Relational Database (RDBMS), the data change is transmitted into a real-time computing framework, the Flink real-time computing is used, and the data is mapped in real time.

The graph database processing engine 120 is further configured to perform a merging process on the first data stream and the second data stream, so as to obtain a merging result.

Optionally, the merging processing of the first data stream and the second data stream to obtain a merging processing result includes: and carrying out association combination on the first data stream and the second data stream according to the association relation between each field in the first data stream and the second data stream to obtain a combination processing result.

For example, the data stream merging operator calculates data from different data sources according to the mapping rule and the conversion rule, and converts and encapsulates field data in the source data according to the configured rule. Alternatively, the mapping rule and the conversion rule may be association manners specified between various data flows, for example, the mapping rule may be a field mapping relationship specified in field mapping configuration information, and the conversion rule may be field matching, that is, conversion.

Optionally, according to an association relationship between each field in the first data stream and the second data stream, performing association combination on the first data stream and the second data stream to obtain a combination processing result, where the combination processing result includes: determining a first field matched with the field mapping configuration information in the first data stream, and determining a second field matched with the field mapping configuration information in the second data stream; and combining the data corresponding to the first field in the first data stream with the data corresponding to the second field in the second data stream to obtain a combined result. For example: the fields A1, A2 of the a flow and the fields B1, B2 of the B flow are mapped to the same side in the field mapping configuration information, and then the a flow and the B flow can be combined and converted into a C flow with a specified format, wherein the C flow carries the field information of the a flow and the B flow.

Alternatively, the stream data having a field mapping relationship with the graph data structure in the first data stream and the second data stream may be combined.

Optionally, according to an association relationship between each field in the first data stream and the second data stream, performing association combination on the first data stream and the second data stream to obtain a combination processing result, where the combination processing result includes: and calling a data stream merging operator (such as a FlinkStream API) of an Flink (an open source stream processing framework) to merge the first data stream and the second data stream to obtain a real-time data stream as a merging processing result.

In one embodiment, the Join class operator of the FlinkStreamAPI may be employed to combine the first data stream and the second data stream.

In this embodiment of the present application, the graph database processing engine 120 is further configured to perform a disassembly process on the merging processing result according to the processing configuration information, so as to generate graph data matched with the graph instance.

As described above, the processing configuration information includes: the field mapping configuration information between the fields of the offline data source and the real-time data source and the points and edges of the graph instance, and optionally, the disassembling processing is performed on the merging processing result according to the processing configuration information, so as to generate graph data matched with the graph instance, including: resolving the merging processing result according to the field mapping configuration information, and resolving the merging processing result into data corresponding to points and edges respectively; and writing the data corresponding to the points and the edges into the graph instance to generate data corresponding to the points and the edges.

In some embodiments of the present application, the merging process results in real-time streaming data, including data of the fields of interest in the graph instance. In order to facilitate the subsequent retrieval and analysis of the graph data, the graph data stored in the graph data storage back end is data indexed based on points or edges, and therefore, the graph database processing engine is required to convert the merging processing result into a data format corresponding to the graph data storage back end.

Optionally, the graph database processing engine needs to perform data conversion according to a specified conversion rule according to a configured function. For example, according to the field mapping configuration information, extracting field information having mapping relation with points, edges and attributes of the graph instance, and outputting an attribute converted value of a single edge or a single node. When writing to the graph data store backend, the point or edge data would be encapsulated into point and edge objects. For example, the graph data store backend stores the data structures as follows: point V1 (attribute 1=value 1, attribute 2=value 2, attribute 3=value 3, …), point V2 (attribute 1=value 1, attribute 2=value 2, attribute 3=value 3, …), edge E1 (identification of start point V1, identification of end point V2, attribute 4=value 4, attribute 5=value 5, attribute 6=value 6, …), and the like. Wherein, both the point V1 and the V2 edge E1 store globally unique identifiers in the back end of the graph data storage. The attributes of the points and edges are predefined attributes for the graph instance. For example, in the device fraud graph model, when the edge E1 corresponds to a person and an IP address, the identification of the starting point V1 may be the identification card number of the person 1, the identification of the end point V2 is the IP address of the person 2, and the attribute may be: transaction time, transaction place and transaction amount.

In the embodiment of the present application, the graph data storage backend 130 is configured to store the graph data.

Alternatively, the graph data store backend 130 can be a database. The graph data store backend 130 is used to store data of the graph database, including points, edges, attributes of points and edges. Different graph databases provide different ways of graph data storage.

In some embodiments of the present application, the index storage backend 140 is configured to store an index corresponding to the graph instance.

Optionally, the index is generated according to the graph data structure, including: comprising the following steps: index of points, index of edges, index of point and edge attributes, and center point index of super points. The index storage backend functions to store an index, and the index data is independent of the graph data storage backend 130, and provides services for the graph data storage backend 130. The indexes stored in the index storage back end 140 are index data generated based on the data stored in the graph data storage back end 130 and the data structure and field mapping configuration information of the graph instance, and mainly serve to accelerate query and graph traversal.

In database query applications, the index directly determines that the query statement cannot efficiently perform the computed result, and if the index is not available, the backend will run the query statement with low efficiency, and even cannot traverse the result and is blocked until the timeout occurs. In this embodiment, the index storage backend 140 may be implemented by using a mainstream index backend supported by a graph database in the prior art, for example: elastic icSeach, apache Solr and Apache Lucene.

In this embodiment of the present application, the query application 150 is configured to query the graph data storage backend 130 based on a query request triggered by a user, so as to obtain a query result corresponding to the query request.

The query application 150 includes: the query front end and the query service end query the graph data storage back end based on the query request triggered by the user to obtain the query result corresponding to the query request, and the query result comprises: the query front end sends a query request to the query server based on query sentences edited by a user, wherein the query request carries the query sentences, and the query sentences are generated according to query grammar selected by the user and a to-be-queried graph example; the query server performs grammar conversion processing on the query statement carried in the query request according to the query grammar supported by the rear end of the graph data storage to obtain a target query statement supported by the rear end of the graph data storage; and the query server queries the rear end of the graph data storage based on the target query statement and acquires a query result output by the rear end of the graph data storage.

For example, the query server receives a graph query grammar, such as Greml in or Cypher grammar, from the query front end, converts the Greml in or Cypher grammar into a grammar supported by the graph data storage back end according to the realization of the graph data storage back end and the supported query grammar, and executes a corresponding graph query statement, or executes a graph traversal algorithm, and returns a query result to the query front end. Through grammar conversion processing, the method can be compatible with multi-query grammar and is convenient for users to query graph data.

In one embodiment of the application, a user can select a query grammar preferred by the user at a front-end page end of a query application or a query grammar supported by a rear-end of the graph data storage, select a graph instance to be queried, edit a query statement and transmit the query statement to a query server for processing through an http request. The query server judges whether grammar conversion is needed according to query grammar supported by the graph data storage rear end, if so, a preset grammar conversion code is called, and query sentences sent by the query front end are converted into sentences which can be executed at the graph data storage rear end. And then, carrying out instance initialization connection on the converted query statement according to the name of the graph instance (the database connection is needed to be acquired for the first time, and then the connection can be directly acquired from a cache), executing the query statement according to the query application interface provided by the rear end of the graph data storage, waiting for the query result to return, and feeding back the query result to the front end of the query.

Optionally, the query server may package the query result into a format required by the component when the front-end page renders the visual graphics, and then return to the query front-end.

The query front end renders the query results into a visual graph through a visual mode selected by a user or defaults to the system.

The graph data storage back end 130 is further configured to, in response to the query of the query application, retrieve the query result satisfying the query request from the locally stored graph data according to the index stored in the index storage back end.

For example, the graph data storage backend 130 responds to the query application interface being called to obtain the query parameters carried in the query application interface; then, according to the query parameters, index acceleration query is performed through the index storage back end 140, and index information is obtained; finally, retrieving the graph data stored in the graph data storage back end 130 according to the index information to obtain the query result satisfying the query request.

The embodiment of the application also discloses a graph data processing system, which is used for managing the data source information of an offline data source and the data source information of a real-time data source which are taken as input data sources when graph data processing is performed, a graph data structure of a pre-configured graph instance and processing configuration information related to the input data sources and the graph data structure by arranging a data source management engine; the setting diagram database processing engine is used for reading the offline data source according to the data source information of the offline data source to obtain a first data stream, and reading the real-time data source according to the data source information of the real-time data source to obtain a second data stream; combining the first data stream and the second data stream to obtain a combined processing result, and then disassembling the combined processing result according to the processing configuration information to generate graph data matched with the graph instance; setting a graph data storage back end for storing the graph data; setting an index storage back end for storing an index corresponding to the graph instance; when a user queries the graph data through the query application, the query result can be returned based on the offline data and the real-time data, so that the comprehensiveness and the completeness of the output graph data query result are improved. Because the offline data and the real-time data use the common storage position, complete data can be always obtained when graph analysis and traversal are carried out, and global traversal inquiry is facilitated.

In the prior art, the offline data processing architecture is: the method comprises the steps of uniformly extracting off-line data on the next day, uniformly processing new data on the next day in a batch mode, writing the new data into a database, and only searching yesterday data during inquiry and real-time data on the current day in a source database due to daily updating of the data, wherein the data are not processed into a graph database, and the off-line data and the real-time data are required to be stored respectively and cannot be uniformly mapped. The graph data processing system disclosed by the embodiment of the application combines the offline data and the real-time data into the data stream to be uniformly stored, so that the same graph entering is realized, and the integrity and the comprehensiveness of the data are improved.

On the other hand, the offline data and the real-time data are first combined through the graph database processing engine, so that the offline data and the real-time data are subjected to subsequent processing by adopting a set of code logic, and the maintenance cost of the graph data processing system can be reduced.

On the other hand, the embodiment of the application also discloses a graph data processing method, and the flow chart shown in fig. 3 and the flowchart shown in fig. 2 are described below to illustrate an embodiment of the graph data processing method. Wherein fig. 3 is a data flow diagram of the data processing system of fig. 1.

As shown in fig. 2, the graph data processing method disclosed in the embodiment of the present application includes: steps 210 to 260.

Step 210, according to the configuration operation of the configurator, obtaining the data source information of the offline data source and the data source information of the real-time data source, which are used as input data sources when the graph data processing is performed, the graph data structure of the graph instance, and the processing configuration information related to the input data source and the graph data structure.

As shown in fig. 3, the functional modules of offline data configuration management, real-time data configuration management, and graph instance configuration management built in the data source management engine 110 are respectively used for performing configuration management of offline data sources, configuration management of real-time data sources, definition of graph data structures, graph instance construction, and field mapping configuration.

For example, the data source management engine 110 of the graph data processing system may store data source information for input data sources according to input data source configuration operations for target graph instances performed by a configurator through a configuration interface of the graph data processing system.

For another example, in the process information configuration stage, the data source management engine 110 may determine an input data source corresponding to the target graph instance in response to a selection operation of the input data source by a configurator; storing field mapping configuration information according to the mapping relation between the data fields of the input data source configured by the configurator and the graph data structure of the target graph instance; and generating processing configuration information according to the input data source corresponding to the target graph instance and the storage field mapping configuration information.

According to the configuration operation of the configuration personnel, obtaining the data source information of the offline data source and the real-time data source, which are used as input data sources when the graph data processing is performed, and the graph data structure of the graph instance, and the specific implementation of the processing configuration information related to the input data source and the graph data structure, refer to the related description of the specific implementation of the data source management engine in the foregoing, and will not be repeated here.

Step 220, reading the offline data source according to the data source information of the offline data source to obtain a first data stream, and reading the real-time data source according to the data source information of the real-time data source to obtain a second data stream.

After the basic configuration is completed, the data source management engine 110 stores and manages the data source information, the created graph instances, and the processing configuration information such as the field mapping configuration information configured for each graph instance, and provides the processing configuration information to the graph database processing engine 120 and the index storage back end 140 for use. As shown in fig. 3, the data source management engine 110 includes a functional module, such as offline data reading conversion, real-time data reading conversion, and merging calculation, for performing operations, such as first data stream acquisition, second data stream acquisition, and data stream merging processing, respectively.

When the graph data needs to be generated, the graph database processing engine 120 reads the processing configuration information from the data source management engine 110, and connects the offline data source and the real-time data source according to the processing configuration information to perform data collection. The graph database processing engine 120 collects the offline data to obtain a first data stream, and collects the real-time data source to obtain a second data stream.

Optionally, the offline data source includes: the first database, the said data base is read according to the data source information of the said off-line data source, get the first data flow, including: writing the total data in the first database corresponding to the data source information of the offline data source into a first message queue through a change data acquisition tool to obtain a first data stream; or, reading the total data in the first database corresponding to the data source information of the offline data source through database connection, and writing the read total data into a first message queue to obtain a first data stream.

The graph database processing engine 120 reads the offline data source according to the data source information of the offline data source to obtain a first data stream, and reads the real-time data source according to the data source information of the real-time data source to obtain a specific embodiment of the second data stream, which is described in the foregoing related description of the specific embodiment of the graph database processing engine 120, and is not repeated herein.

And 230, performing association combination on the first data stream and the second data stream according to the association relation between the fields in the first data stream and the second data stream to obtain a combination processing result.

Next, the graph database processing engine 120 performs a merging process on the first data stream and the second data stream, so as to merge offline data and real-time data into one data stream, and perform a unified process.

Optionally, according to an association relationship between each field in the first data stream and the second data stream, performing association combination on the first data stream and the second data stream to obtain a combination processing result, where the combination processing result includes: determining a first field matched with the field mapping configuration information in the first data stream, and determining a second field matched with the field mapping configuration information in the second data stream; and combining the data corresponding to the first field in the first data stream with the data corresponding to the second field in the second data stream to obtain a combined result.

The association and combination are performed on the first data stream and the second data stream according to the association relationship between the fields in the first data stream and the second data stream, so as to obtain a specific embodiment of the combination processing result, which is referred to the description of the specific embodiment of the graph database processing engine 120, and is not repeated herein.

And 240, carrying out disassembly processing on the merging processing result according to the processing configuration information to generate graph data matched with the graph instance.

The graph database processing engine 120 then performs formatting processing on the data stream obtained by combining the offline data and the real-time data, and converts the data stream into a data format required by the graph data storage back end 130, i.e., a graph data format.

Optionally, the processing configuration information includes: the field mapping configuration information between the fields of the offline data source and the real-time data source and the points and edges of the graph instance, the merging processing result is disassembled according to the processing configuration information, and graph data matched with the graph instance is generated, including: resolving the merging processing result according to the field mapping configuration information, and resolving the merging processing result into data corresponding to points and edges respectively; and writing the data corresponding to the points and the edges into the graph instance to generate data corresponding to the points and the edges.

And performing a disassembly process on the merging processing result according to the processing configuration information to generate a specific embodiment of the graph data matched with the graph instance, which is described in the foregoing related description of the specific embodiment of the graph database processing engine 120, and is not repeated herein.

Step 250, storing the graph data and the index corresponding to the graph instance.

The graph database processing engine 120 sends the converted graph data to the graph data storage backend 130 for storage.

The method for generating and storing the index corresponding to the drawing example is referred to the description of the specific embodiment of the index storage backend 140 hereinabove, and will not be repeated here.

To this end, the graph data storage backend 130 stores data of each point and data of each edge of the graph instance, and the index storage backend 140 stores indexes of the points and edges of the graph instance.

Step 260, responding to a user-triggered query request, and querying the stored graph data based on the index to obtain a query result corresponding to the query request.

When a user inputs a query statement through the query application 150 of the graph data processing system to query information in a target graph instance, the query application 150 queries stored graph data based on the index to obtain a query result corresponding to the query request.

Optionally, the query application includes: the query front end and the query service end respond to the user to trigger a query request, query the stored graph data based on the index to obtain a query result corresponding to the query request, and the query result comprises: the query front end sends a query request to the query server based on query sentences edited by a user, wherein the query request carries the query sentences, and the query sentences are generated according to query grammar selected by the user and a to-be-queried graph example; the query server performs grammar conversion processing on the query statement carried in the query request according to the query grammar supported by the rear end of the graph data storage to obtain a target query statement supported by the rear end of the graph data storage; and the query server queries the rear end of the graph data storage based on the target query statement and acquires a query result output by the rear end of the graph data storage.

The query application 150 queries the stored graph data based on the index in response to the user triggering the query request to obtain the specific embodiment of the query result corresponding to the query request, which is described in the foregoing related description of the specific embodiment of the query application 150 and will not be repeated herein.

According to the graph data processing method disclosed by the embodiment of the application, the data source information of the offline data source and the real-time data source which are used as input data sources when graph data processing is performed and the graph data structure of the graph instance are obtained through configuration operation of configuration personnel, and the input data sources and the processing configuration information related to the graph data structure are processed; reading the offline data source according to the data source information of the offline data source to obtain a first data stream, and reading the real-time data source according to the data source information to obtain a second data stream; carrying out association combination on the first data stream and the second data stream according to the association relation between each field in the first data stream and the second data stream to obtain a combination processing result; performing disassembly processing on the merging processing result according to the processing configuration information to generate graph data matched with the graph instance; then, storing the graph data and indexes corresponding to the graph examples; when a user triggers a query request, acquiring a query result corresponding to the query request based on the map data which is stored by the index query and contains offline data and real-time data, so that the comprehensiveness and the completeness of the output map data query result are improved.

Because the offline data and the real-time data use the common storage position, complete data can be always obtained when graph analysis and traversal are carried out, and global traversal inquiry is facilitated.

In the prior art, the offline data processing architecture is: the method comprises the steps of uniformly extracting off-line data on the next day, uniformly processing new data on the next day in a batch mode, writing the new data into a database, and only searching yesterday data during inquiry and real-time data on the current day in a source database due to daily updating of the data, wherein the data are not processed into a graph database, and the off-line data and the real-time data are required to be stored respectively and cannot be uniformly mapped. The graph data processing method disclosed by the embodiment of the application combines the offline data and the real-time data into the data stream to be uniformly stored, so that the same graph entering is realized, and the integrity and the comprehensiveness of the data are improved.

Correspondingly, the embodiment of the application also discloses a graph data processing device, as shown in fig. 4, which comprises:

A configuration management module 410, configured to obtain, according to a configuration operation of a configurator, data source information of an offline data source and data source information of a real-time data source, which are input data sources when performing graph data processing, a graph data structure of a graph instance, and processing configuration information associated with the graph data structure by the input data source;

the data stream acquisition module 420 is configured to read the offline data source according to the data source information of the offline data source to obtain a first data stream, and read the real-time data source according to the data source information of the real-time data source to obtain a second data stream;

the data stream merging module 430 is configured to perform association merging on the first data stream and the second data stream according to association relationships between fields in the first data stream and the second data stream, so as to obtain a merging processing result;

a graph data generating module 440, configured to perform a disassembly process on the merging processing result according to the processing configuration information, to generate graph data matched with the graph instance;

a graph data and index storage module 450, configured to store the graph data and an index corresponding to the graph instance;

And a query output module 460, configured to query the stored graph data based on the index in response to a user triggering a query request, so as to obtain a query result corresponding to the query request.

Optionally, the data stream merging module 430 is further configured to:

and carrying out association combination on the first data stream and the second data stream according to the association relation between each field in the first data stream and the second data stream to obtain a combination processing result.

Optionally, the processing configuration information includes: the field mapping configuration information between the fields of the offline data source and the real-time data source and the points and edges of the graph instance, the graph data generation module further configured to:

resolving the merging processing result according to the field mapping configuration information, and resolving the merging processing result into data corresponding to points and edges respectively;

and writing the data corresponding to the points and the edges into the graph instance to generate data corresponding to the points and the edges.

Optionally, the offline data source includes: the first database reads the offline data source according to the data source information of the offline data source to obtain a first data stream, and the first data stream comprises:

Writing the total data in the first database corresponding to the data source information of the offline data source into a first message queue through a change data acquisition tool to obtain a first data stream; or alternatively, the process may be performed,

and reading the total data in the first database corresponding to the data source information of the offline data source through database connection, and writing the read total data into a first message queue to obtain a first data stream.

Optionally, the real-time data source includes: the second database and the message queue, the reading the real-time data source according to the data source information of the real-time data source, and obtaining a second data stream, including:

acquiring the change data in the second database corresponding to the data source information of the real-time data source through a change data acquisition tool to obtain a second data stream; or alternatively, the process may be performed,

and reading the real-time data in the message queue appointed by the data source information of the real-time data source to obtain a second data stream.

Optionally, the query application includes: query front end and query service end, the query output module 460 is further configured to:

the query front end sends a query request to the query server based on query sentences edited by a user, wherein the query request carries the query sentences, and the query sentences are generated according to query grammar selected by the user and a to-be-queried graph example;

The query server performs grammar conversion processing on the query statement carried in the query request according to the query grammar supported by the rear end of the graph data storage to obtain a target query statement supported by the rear end of the graph data storage;

and the query server queries the rear end of the graph data storage based on the target query statement and acquires a query result output by the rear end of the graph data storage.

The embodiment of the module of the device is not repeated, and reference may be made to the specific implementation of the corresponding steps of the method embodiment.

According to the image data processing device disclosed by the embodiment of the application, data source information of an offline data source serving as an input data source and data source information of a real-time data source and an image data structure of an image instance when image data processing is performed are obtained through configuration operation of configuration personnel, and processing configuration information related to the input data source and the image data structure; reading the offline data source according to the data source information of the offline data source to obtain a first data stream, and reading the real-time data source according to the data source information of the real-time data source to obtain a second data stream; carrying out association combination on the first data stream and the second data stream according to the association relation between each field in the first data stream and the second data stream to obtain a combination processing result; performing disassembly processing on the merging processing result according to the processing configuration information to generate graph data matched with the graph instance; then, storing the graph data and indexes corresponding to the graph examples; when a user triggers a query request, acquiring a query result corresponding to the query request based on the map data which is stored by the index query and contains offline data and real-time data, so that the comprehensiveness and the completeness of the output map data query result are improved.

In this specification, each embodiment is described in a progressive manner, and each embodiment is mainly described by differences from other embodiments, and identical and similar parts between the embodiments are all enough to be referred to each other. For the device embodiments, since they are substantially similar to the method embodiments, the description is relatively simple, and reference is made to the description of the method embodiments for relevant points.

The foregoing has described in detail a method and apparatus for processing image data provided by the present application, and specific examples have been applied herein to illustrate the principles and embodiments of the present application, where the foregoing examples are provided to assist in understanding the method and core idea of the present application; meanwhile, as those skilled in the art will have modifications in the specific embodiments and application scope in accordance with the ideas of the present application, the present description should not be construed as limiting the present application in view of the above.

The apparatus embodiments described above are merely illustrative, wherein the elements illustrated as separate elements may or may not be physically separate, and the elements shown as elements may or may not be physical elements, may be located in one place, or may be distributed over a plurality of network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art will understand and implement the present invention without undue burden.

Various component embodiments of the present application may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. Those skilled in the art will appreciate that some or all of the functions of some or all of the components in an electronic device according to embodiments of the present application may be implemented in practice using a microprocessor or Digital Signal Processor (DSP). The present application may also be embodied as an apparatus or device program (e.g., computer program and computer program product) for performing a portion or all of the methods described herein. Such a program embodying the present application may be stored on a computer readable medium, or may have the form of one or more signals. Such signals may be downloaded from an internet website, provided on a carrier signal, or provided in any other form.

For example, fig. 5 shows an electronic device in which a method according to the present application may be implemented. The electronic device may be a PC, a mobile terminal, a personal digital assistant, a tablet computer, etc. The electronic device conventionally comprises a processor 510 and a memory 520 and a program code 530 stored on said memory 520 and executable on the processor 510, said processor 510 implementing the method described in the above embodiments when said program code 530 is executed. The memory 520 may be a computer program product or a computer readable medium. The memory 520 may be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read only memory), an EPROM, a hard disk, or a ROM. The memory 520 has a storage space 5201 for program code 530 of a computer program for performing any of the method steps described above. For example, the memory space 5201 for the program code 530 may include individual computer programs for implementing the various steps in the above method, respectively. The program code 530 is computer readable code. These computer programs may be read from or written to one or more computer program products. These computer program products comprise a program code carrier such as a hard disk, a Compact Disc (CD), a memory card or a floppy disk. The computer program comprises computer readable code which, when run on an electronic device, causes the electronic device to perform a method according to the above-described embodiments.

The embodiment of the application also discloses a computer readable storage medium, on which a computer program is stored, which when being executed by a processor, implements the steps of the graph data processing method according to the embodiment of the application.

Such a computer program product may be a computer readable storage medium, which may have memory segments, memory spaces, etc. arranged similarly to the memory 520 in the electronic device shown in fig. 5. The program code may be stored in the computer readable storage medium, for example, in a suitable form. The computer readable storage medium is typically a portable or fixed storage unit as described with reference to fig. 6. In general, the memory unit comprises computer readable code 530', which computer readable code 530' is code that is read by a processor, which code, when executed by the processor, implements the steps of the method described above.

Reference herein to "one embodiment," "an embodiment," or "one or more embodiments" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present application. Furthermore, it is noted that the word examples "in one embodiment" herein do not necessarily all refer to the same embodiment.

In the description provided herein, numerous specific details are set forth. However, it is understood that embodiments of the present application may be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description.

In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps not listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The application may be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by one and the same item of hardware. The use of the words first, second, third, etc. do not denote any order. These words may be interpreted as names.

Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present application, and are not limiting thereof; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the corresponding technical solutions.

Claims

1. A graph data processing system, the system comprising: a data source management engine, a graph database processing engine, a graph data store backend, an index store backend, and a query application, wherein,

The graph data storage back end is used for storing the graph data;

2. The system of claim 1, wherein the merging the first data stream and the second data stream to obtain a merged result comprises:

3. The system of claim 1, wherein the processing configuration information comprises: the field mapping configuration information between the fields of the offline data source and the real-time data source and the points and edges of the graph instance, the merging processing result is disassembled according to the processing configuration information, and graph data matched with the graph instance is generated, including:

4. The system of claim 1, wherein the offline data source comprises: the first database reads the offline data source according to the data source information of the offline data source to obtain a first data stream, and the first data stream comprises:

reading the total data in the first database corresponding to the data source information of the offline data source through database connection, and writing the read full data into a first message queue to obtain a first data stream.

5. The system of claim 1, wherein the real-time data source comprises: the second database and the message queue, the reading the real-time data source according to the data source information of the real-time data source, and obtaining a second data stream, including:

6. The system of claim 1, wherein the query application comprises: the query front end and the query service end query the graph data storage back end based on the query request triggered by the user to obtain the query result corresponding to the query request, and the query result comprises:

7. A graph data processing method, the method comprising:

storing the graph data and indexes corresponding to the graph examples;

8. A graph data processing apparatus, the apparatus comprising:

9. An electronic device comprising a memory, a processor and program code stored on the memory and executable on the processor, wherein the processor implements the graph data processing method of claim 7 when executing the program code.

10. A computer readable storage medium having stored thereon program code, which when executed by a processor realizes the steps of the graph data processing method of claim 7.