CN113032419B - Multi-source data aggregation search method, device, equipment and storage medium - Google Patents
Multi-source data aggregation search method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN113032419B CN113032419B CN202110429692.1A CN202110429692A CN113032419B CN 113032419 B CN113032419 B CN 113032419B CN 202110429692 A CN202110429692 A CN 202110429692A CN 113032419 B CN113032419 B CN 113032419B
- Authority
- CN
- China
- Prior art keywords
- data
- aggregation
- target
- search
- service
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 230000002776 aggregation Effects 0.000 title claims abstract description 156
- 238000004220 aggregation Methods 0.000 title claims abstract description 156
- 238000000034 method Methods 0.000 title claims abstract description 42
- 238000012544 monitoring process Methods 0.000 claims abstract description 24
- 238000012423 maintenance Methods 0.000 claims abstract description 21
- 230000008859 change Effects 0.000 claims description 20
- 238000004590 computer program Methods 0.000 claims description 15
- 238000010276 construction Methods 0.000 claims description 3
- 238000004891 communication Methods 0.000 description 9
- 230000008569 process Effects 0.000 description 7
- 230000005540 biological transmission Effects 0.000 description 5
- 238000010586 diagram Methods 0.000 description 5
- 230000009471 action Effects 0.000 description 4
- 230000004931 aggregating effect Effects 0.000 description 4
- 238000013500 data storage Methods 0.000 description 4
- 230000001360 synchronised effect Effects 0.000 description 3
- 206010033799 Paralysis Diseases 0.000 description 1
- 230000006978 adaptation Effects 0.000 description 1
- 238000007792 addition Methods 0.000 description 1
- 230000006399 behavior Effects 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000005111 flow chemistry technique Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 230000001052 transient effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/242—Query formulation
- G06F16/2433—Query languages
- G06F16/244—Grouping and aggregation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24553—Query execution of query operations
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/547—Messaging middleware
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/548—Queue
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
- Mathematical Physics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application discloses a multi-source data aggregation searching method, device, equipment and storage medium. The method comprises the following steps: constructing an aggregation search server based on GraphQL, and configuring data aggregation formats corresponding to various business microservices according to the target application type; monitoring a target service database to obtain corresponding changed data, calling corresponding service microservices by using the aggregation search server according to the data types of the changed data to obtain aggregated data corresponding to the changed data, and storing the aggregated data to an ES database; and receiving a data search request customized and sent by a front end by using the aggregation search server, and inquiring from the ES database according to the data search request to obtain corresponding target aggregation data and feeding the target aggregation data back to the front end. The method can automatically realize the maintenance of the aggregated data, reduce the data volume of the aggregated data by utilizing the GraphQL characteristic and avoid the data redundancy problem in the feedback data.
Description
Technical Field
The invention relates to the technical field of computers, in particular to a multi-source data aggregation searching method, device, equipment and storage medium.
Background
At present, multiple fields have strong requirements for aggregated search of multiple service source data, and in the prior art, data of multiple source libraries are synchronized into the same database instance, then a structured query language is used for performing aggregated query in a JOIN form on multiple library tables, and different API interfaces are developed for different service requirements to provide data services. However, since the multi-source database data synchronization is generally synchronized to the designated aggregated database instance in the form of a backup database, there are many data in tables and table fields that are not needed in the search scenario, which results in large aggregated data amount, slow synchronization, and low performance. Moreover, when data is searched by using the traditional structured query language, the performance problem caused by some range and fuzzy search conditions is difficult to solve, and the database instance is paralyzed in serious cases. In addition, the same kind of data often needs different aggregated search results in different service scenes, the traditional method provides a general interface to a certain extent, but needs much redundant data, the data is often compatible with all scenes, which causes pressure on network communication, and users cannot accept the data when the network speed is slow; when the service requirements of different types of data need to be searched, the front end often needs to call a plurality of interfaces to acquire different data, which may cause a plurality of network requests and also cause pressure on network communication. The efficiency of multi-source data aggregation and search is reduced.
Disclosure of Invention
In view of the above, an object of the present invention is to provide a method, an apparatus, a device and a medium for searching and aggregating multi-source data, which can improve the efficiency of searching and aggregating multi-source data. The specific scheme is as follows:
in a first aspect, the application discloses a multi-source data aggregation search method, which includes:
constructing an aggregation search server based on GraphQL, and configuring data aggregation formats corresponding to various business microservices according to the target application type;
monitoring a target service database to obtain corresponding changed data, calling corresponding service micro-services by using the aggregation search server according to the data types of the changed data to obtain aggregated data corresponding to the changed data, and storing the aggregated data to an ES database;
and receiving a data search request customized and sent by a front end by using the aggregation search server, and inquiring from the ES database according to the data search request to obtain corresponding target aggregation data and feeding the target aggregation data back to the front end.
Optionally, the invoking, by using the aggregation search server, the corresponding service microservice according to the data type of the changed data includes:
generating a Kafka message queue of index data to be maintained based on the monitored change data;
reading target key fields of index data to be maintained in the Kafka message queue according to a queue sequence by a pre-established Kafka consumer;
and generating a service calling request based on the target key field and sending the service calling request to the aggregation search server, so that the aggregation search server identifies the data type of the target key field in the service calling request after receiving the service calling request, and calls the corresponding service micro service according to the data type.
Optionally, the receiving, by the aggregation search server, a data search request customized and sent by a front end, and obtaining, by querying from the ES database according to the data search request, corresponding target aggregation data to feed back to the front end includes:
receiving a search service calling request sent by a gateway route after acquiring the data search request sent by the front end by using the aggregation search server; the data search request is an aggregated data search request which is customized by the front end based on the GraphQL according to the target demand data type;
and querying from the ES database according to the target demand data type to obtain corresponding target aggregation data, and feeding back the target aggregation data to the front end.
Optionally, the monitoring the target service database to obtain corresponding changed data includes:
and monitoring the binlog log of the target service database through a Canal server to obtain the change data of the target service database through monitoring.
In a second aspect, the present application discloses a multi-source data aggregation search apparatus, comprising:
the aggregation search server construction module is used for constructing an aggregation search server based on GraphQL and configuring data aggregation formats corresponding to various business microservices according to the target application types;
the index data maintenance module is used for monitoring a target business database to obtain corresponding changed data, calling corresponding business microservices by using the aggregation search server according to the data types of the changed data to obtain aggregated data corresponding to the changed data, and storing the aggregated data to an ES database;
and the aggregated data searching module is used for receiving a data searching request customized and sent by a front end by using the aggregated searching server, and inquiring from the ES database according to the data searching request to obtain corresponding target aggregated data and feeding the target aggregated data back to the front end.
Optionally, the index data maintenance module includes:
the message queue generating unit is used for generating a Kafka message queue of index data to be maintained based on the monitored change data;
a key field reading unit, configured to read, by a pre-created Kafka consumer, a target key field of index data to be maintained in the Kafka message queue according to a queue order;
and the service calling unit is used for generating a service calling request based on the target key field and sending the service calling request to the aggregation search server, so that the aggregation search server can identify the data type of the target key field in the service calling request after receiving the service calling request, and call the corresponding service microservice according to the data type.
Optionally, the aggregated data searching module includes:
a calling request receiving unit, configured to receive, by using the aggregation search server, a search service calling request sent by a gateway route after acquiring the data search request sent by the front end; the data search request is an aggregated data search request which is customized by the front end based on the GraphQL according to the target demand data type;
and the data query unit is used for querying the corresponding target aggregation data from the ES database according to the target demand data type and feeding the target aggregation data back to the front end.
Optionally, the index data maintenance module includes:
and the data monitoring unit is used for monitoring the binlog log of the target business database through a Canal server so as to monitor and obtain the change data of the target business database.
In a third aspect, the present application discloses an electronic device, comprising:
a memory for storing a computer program;
and the processor is used for executing the computer program to realize the multi-source data aggregation searching method.
In a fourth aspect, the present application discloses a computer readable storage medium for storing a computer program; wherein the computer program when executed by the processor implements the aforementioned multi-source data aggregation search method.
In the method, an aggregation search server based on GraphQL is constructed, and data aggregation formats corresponding to various business microservices are configured according to target application types; monitoring a target service database to obtain corresponding changed data, calling corresponding service microservices by using the aggregation search server according to the data types of the changed data to obtain aggregated data corresponding to the changed data, and storing the aggregated data to an ES database; and receiving a data search request customized and sent by a front end by using the aggregation search server, and inquiring from the ES database according to the data search request to obtain corresponding target aggregation data and feeding the target aggregation data back to the front end.
Therefore, the aggregation search server based on GraphQL is constructed, the data aggregation formats corresponding to various business microservices are configured according to the target application types, and automatic aggregated data maintenance is realized by monitoring the change of the database. Therefore, aggregated index data is obtained by combining a GraphQL configuration field with service micro-service interface query, aggregation is only carried out on a relevant service table, useless data are not stored, the data amount after aggregation is greatly reduced, and the aggregated data storage capacity is further reduced; meanwhile, by utilizing the GraphQL characteristics, when different service scenes of the same type of data are used for tuning and aggregating the search server, the server only returns corresponding data through specifying required fields, the data redundancy problem does not exist, the network transmission data volume is reduced, the request speed is further improved, in addition, the front end can simultaneously call a plurality of GraphQL interfaces through one network request, different data are simultaneously obtained, and the aggregation search capability of multi-source data is improved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the embodiments or the prior art descriptions will be briefly described below, it is obvious that the drawings in the following description are only embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the provided drawings without creative efforts.
FIG. 1 is a flow chart of a multi-source data aggregation search method provided by the present application;
FIG. 2 is a flowchart of a specific multi-source data aggregation search method provided in the present application;
FIG. 3 is a schematic structural diagram of a specific multi-source data aggregation search system provided in the present application;
fig. 4 is a flowchart illustrating a specific graph ql order aggregation search service operation provided in the present application;
FIG. 5 is a flowchart illustrating a specific automatic maintenance process for index data provided herein;
FIG. 6 is a flow chart of a specific free search of aggregated data provided herein;
FIG. 7 is a schematic structural diagram of a multi-source data aggregation search apparatus provided in the present application;
fig. 8 is a block diagram of an electronic device provided in the present application.
Detailed Description
In the prior art, data of a plurality of source libraries are synchronized to the same database instance, then a structured query language is used for performing JOIN-form aggregate query on a plurality of library tables, and different API interfaces are developed for different service requirements to provide data services. The problems of useless data synchronization, redundant data transmission and the like exist, and the efficiency of multi-source data aggregation searching is reduced. In order to overcome the technical problem, the application provides a multi-source data aggregation search method based on GraphQL, which can improve the aggregation search capability of a multi-source database.
The embodiment of the application discloses a multi-source data aggregation searching method, and as shown in fig. 1, the method can comprise the following steps:
step S11: and constructing an aggregation search server based on GraphQL, and configuring data aggregation formats corresponding to various business microservices according to the target application type.
In the embodiment, firstly, an aggregation search server based on Graph Query Language (Graph ql) is constructed, and data aggregation formats corresponding to various business microservices are configured according to a target application type; the graph ql is a query language for API (Application Programming Interface), and can also provide a set of easily understood complete description for data in API when data query runs, so that the client can accurately obtain the data it needs, and there is no redundant data. Specifically, the schema file of the GraphQL is used to configure information fields that need to be aggregated for various services, that is, a customized data interface is provided through schema file configuration, so that when a service microservice is called to perform data aggregation, corresponding aggregated data can be obtained by aggregation according to a corresponding data aggregation format. For example, when the target application type is order aggregation search, a corresponding data aggregation format is configured for corresponding order business microservices, fulfillment business microservices and the like.
Step S12: monitoring a target service database to obtain corresponding changed data, calling corresponding service micro-services by using the aggregation search server according to the data types of the changed data to obtain aggregated data corresponding to the changed data, and storing the aggregated data to an ES database.
In this embodiment, the target service database is monitored in real time to obtain corresponding changed data, the aggregation search server is used to invoke corresponding service microservices according to the data types of the changed data to obtain aggregated data corresponding to the changed data, and the aggregated data is stored in the ES database. It is understood that each time a change of the database is monitored, the aggregation search server is notified to invoke a corresponding business microservice to obtain aggregated data about the changed data and conforming to a corresponding data aggregation format, and then the aggregated data is stored in an es (elastic search) database.
Step S13: and receiving a data search request customized and sent by a front end by using the aggregation search server, and inquiring from the ES database according to the data search request to obtain corresponding target aggregation data and feeding the target aggregation data back to the front end.
In this embodiment, after receiving a data search request customized and sent by a front end, the aggregation search server queries from the ES database according to the data search request to obtain corresponding target aggregation data, and feeds the target aggregation data back to the front end, that is, determines a search condition according to the data search request to directly search the ES database and obtain result data of aggregation search, and feeds the result data back to the front end, so that search for ES usually takes less than 500 milliseconds, and the efficiency of data search is improved. It can be understood that the aggregation search server can maintain aggregate data and search aggregate data, and when maintaining aggregate data, calls the micro service of the corresponding service subsystem to obtain the data after relevant aggregation; and when the data search request is aggregated, directly searching the corresponding result from the ES and returning the result to the front end. Therefore, the ES index is automatically maintained after the data change is monitored, and various customized search conditions at the front end are met by utilizing the GraphQL characteristic.
In the using process, the aggregated specific data can be changed through flexible use configuration without additionally developing a data interface; and moreover, the data can be freely customized by self-matching with a rear-end interface according to the data requirements of the front-end multi-service scene without carrying out various adaptations by the rear end, so that the development cost can be comprehensively reduced.
As can be seen from the above, in this embodiment, an aggregation search server based on GraphQL is constructed, data aggregation formats corresponding to various business microservices are configured according to a target application type, and automatic aggregated data maintenance is implemented by monitoring changes of a database. Therefore, aggregated index data is obtained by combining a GraphQL configuration field with service micro-service interface query, aggregation is performed only on a relevant service table, useless data are not stored, the data size after aggregation is greatly reduced, and the aggregated data storage capacity is further reduced; meanwhile, by utilizing the GraphQL characteristic, when different service scenes of the same type of data are aggregated and adjusted to the search server, the server only returns corresponding data through specifying required fields, the problem of data redundancy does not exist, the data volume of network transmission is reduced, the request speed is further improved, in addition, the front end can simultaneously call a plurality of GraphQL interfaces through one network request, different data are simultaneously obtained, the network request times are further reduced, the network communication pressure is further reduced, and the relatively poor network environment can be compatible. Meanwhile, the aggregated data is directly stored in the ES index, the ES can provide a powerful real-time searching function, the searching result can be directly returned to the front end, and the response speed of the service interface of the front end is greatly improved.
The embodiment of the application discloses a specific multi-source data aggregation searching method, and as shown in fig. 2, the method may include the following steps:
step S21: and constructing an aggregation search server based on GraphQL, and configuring data aggregation formats corresponding to various business microservices according to the target application type.
In this embodiment, in combination with the description of the above embodiments, taking an order application scenario as an example, the e-commerce background order search needs to search data according to the double dimensions of order placing time of the order system and fulfillment time of the fulfillment system. Firstly, a GraphQL order aggregation search server of order aggregation information is constructed, an order is taken as a main dimension, and aggregation information fields required by a service are configured by using schema of the GraphQL. For example, fig. 3 is a schematic structural diagram of a multi-source data aggregation search system, which includes a GraphQL order aggregation search service, automatic maintenance of index data, and automatic search of aggregation data. The graph ql order aggregation search service is shown in fig. 4, the graph ql order aggregation search server supports index data maintenance and aggregation data search, the index data maintenance supports invocation of order microservice and fulfillment microservice, and the aggregation data search supports search of the ES database.
Step S22: and monitoring the binlog log of the target service database through a Canal server to obtain the change data of the target service database through monitoring.
In this embodiment, as shown in fig. 5, a binary log of the target service database is monitored by a Canal server to obtain changed data of the target service database by monitoring. The target business database in the order field may include, but is not limited to, an order business database and a fulfillment business database. Namely, a Canal server is set up to monitor a binlog log of a database, and corresponding change data is obtained in time when data addition, deletion or modification behaviors exist in the database, for example, a document ID is monitored to be newly added in the database.
Step S23: and generating a Kafka message queue of index data to be maintained based on the monitored change data.
In this embodiment, a Kafka message queue of index data to be maintained is generated based on the monitored change data. E.g., sending the intercepted document data to a Kafka message queue, ready for automatic index maintenance. Kafka is an open source flow processing platform, is a high-throughput distributed publish-subscribe message system, and can process all action flow data of a consumer in a website
Step S24: reading target key fields of index data to be maintained in the Kafka message queue according to the queue order through a pre-created Kafka consumer.
In this embodiment, target key fields of index data to be maintained in the Kafka message queue are read according to the queue order by a pre-created Kafka consumer. It can be understood that, in order to determine the business microservice that needs to be invoked, a corresponding target key field, such as an order ID or a fulfillment form ID, needs to be transmitted to the GraphQL order aggregation search server.
Step S25: and generating a service calling request based on the target key field and sending the service calling request to the aggregation search server, so that the aggregation search server identifies the data type of the target key field in the service calling request after receiving the service calling request, calls the corresponding business microservice according to the data type to obtain aggregated data corresponding to the changed data, and stores the aggregated data to an ES database.
In this embodiment, a service invocation request is generated based on the target key field and sent to the graph ql order aggregation search server, so that the graph ql order aggregation search server invokes a corresponding business microservice according to the data type of the target key field after receiving the service invocation request, so as to obtain aggregated data corresponding to the changed data, and stores the aggregated data in an ES database. And if a service calling request containing the order ID is received, calling the order micro-service. And calling a fulfillment list micro-service when receiving the fulfillment list ID so as to acquire single aggregated data of the current order and return the aggregated data to the index data automatic maintenance module, and storing the aggregated data to the ES database by the index data automatic maintenance module. The data storage structure of the ES database may be a GraphQL order aggregation information model in fig. 4, that is, order information is used as a main component, and a plurality of fulfillment documents are stored in a partitioned manner as a subset.
Step S26: and receiving a data search request customized and sent by a front end by using the aggregation search server, and inquiring from the ES database according to the data search request to obtain corresponding target aggregation data and feeding the target aggregation data back to the front end.
In this embodiment, the receiving, by the aggregation search server, a data search request customized and sent by a front end, and obtaining, by querying from the ES database according to the data search request, corresponding target aggregated data to feed back to the front end may include: receiving a search service calling request sent by a gateway route after acquiring the data search request sent by the front end by using the aggregation search server; the data search request is an aggregated data search request which is customized by the front end based on the GraphQL according to the target demand data type; and inquiring from the ES database according to the target demand data type to obtain corresponding target aggregated data, and feeding back to the front end. For example, as shown in fig. 6, that is, the front end may call the gateway in a GraphQL manner to request corresponding aggregated data, the gateway configuration will directly point to the GraphQL order aggregated search service, and specify a source for aggregated data search, so that the GraphQL order aggregated search service directly searches the ES according to the specified search condition and obtains corresponding target aggregated data, and then transmit the search result to the gateway and return the search result to the front end. Namely, the front end sends a customized aggregated data search request in a GraphQL mode, so that various types of data can be acquired without multiple requests.
For the specific process of the step S21, reference may be made to the corresponding contents disclosed in the foregoing embodiments, and details are not repeated here.
As can be seen from the above, in this embodiment, the Canal log of the target service database is monitored by the Canal server to monitor change data of the target service database, and a Kafka message queue of index data to be maintained is generated based on the monitored change data, so that the aggregation search server calls a corresponding service microservice according to the Kafka message queue to obtain aggregated data, and stores the aggregated data in the ES database, thereby implementing automatic maintenance of the index data and improving efficiency of multisource data aggregation search.
Correspondingly, the embodiment of the present application further discloses a multi-source data aggregation search apparatus, as shown in fig. 7, the apparatus includes:
the aggregation search server construction module 11 is configured to construct an aggregation search server based on GraphQL, and configure data aggregation formats corresponding to various types of business microservices according to a target application type;
the index data maintenance module 12 is configured to monitor a target service database to obtain corresponding changed data, call a corresponding service microservice according to a data type of the changed data by using the aggregation search server to obtain aggregated data corresponding to the changed data, and store the aggregated data in an ES database;
and the aggregated data searching module 13 is configured to receive a data searching request customized and sent by a front end by using the aggregated search server, and query from the ES database according to the data searching request to obtain corresponding target aggregated data and feed the target aggregated data back to the front end.
As can be seen from the above, in this embodiment, an aggregated search server based on GraphQL is constructed, data aggregation formats corresponding to various types of business microservices are configured according to a target application type, and automatic aggregated data maintenance is implemented by monitoring changes in a database. Therefore, aggregated index data is obtained by combining a GraphQL configuration field with service micro-service interface query, aggregation is only carried out on a relevant service table, useless data are not stored, the data amount after aggregation is greatly reduced, and the aggregated data storage capacity is further reduced; meanwhile, by utilizing the GraphQL characteristics, when different service scenes of the same type of data are used for tuning and aggregating the search server, the server only returns corresponding data through specifying required fields, the data redundancy problem does not exist, the network transmission data volume is reduced, the request speed is further improved, in addition, the front end can simultaneously call a plurality of GraphQL interfaces through one network request, different data are simultaneously obtained, and the aggregation search capability of multi-source data is improved.
In some embodiments, the index data maintenance module 12 may specifically include:
the message queue generating unit is used for generating a Kafka message queue of index data to be maintained based on the monitored change data;
a key field reading unit, configured to read, by a pre-created Kafka consumer, a target key field of index data to be maintained in the Kafka message queue according to a queue order;
and the service calling unit is used for generating a service calling request based on the target key field and sending the service calling request to the aggregation search server so that the aggregation search server can identify the data type of the target key field in the service calling request after receiving the service calling request, and call the corresponding service microservice according to the data type.
In some specific embodiments, the aggregated data searching module 13 may specifically include:
a calling request receiving unit, configured to receive, by using the aggregation search server, a search service calling request sent by the gateway route after acquiring the data search request sent by the front end; the data search request is an aggregated data search request which is customized by the front end based on the GraphQL according to the target demand data type;
and the data query unit is used for querying the corresponding target aggregation data from the ES database according to the target demand data type and feeding the target aggregation data back to the front end.
In some embodiments, the index data maintenance module 12 may specifically include:
and the data monitoring unit is used for monitoring the binlog log of the target service database through a Canal server so as to monitor and obtain the change data of the target service database.
Further, the embodiment of the present application also discloses an electronic device, which is shown in fig. 8, and the content in the drawing cannot be considered as any limitation to the application scope.
Fig. 8 is a schematic structural diagram of an electronic device 20 according to an embodiment of the present disclosure. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input output interface 25, and a communication bus 26. The memory 22 is configured to store a computer program, and the computer program is loaded and executed by the processor 21 to implement relevant steps in the multi-source data aggregation search method disclosed in any one of the foregoing embodiments.
In this embodiment, the power supply 23 is configured to provide a working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and an external device, and a communication protocol followed by the communication interface is any communication protocol applicable to the technical solution of the present application, and is not specifically limited herein; the input/output interface 25 is configured to obtain external input data or output data to the outside, and a specific interface type thereof may be selected according to specific application requirements, which is not specifically limited herein.
In addition, the storage 22 is used as a carrier for resource storage, and may be a read-only memory, a random access memory, a magnetic disk or an optical disk, etc., and the resources stored thereon include an operating system 221, a computer program 222, data 223 including aggregated data, etc., and the storage may be a transient storage or a permanent storage.
The operating system 221 is configured to manage and control each hardware device and the computer program 222 on the electronic device 20, so as to implement the operation and processing of the mass data 223 in the memory 22 by the processor 21, and may be Windows Server, Netware, Unix, Linux, or the like. The computer program 222 may further include a computer program that can be used to perform other specific tasks in addition to the computer program that can be used to perform the multi-source data aggregation search method performed by the electronic device 20 disclosed in any of the foregoing embodiments.
Further, an embodiment of the present application further discloses a computer storage medium, where computer-executable instructions are stored in the computer storage medium, and when the computer-executable instructions are loaded and executed by a processor, the steps of the multi-source data aggregation search method disclosed in any of the foregoing embodiments are implemented.
The embodiments are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same or similar parts among the embodiments are referred to each other. The device disclosed by the embodiment corresponds to the method disclosed by the embodiment, so that the description is simple, and the relevant points can be referred to the method part for description.
The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in Random Access Memory (RAM), memory, Read Only Memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
Finally, it should also be noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
The method, the device, the equipment and the medium for multi-source data aggregation search provided by the invention are described in detail, a specific example is applied in the text to explain the principle and the implementation mode of the invention, and the description of the embodiment is only used for helping to understand the method and the core idea of the invention; meanwhile, for a person skilled in the art, according to the idea of the present invention, the specific embodiments and the application range may be changed, and in summary, the content of the present specification should not be construed as a limitation to the present invention.
Claims (8)
1. A multi-source data aggregation search method is characterized by comprising the following steps:
constructing an aggregation search server based on GraphQL, and configuring data aggregation formats corresponding to various business microservices according to the target application type;
monitoring a target service database to obtain corresponding changed data, calling corresponding service microservices by using the aggregation search server according to the data types of the changed data to obtain aggregated data corresponding to the changed data, and storing the aggregated data to an ES database;
receiving a data search request customized and sent by a front end by using the aggregation search server, and inquiring from the ES database according to the data search request to obtain corresponding target aggregation data and feeding the target aggregation data back to the front end;
the calling of the corresponding business microservices by the aggregation search server according to the data types of the changed data comprises the following steps:
generating a Kafka message queue of index data to be maintained based on the monitored change data;
reading target key fields of index data to be maintained in the Kafka message queue according to a queue sequence by a pre-established Kafka consumer;
and generating a service calling request based on the target key field and sending the service calling request to the aggregation search server, so that the aggregation search server identifies the data type of the target key field in the service calling request after receiving the service calling request, and calls the corresponding service micro service according to the data type.
2. The multi-source data aggregation search method according to claim 1, wherein the receiving, by the aggregation search server, a data search request customized and sent by a front end, and querying from the ES database according to the data search request to obtain corresponding target aggregated data and feeding back the target aggregated data to the front end, comprises:
receiving a search service calling request sent by a gateway route after acquiring the data search request sent by the front end by using the aggregation search server; the data search request is an aggregated data search request which is customized by the front end based on the GraphQL according to the target demand data type;
and inquiring from the ES database according to the target demand data type to obtain corresponding target aggregated data, and feeding back to the front end.
3. The multi-source data aggregation search method according to any one of claims 1 to 2, wherein the listening to the target business database to obtain corresponding change data comprises:
and monitoring the binlog log of the target service database through a Canal server to obtain the change data of the target service database through monitoring.
4. A multi-source data aggregation search apparatus, comprising:
the aggregation search server construction module is used for constructing an aggregation search server based on GraphQL and configuring data aggregation formats corresponding to various business microservices according to the target application types;
the index data maintenance module is used for monitoring a target business database to obtain corresponding changed data, calling corresponding business microservices by using the aggregation search server according to the data types of the changed data to obtain aggregated data corresponding to the changed data, and storing the aggregated data to an ES database;
the aggregated data searching module is used for receiving a data searching request customized and sent by a front end by using the aggregated searching server, and inquiring from the ES database according to the data searching request to obtain corresponding target aggregated data to feed back to the front end;
the index data maintenance module comprises:
a message queue generating unit, configured to generate a Kafka message queue of index data to be maintained based on the monitored change data;
a key field reading unit, configured to read, by a pre-created Kafka consumer, a target key field of index data to be maintained in the Kafka message queue according to a queue order;
and the service calling unit is used for generating a service calling request based on the target key field and sending the service calling request to the aggregation search server, so that the aggregation search server can identify the data type of the target key field in the service calling request after receiving the service calling request, and call the corresponding service microservice according to the data type.
5. The multi-source data aggregation search device according to claim 4, wherein the aggregated data search module comprises:
a calling request receiving unit, configured to receive, by using the aggregation search server, a search service calling request sent by a gateway route after acquiring the data search request sent by the front end; the data search request is an aggregated data search request which is customized by the front end based on the GraphQL according to the target demand data type;
and the data query unit is used for querying the corresponding target aggregation data from the ES database according to the target demand data type and feeding the target aggregation data back to the front end.
6. The multi-source data aggregation search apparatus according to any one of claims 4 to 5, wherein the index data maintenance module includes:
and the data monitoring unit is used for monitoring the binlog log of the target service database through a Canal server so as to monitor and obtain the change data of the target service database.
7. An electronic device, comprising:
a memory for storing a computer program;
a processor for executing the computer program to implement the multi-source data aggregation search method of any one of claims 1 to 3.
8. A computer-readable storage medium for storing a computer program; wherein the computer program when executed by the processor implements the multi-source data aggregation search method of any one of claims 1 to 3.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110429692.1A CN113032419B (en) | 2021-04-21 | 2021-04-21 | Multi-source data aggregation search method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110429692.1A CN113032419B (en) | 2021-04-21 | 2021-04-21 | Multi-source data aggregation search method, device, equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113032419A CN113032419A (en) | 2021-06-25 |
| CN113032419B true CN113032419B (en) | 2022-08-30 |
Family
ID=76457179
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110429692.1A Active CN113032419B (en) | 2021-04-21 | 2021-04-21 | Multi-source data aggregation search method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113032419B (en) |
Families Citing this family (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113923038A (en) * | 2021-10-19 | 2022-01-11 | 北京快确信息科技有限公司 | Aggregation test system for multi-scenario financial interface |
| CN113867921A (en) * | 2021-10-25 | 2021-12-31 | 联想(北京)有限公司 | Processing method, processing device, electronic equipment and storage medium |
| CN114374727B (en) * | 2022-01-13 | 2024-04-02 | 平安付科技服务有限公司 | Data calling method and device based on artificial intelligence, electronic equipment and medium |
| CN115510101A (en) * | 2022-09-13 | 2022-12-23 | 中电金信软件有限公司 | Data processing method and device, electronic equipment and readable storage medium |
| CN116521969B (en) * | 2023-02-28 | 2023-12-29 | 华为云计算技术有限公司 | Data retrieval method, server, system and related equipment |
| CN116821245A (en) * | 2023-07-05 | 2023-09-29 | 贝壳找房(北京)科技有限公司 | Data aggregation synchronization method and storage medium in distributed scene |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109587212A (en) * | 2018-10-31 | 2019-04-05 | 武汉艺果互动科技股份有限公司 | A kind of micro services architecture design method based on GraphQL, equipment and storage equipment |
| US10409811B1 (en) * | 2017-01-27 | 2019-09-10 | Intuit, Inc. | Decomposition of queries |
| CN110958131A (en) * | 2018-09-27 | 2020-04-03 | 丛林网络公司 | GraphQL-based queries supported on a YANG configuration data model |
| CN111597160A (en) * | 2020-04-21 | 2020-08-28 | 中国人民财产保险股份有限公司 | Distributed database system, distributed data processing method and device |
| CN112084245A (en) * | 2020-09-03 | 2020-12-15 | 深圳力维智联技术有限公司 | Data management method, device and equipment based on micro-service architecture and storage medium |
| CN112215533A (en) * | 2019-06-24 | 2021-01-12 | 秀铺菲公司 | System and method for facilitating electronic commerce product returns using orders for returned items |
| CN112231168A (en) * | 2020-09-03 | 2021-01-15 | 平安信托有限责任公司 | Micro server control method, device, equipment and storage medium |
| CN112363775A (en) * | 2020-11-11 | 2021-02-12 | 深圳力维智联技术有限公司 | Method, system and storage medium for dynamically modifying microservice configuration information |
| CN112363854A (en) * | 2020-11-11 | 2021-02-12 | 南京莱斯信息技术股份有限公司 | Front-end service framework optimization design method based on novel interface standard |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103198067B (en) * | 2012-01-06 | 2016-08-10 | 卓望数码技术（深圳）有限公司 | A kind of service search method and system |
| CN106708996B (en) * | 2016-12-19 | 2020-09-25 | 北京天广汇通科技有限公司 | Method and system for full text search of relational database |
| US11580165B2 (en) * | 2019-08-06 | 2023-02-14 | Twitter, Inc. | Event producer system of a messaging platform for delivering real-time messages |
| CN111737288B (en) * | 2020-06-05 | 2023-07-25 | 富途网络科技(深圳)有限公司 | Search control method, device, terminal equipment, server and storage medium |
-
2021
- 2021-04-21 CN CN202110429692.1A patent/CN113032419B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10409811B1 (en) * | 2017-01-27 | 2019-09-10 | Intuit, Inc. | Decomposition of queries |
| CN110958131A (en) * | 2018-09-27 | 2020-04-03 | 丛林网络公司 | GraphQL-based queries supported on a YANG configuration data model |
| CN109587212A (en) * | 2018-10-31 | 2019-04-05 | 武汉艺果互动科技股份有限公司 | A kind of micro services architecture design method based on GraphQL, equipment and storage equipment |
| CN112215533A (en) * | 2019-06-24 | 2021-01-12 | 秀铺菲公司 | System and method for facilitating electronic commerce product returns using orders for returned items |
| CN111597160A (en) * | 2020-04-21 | 2020-08-28 | 中国人民财产保险股份有限公司 | Distributed database system, distributed data processing method and device |
| CN112084245A (en) * | 2020-09-03 | 2020-12-15 | 深圳力维智联技术有限公司 | Data management method, device and equipment based on micro-service architecture and storage medium |
| CN112231168A (en) * | 2020-09-03 | 2021-01-15 | 平安信托有限责任公司 | Micro server control method, device, equipment and storage medium |
| CN112363775A (en) * | 2020-11-11 | 2021-02-12 | 深圳力维智联技术有限公司 | Method, system and storage medium for dynamically modifying microservice configuration information |
| CN112363854A (en) * | 2020-11-11 | 2021-02-12 | 南京莱斯信息技术股份有限公司 | Front-end service framework optimization design method based on novel interface standard |
Non-Patent Citations (1)
| Title |
|---|
| 一种云环境下分布式应用业务态势感知系统设计方法;郭杰等;《冶金自动化》;20200715(第04期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113032419A (en) | 2021-06-25 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113032419B (en) | Multi-source data aggregation search method, device, equipment and storage medium | |
| US8069224B2 (en) | Method, equipment and system for resource acquisition | |
| CN110351283B (en) | Data transmission method, device, equipment and storage medium | |
| CN110636093B (en) | Microservice registration and discovery method, microservice registration and discovery device, storage medium and microservice system | |
| CN111092877B (en) | Data processing method and device, electronic equipment and storage medium | |
| US10243919B1 (en) | Rule-based automation of DNS service discovery | |
| WO2020173080A1 (en) | Method and apparatus for querying call chain information | |
| CN111045833B (en) | Interface calling method and device | |
| CN113839977A (en) | Message pushing method and device, computer equipment and storage medium | |
| CN111984849A (en) | Information query method, device, equipment and medium | |
| WO2022169686A1 (en) | Distributed multi-source data processing and publishing platform | |
| CN111427899A (en) | Method, device, equipment and computer readable medium for storing file | |
| CN111241189A (en) | Method and device for synchronizing data | |
| CN112817539A (en) | Industrial data storage method and system, electronic device and storage medium | |
| US20180316780A1 (en) | Scalable programming architecture for telecommunications devices | |
| CN117041284A (en) | External interface management method, system, equipment and medium based on interface cloud | |
| CN114996361B (en) | Data processing method and system | |
| CN113645260A (en) | Service retry method, device, storage medium and electronic equipment | |
| CN116108036A (en) | Method and device for off-line exporting back-end system data | |
| CN113742172B (en) | Method, system and related device for collecting server logs | |
| CN112783842B (en) | Log collection method and device | |
| CN108491448B (en) | Data pushing method and device | |
| CN111913789A (en) | Program tracking method, device and medium supporting micro-service architecture | |
| CN111459981A (en) | Query task processing method, device, server and system | |
| CN112866587B (en) | Method and system for distributing spoken language video synthesis tasks |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20231107 Address after: 200441 Room B1-7001, Building 3, No. 258 Changjiang Road, Baoshan District, Shanghai Patentee after: Shanghai Mengju Information Technology Co.,Ltd. Address before: Room b1-8008, building 3, 258 Changjiang Road, Baoshan District, Shanghai 200441 Patentee before: Shanghai Weimeng Enterprise Development Co.,Ltd. |