CN109902105B - Data query system, method, device and storage medium for micro-service architecture - Google Patents
Data query system, method, device and storage medium for micro-service architecture Download PDFInfo
- Publication number
- CN109902105B CN109902105B CN201910148823.1A CN201910148823A CN109902105B CN 109902105 B CN109902105 B CN 109902105B CN 201910148823 A CN201910148823 A CN 201910148823A CN 109902105 B CN109902105 B CN 109902105B
- Authority
- CN
- China
- Prior art keywords
- micro
- service
- data
- log
- microservice
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Abstract
The invention provides a data query system, a method, equipment and a storage medium for a micro-service architecture, wherein the system comprises the following components: the micro-service data table management module is used for establishing a micro-service data table; the scheduling task management module is used for establishing a scheduling task library according to the micro-service data table, and the scheduling task consumes micro-service production data from the message middleware according to the corresponding message middleware theme of the micro-service, wherein the micro-service production data comprises a global unique identification code of the micro-service; a data storage module; and the data query module is used for acquiring a query request and querying a corresponding micro-service log from the search server according to the global unique identification code in the query request. By adopting the technical scheme of the invention, the data is quickly inquired and positioned in a big data environment, the data inquiry and problem diagnosis efficiency is improved, the visual inquiry result display can be provided, and the user experience is improved.
Description
Technical Field
The present invention relates to the field of data processing technologies, and in particular, to a data query system, method, device, and storage medium for a microservice architecture.
Background
In OTA (on line Travel Agency) enterprises or other internet enterprises, micro service architectures have become a trend or standard in the industry, with popularization and application of micro service architectures, numerous micro service combinations and architectures inevitably exist in the enterprises, the amount of data and logs generated by micro service ecology is very large, generally can reach the scale of hundreds of TBs or even tens of to hundreds of PBs, and the query processing of data and logs with large data amount is very troublesome due to the calling relationship between services in the upstream and downstream, and the problem that how to query data to be queried in the data and logs and perform data visualization is a problem to be faced due to the fact that the services are of excessive types and formats are not uniform.
For example, we need to query what happens to an order, for example, a user complains about finding that the order money is too much or too little, at this time, how to query logs and data, perform problem diagnosis and location, and accurately and quickly find a problem, and certainly we can check an original message or log by performing chain tracking on micro-service logs in an upstream-downstream relationship one by one service, but not only is the efficiency very low, but also a plurality of micro-service related operation and maintenance personnel are needed to support cooperation, and a large amount of resources are called to find and locate a problem.
Disclosure of Invention
Aiming at the problems in the prior art, the invention aims to provide a data query system, a data query method, data query equipment and a storage medium for a micro-service architecture, which can realize the rapid query and positioning of data in a big data environment and improve the efficiency of data query and problem diagnosis.
The embodiment of the invention provides a data query system for a microservice architecture, which comprises:
the micro-service data table management module is used for establishing a micro-service data table, and the micro-service data table comprises micro-service identifiers and message middleware themes of all micro-services;
the scheduling task management module is used for establishing a scheduling task library according to the micro-service data table, wherein scheduling tasks in the scheduling task library correspond to the micro-services, the scheduling tasks consume data produced by the micro-services from message middleware according to message middleware themes of the corresponding micro-services, and the data produced by the micro-services comprises global unique identification codes of the micro-services;
the data storage module is used for storing the data produced by the micro-services consumed by the scheduling tasks to a search server in the form of micro-service logs;
and the data query module is used for acquiring a query request and querying a corresponding micro-service log from the search server according to the global unique identification code in the query request.
Optionally, the microservice data table further includes microservice calling information of each microservice, and the system further includes a calling relation chain establishing module, where the calling relation chain establishing module is configured to establish a microservice calling chain according to the microservice calling information of each microservice.
Optionally, the globally unique identification code of each micro service is generated by an initial micro service in the micro service call chain, where the initial micro service is a micro service that is not called by other micro services in the call chain;
after the initial micro-service generates the global unique identification code of each micro-service, the downstream micro-service sequentially transfers the global unique identification code to the corresponding micro-service when calling the downstream micro-service, and each micro-service writes the global unique identification code into the message middleware when receiving the global unique identification code transferred by the upstream micro-service and continues to transfer the global unique identification code to the downstream micro-service.
Optionally, the micro-service data table further includes log index information of each micro-service, and when the data storage module stores the data produced by each micro-service to the search server, the data produced by each micro-service is stored with the corresponding log index information as an identifier.
Optionally, after the data query module obtains a query request from a user, the following steps are performed:
searching a micro-service log from the search server according to the global unique identification code in the query request;
determining search-associated microservices according to the incidence relation between the log index information of the searched microservice logs and microservice marks of the microservices;
generating a visual micro-service calling relation graph according to the calling relation chain, wherein the calling relation graph shows each micro-service and the calling relation between each micro-service and an upstream micro-service and a downstream micro-service;
and in the calling relation graph, identifying the micro service related to the search in a preset highlighting mode.
Optionally, after the data query module obtains a query request from a user, the following steps are performed:
searching a micro-service log from the search server according to the global unique identification code in the query request;
inquiring an upstream calling message, a downstream returning response message and a request message of the micro service log according to the searched micro service log;
and generating a visual graph or a visual chart by using the searched micro service log and the inquired message in a preset visual display mode.
Optionally, the system further comprises:
and the problem diagnosis module is used for inquiring the micro service logs to be analyzed from the search server and analyzing the inquired micro service logs according to a preset problem diagnosis rule.
The embodiment of the invention also provides a data query method for the micro-service architecture, which adopts the data query system for the micro-service architecture and comprises the following steps:
acquiring a query request, and extracting a global unique identification code from the query request;
and inquiring a corresponding micro service log from the search server according to the global unique identification code.
Optionally, the microservice data table further includes microservice calling information and log index information of each microservice, and when the data storage module stores the data produced by each microservice to the search server, the data produced by each microservice is stored with the corresponding log index information as an identifier; the method further comprises the steps of:
searching a micro-service log from the search server according to the global unique identification code in the query request;
determining search-associated microservices according to the incidence relation between the log index information of the searched microservice logs and microservice marks of the microservices;
generating a visual micro-service calling relation graph according to the micro-service calling information of each micro-service, wherein the calling relation graph shows the calling relation between each micro-service and each micro-service as well as the upstream micro-service and the downstream micro-service;
and in the calling relation graph, identifying the micro service related to the search in a preset highlighting mode.
Optionally, the method further comprises the steps of:
searching a micro-service log from the search server according to the global unique identification code in the query request;
inquiring an upstream calling message, a downstream returning response message and a request message of the micro service log according to the searched micro service log;
and generating a visual graph or a visual chart by using the searched micro service log and the inquired message in a preset visual display mode.
The embodiment of the invention also provides data query equipment, which comprises a processor; a memory having stored therein executable instructions of the processor;
wherein the processor is configured to perform the steps of the data query method for a microservice architecture via execution of the executable instructions.
The embodiment of the present invention further provides a computer-readable storage medium for storing a program, where the program is configured to implement the steps of the data query method for a microservice architecture when executed.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.
The data query system, the data query method, the data query equipment and the storage medium for the micro service architecture have the following advantages that:
according to the method, micro-service data are collected and stored one by one according to the upstream and downstream calling relations, a certain data structure is utilized to guarantee the fast and accurate query, and the fast query and positioning of the data are realized in a big data environment; the method has the advantages that the large data query of the full link based on the global unique identification code is realized, the micro-service data and logs of all related upstream and downstream calling relations can be obtained, the data query and problem diagnosis efficiency is improved, the analysis efficiency of possible problems of the application program is improved, and the related logic of the generated problems is helped to be quickly positioned; and visual query result display can be provided, and the use experience of a user is improved.
Drawings
Other features, objects and advantages of the present invention will become more apparent upon reading of the following detailed description of non-limiting embodiments thereof, with reference to the accompanying drawings.
FIG. 1 is a block diagram of a data query system for a microservice architecture according to an embodiment of the present invention;
FIG. 2 is a flow chart of a data query method for a microservice architecture in accordance with an embodiment of the present invention;
FIG. 3 is a diagram illustrating a call relationship diagram according to an embodiment of the invention;
FIG. 4 is a flow diagram of a call relationship graphical display of an embodiment of the present invention;
FIG. 5 is a flow diagram of query log visualization according to an embodiment of the invention;
FIG. 6 is a schematic structural diagram of a data query device for a microservice architecture according to an embodiment of the present invention;
fig. 7 is a schematic structural diagram of a computer-readable storage medium according to an embodiment of the present invention.
Detailed Description
Example embodiments will now be described more fully with reference to the accompanying drawings. Example embodiments may, however, be embodied in many different forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.
Furthermore, the drawings are merely schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and thus their repetitive description will be omitted. Some of the block diagrams shown in the figures are functional entities and do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in the form of software, or in one or more hardware modules or integrated circuits, or in different networks and/or processor devices and/or microcontroller devices.
As shown in fig. 1, to solve the technical problems in the prior art, an embodiment of the present invention provides a data query system for a microservice architecture, where the system includes:
the micro-service data table management module M100 is used for establishing a micro-service data table, wherein the micro-service data table comprises micro-service identifiers and message middleware themes of all micro-services;
the scheduling task management module M200 is used for establishing a scheduling task library according to the micro-service data table, wherein scheduling tasks in the scheduling task library correspond to the micro-services, the scheduling tasks consume data produced by the micro-services from message middleware according to message middleware themes of the corresponding micro-services, and the data produced by the micro-services comprises global unique identification codes of the micro-services;
the scheduling task is an ETL (Extract Transform Load) scheduling task.
The data storage module M300 is used for storing the data produced by the micro-services consumed by the scheduling tasks to a search server in the form of micro-service logs;
and the data query module M400 is configured to obtain a query request, and query a corresponding micro service log from the search server according to the globally unique identification code in the query request.
As shown in fig. 2, an embodiment of the present invention further provides a data query method for a microservice architecture, where the data query system for the microservice architecture is adopted, and the method includes the following steps:
s100: acquiring a query request;
s200: extracting a globally unique identification code from the query request;
s300: and inquiring a corresponding micro service log from the search server according to the global unique identification code.
Therefore, the method and the device can ensure the fast and accurate query by acquiring the micro-service data and storing the micro-service data one by one according to the upstream and downstream calling relations and utilize a certain data structure to realize the fast query and positioning of the data in a big data environment; the method and the device realize the large data query of the full link based on the global unique identification code, can obtain the micro service data and logs of all related upstream and downstream calling relations, and improve the data query efficiency.
In this embodiment, the microservice data table further includes microservice invocation information for each microservice. In the relationship, the international aggregation layer is upstream of the international engine, the international engine is downstream of the international aggregation, and the GDS is downstream of the international engine, and the upstream and downstream relationship is not 1 to 1, usually 1 to more, and can form a graph structure in a similar data structure.
In this embodiment, the microservice data table further includes log index information of each microservice, and when the data storage module stores the data produced by each microservice to the search server, the data produced by each microservice is stored with the corresponding log index information as an identifier.
Table 1 below is a schematic diagram of a microservice data table in this embodiment:
wherein, ID is micro service identification, and patent ID identifies the calling relationship of micro service, for example, micro service 1 is the initial micro service and is not called by other micro services, micro services 2, 3, 4 are all called by micro service 1, and micro services 5, 6, 7 are all called by micro service 2.
The data table can be maintained, and expanded or reduced according to the development of business, each microservice has a corresponding unique identifier, a name, a theme, a log index and the like, and a calling chain diagram can be formed according to the upstream and downstream relation.
The system further comprises a calling relation chain establishing module, wherein the calling relation chain establishing module is used for establishing the micro-service calling chain according to the micro-service calling information of each micro-service, and the micro-service calling chain corresponding to the table 1 is shown in fig. 3.
In this embodiment, the globally unique identification code of each microservice is generated by an initial microservice in the microservice call chain, and the initial microservice is a microservice that is not called by other microservices in the call chain; after the initial micro service generates the global unique identification code of each micro service, the downstream micro service sequentially transmits the global unique identification code to the corresponding micro service when calling the downstream micro service, and each micro service writes the global unique identification code (UUID) into the message middleware when receiving the global unique identification code transmitted by the upstream micro service and continues to transmit the global unique identification code to the downstream micro service.
I.e., the generation of the global ID, is generated by the first initial microservice in the call chain relationship, which is usually associated with the user interface, in the form of a 32-bit UUID.
A UUID consists of a combination of:
(1) the current date and time, the first part of the UUID is time dependent, and if you generate a UUID again a few seconds after it, the first part is different and the rest is the same.
(2) A clock sequence.
(3) The globally unique IEEE machine identification number, if there is a network card, is obtained from the network card MAC address, and no network card is obtained in other ways.
After the initial micro service generates the UUID, the UUID is transferred when the subsequent micro service is called until the end of the call chain is reached. When a micro service has no upstream call or the upstream call does not contain the UUID, the micro service can also generate the UUID, and the smoothness of a subsequent link is ensured.
As shown in fig. 4, in this embodiment, after the data query module obtains a query request from a user, the following steps are performed:
s410: searching a micro-service log from the search server according to the global unique identification code in the query request;
s420: determining search-associated microservices according to the incidence relation between the log index information of the searched microservice logs and microservice marks of the microservices;
s430: generating a visual micro-service calling relation graph according to the calling relation chain, wherein the calling relation graph shows each micro-service and the calling relation between each micro-service and an upstream micro-service and a downstream micro-service;
s440: and in the calling relation graph, identifying the micro service related to the search in a preset highlighting mode.
When the user inquires by the global ID, the data inquiry module searches and obtains data from the search server according to the inquiry condition. Because the subjects falling to the ground all contain the global ID, all logs can be searched by using the global ID.
If the micro service data table contains 7 micro services, the system searches the 7 micro services, searches whether each micro service log contains the global ID, and if the micro service log contains the global ID, the system queries the micro service data table according to the log index name to obtain the corresponding micro service name and the corresponding micro service identifier.
And then constructing visual analysis according to the micro-service relationship, wherein the initial visual analysis comprises a log data table and a micro-service relationship calling chain chart related to the query.
For example, the call relation graph uses an output structure similar to that of appendix FIG. 3, with invoked microservices being displayed in green and those not invoked in gray. As in fig. 3, it is possible that the call chain is microservice 1 → microservice 2 → microservice 5, and the circles represented by these three passing microservices will be labeled green.
Generally, all micro services are not lighted up to be green, and since the micro services passed by each call are not identical and are matched differently according to the services, paths formed by nodes of the micro services passed by the call are different.
As shown in fig. 5, in this embodiment, after the data query module obtains a query request from a user, the following steps are performed:
s510: searching a micro-service log from the search server according to the global unique identification code in the query request;
s520: inquiring an upstream calling message, a downstream returning response message and a request message of the micro service log according to the searched micro service log;
s530: and generating a visual graph or a visual chart by using the searched micro service log and the inquired message in a preset visual display mode.
Clicking a single log in the log data table can perform visual display on an upstream call message, a downstream return response message and a request message of the log, wherein the visual display comprises a tree structure, a data table, a pie chart, a histogram and a broken line graph.
a. The tree structure can provide visual analysis based on the tree structure if the message native structure is JSON or XML.
b. And the data table shows the structured data in the log data, and the data is composed of a plurality of pieces.
c. A pie chart, which shows the composition of data in the log data, and this data shows the proportion of a certain dimension in the total composition.
d. The histogram shows the arrangement of certain value data in the log data according to certain dimension.
e. And the line graph shows the development trend of certain value data in the log data according to a certain dimension.
The data visualization is customized because the format of some microservice log is relatively fixed and does not change freely. The data visualization can be a tree mechanism, a single data table, a pie chart, a bar chart, a line chart, or a combination of several of 5 ways. For example, the international engine microservice displays a flight information table, and simultaneously can display a price trend chart according to flight dimensions and a JSON tree structure chart.
In this embodiment, the data query system for a microservice architecture may further include a problem diagnosis module, configured to query a microservice log to be analyzed from the search server, and analyze the queried microservice log according to a preset problem diagnosis rule.
Specifically, the problem diagnosis module mainly combines historical data, firstly analyzes whether the micro service call joint debugging has deficiency, omission and change, secondly analyzes the difference of the message and the recent message, including the aspects of structure, size, key data and the like, and provides a diagnosis analysis conclusion and a corresponding possible solution.
a. History comparison, which micro-services may be lacked and whether missing, missing or changing exists according to the history data comparison of similar call chains. And comparing the log of the same microservice with historical data, and analyzing the structure, the size and key data.
b. In the upstream and downstream comparison, according to the calling relationship, the difference between the upstream and downstream data, such as the price, is 100 in the a micro service and 90 in the B micro service, which may be problematic or may not be problematic.
c. And (4) rule comparison, namely comparing history with synchronization, upstream and downstream according to some special set comparison rules, and giving problem diagnosis analysis.
The analysis result can provide a certain problem diagnosis analysis reference for the user. Therefore, the method and the device can improve the efficiency of analyzing the possible problems of the application program and help to quickly locate the relevant logic generating the problems.
Therefore, the following beneficial effects can be obtained by adopting the data query system and method for the micro service architecture of the embodiment:
1) each micro service records and writes a log into the message middleware, then consumes the message data in the ETL angle task process and falls to the ES for storage, and the delay from the acquisition to the falling of the data does not exceed 10 seconds.
2) And inquiring by using inquiry conditions such as global ID, order ID, date and the like, quickly executing big data inquiry, and obtaining micro service data and log lists of all related upstream and downstream calling relations according to conditions.
3) Through a visualization mode, specific details of a piece of micro-service data or log can be checked, including request data from upstream and call return from downstream, and the specific details are displayed in a table or chart form.
4) By means of problem diagnosis or intelligent analysis, efficiency and accuracy of big data query are improved, possible problems of the application program can be analyzed, and micro services which may cause problems can be rapidly located.
The embodiment of the invention also provides data query equipment for the micro-service architecture, which comprises a processor; a memory having stored therein executable instructions of the processor; wherein the processor is configured to perform the steps of the data query method for a microservice architecture via execution of the executable instructions.
As will be appreciated by one skilled in the art, aspects of the present invention may be embodied as a system, method or program product. Thus, various aspects of the invention may be embodied in the form of: an entirely hardware embodiment, an entirely software embodiment (including firmware, microcode, etc.) or an embodiment combining hardware and software aspects that may all generally be referred to herein as a "circuit," module "or" platform.
An electronic device 600 according to this embodiment of the invention is described below with reference to fig. 6. The electronic device 600 shown in fig. 6 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present invention.
As shown in fig. 6, the electronic device 600 is embodied in the form of a general purpose computing device. The components of the electronic device 600 may include, but are not limited to: at least one processing unit 610, at least one memory unit 620, a bus 630 connecting the different platform components (including the memory unit 620 and the processing unit 610), a display unit 640, etc.
Wherein the storage unit stores program code executable by the processing unit 610 to cause the processing unit 610 to perform steps according to various exemplary embodiments of the present invention described in the above-mentioned electronic prescription flow processing method section of the present specification. For example, the processing unit 610 may perform the steps as shown in fig. 2.
The storage unit 620 may include readable media in the form of volatile memory units, such as a random access memory unit (RAM)6201 and/or a cache memory unit 6202, and may further include a read-only memory unit (ROM) 6203.
The memory unit 620 may also include a program/utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including, but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which, or some combination thereof, may comprise an implementation of a network environment.
The electronic device 600 may also communicate with one or more external devices 700 (e.g., keyboard, pointing device, bluetooth device, etc.), with one or more devices that enable a user to interact with the electronic device 600, and/or with any devices (e.g., router, modem, etc.) that enable the electronic device 600 to communicate with one or more other computing devices. Such communication may occur via an input/output (I/O) interface 650. Also, the electronic device 600 may communicate with one or more networks (e.g., a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network such as the Internet) via the network adapter 660. The network adapter 660 may communicate with other modules of the electronic device 600 via the bus 630. It should be appreciated that although not shown in the figures, other hardware and/or software modules may be used in conjunction with the electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms, to name a few.
The embodiment of the present invention further provides a computer-readable storage medium for storing a program, where the program implements the steps of the data query method for a microservice architecture when executed. In some possible embodiments, aspects of the present invention may also be implemented in the form of a program product comprising program code for causing a terminal device to perform the steps according to various exemplary embodiments of the present invention described in the above-mentioned electronic prescription flow processing method section of this specification, when the program product is run on the terminal device.
Referring to fig. 7, a program product 800 for implementing the above method according to an embodiment of the present invention is described, which may employ a portable compact disc read only memory (CD-ROM) and include program code, and may be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited in this regard and, in the present document, a readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
The computer readable storage medium may include a propagated data signal with readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A readable storage medium may also be any readable medium that is not a readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., through the internet using an internet service provider).
The data query system, the data query method, the data query equipment and the storage medium for the micro-service architecture have the following advantages that:
according to the method, micro-service data are collected and stored one by one according to the upstream and downstream calling relations, a certain data structure is utilized to guarantee the fast and accurate query, and the fast query and positioning of the data are realized in a big data environment; the method has the advantages that the large data query of the full link based on the global unique identification code is realized, the micro-service data and logs of all related upstream and downstream calling relations can be obtained, the data query and problem diagnosis efficiency is improved, the analysis efficiency of possible problems of the application program is improved, and the related logic of the generated problems is helped to be quickly positioned; and visual query result display can be provided, and the use experience of a user is improved.
The foregoing is a more detailed description of the invention in connection with specific preferred embodiments and it is not intended that the invention be limited to these specific details. For those skilled in the art to which the invention pertains, several simple deductions or substitutions can be made without departing from the spirit of the invention, and all shall be considered as belonging to the protection scope of the invention.
Claims (8)
1. A data query system for a microservice architecture, comprising:
the micro-service data table management module is used for establishing a micro-service data table, and the micro-service data table comprises micro-service identification of each micro-service, a message middleware theme, micro-service calling information of each micro-service and log index information of each micro-service;
the scheduling task management module is used for establishing a scheduling task library according to the micro-service data table, wherein scheduling tasks in the scheduling task library correspond to the micro-services, the scheduling tasks consume data produced by the micro-services from message middleware according to message middleware themes of the corresponding micro-services, and the data produced by the micro-services comprises global unique identification codes of the micro-services;
the data storage module is used for storing the data produced by the micro-service and consumed by the scheduling task to a search server in the form of a micro-service log by taking the corresponding log index information as an identifier;
the data query module is used for acquiring a query request and querying a corresponding micro-service log from the search server according to the global unique identification code in the query request;
the calling relation chain establishing module is used for establishing a micro-service calling chain according to micro-service calling information of each micro-service;
after the data query module acquires a query request of a user, the following steps are executed:
searching a micro-service log from the search server according to the global unique identification code in the query request;
determining search-associated microservices according to the incidence relation between the log index information of the searched microservice logs and microservice marks of the microservices;
generating a visual micro-service calling relation graph according to the calling relation chain, wherein the calling relation graph shows each micro-service and the calling relation between each micro-service and an upstream micro-service and a downstream micro-service;
and in the calling relation graph, identifying the micro service related to the search in a preset highlighting mode.
2. The data query system for a microservice architecture of claim 1, wherein the globally unique identifier of each microservice is generated by an initial microservice in the microservice call chain, the initial microservice being a microservice in the call chain that is not called by other microservices;
after the initial micro-service generates the global unique identification code of each micro-service, the downstream micro-service sequentially transfers the global unique identification code to the corresponding micro-service when calling the downstream micro-service, and each micro-service writes the global unique identification code into the message middleware when receiving the global unique identification code transferred by the upstream micro-service and continues to transfer the global unique identification code to the downstream micro-service.
3. The data query system for microservice architecture of claim 1, wherein the data query module, after obtaining the query request from the user, performs the following steps:
searching a micro-service log from the search server according to the global unique identification code in the query request;
inquiring an upstream calling message, a downstream returning response message and a request message of the micro service log according to the searched micro service log;
and generating a visual graph or a visual chart by using the searched micro service log and the inquired message in a preset visual display mode.
4. The data query system for a microservice architecture of claim 1, wherein the system further comprises:
and the problem diagnosis module is used for inquiring the micro service logs to be analyzed from the search server and analyzing the inquired micro service logs according to a preset problem diagnosis rule.
5. A data query method for a micro service architecture, which is characterized in that the data query system for the micro service architecture of any one of claims 1 to 4 is adopted, and the method comprises the following steps:
acquiring a query request, and extracting a global unique identification code from the query request;
inquiring a corresponding micro service log from the search server according to the global unique identification code;
the micro-service data table also comprises micro-service calling information and log index information of each micro-service, and the data storage module stores the data produced by each micro-service by taking the corresponding log index information as an identifier when storing the data produced by each micro-service to the search server; the method further comprises the steps of:
searching a micro-service log from the search server according to the global unique identification code in the query request;
determining search-associated microservices according to the incidence relation between the log index information of the searched microservice logs and microservice marks of the microservices;
generating a visual micro-service calling relation graph according to the micro-service calling information of each micro-service, wherein the calling relation graph shows the calling relation between each micro-service and each micro-service as well as the upstream micro-service and the downstream micro-service;
and in the calling relation graph, identifying the micro service related to the search in a preset highlighting mode.
6. The data query method for microservice architecture of claim 5, wherein the method further comprises the steps of:
searching a micro-service log from the search server according to the global unique identification code in the query request;
inquiring an upstream calling message, a downstream returning response message and a request message of the micro service log according to the searched micro service log;
and generating a visual graph or a visual chart by using the searched micro service log and the inquired message in a preset visual display mode.
7. A data query device for a microservice architecture, comprising:
a processor;
a memory having stored therein executable instructions of the processor;
wherein the processor is configured to perform the steps of the data query method for a microservice architecture of claim 5 or 6 via execution of the executable instructions.
8. A computer-readable storage medium storing a program which, when executed, performs the steps of the data query method for a microservice architecture of claim 5 or 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910148823.1A CN109902105B (en) | 2019-02-28 | 2019-02-28 | Data query system, method, device and storage medium for micro-service architecture |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910148823.1A CN109902105B (en) | 2019-02-28 | 2019-02-28 | Data query system, method, device and storage medium for micro-service architecture |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109902105A CN109902105A (en) | 2019-06-18 |
| CN109902105B true CN109902105B (en) | 2021-08-17 |
Family
ID=66945915
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910148823.1A Active CN109902105B (en) | 2019-02-28 | 2019-02-28 | Data query system, method, device and storage medium for micro-service architecture |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109902105B (en) |
Families Citing this family (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110427299B (en) * | 2019-07-19 | 2022-02-11 | 腾讯科技（深圳）有限公司 | Log processing method, related device and system for micro-service system application |
| CN112306848B (en) * | 2019-07-31 | 2023-11-03 | 中国移动通信集团浙江有限公司 | Architecture view generation method and device of micro-service system |
| CN111049878B (en) * | 2019-10-23 | 2023-03-21 | 平安科技（深圳）有限公司 | Method, device, electronic device and storage medium for visualizing micro-service call chain |
| CN110795484A (en) * | 2019-10-25 | 2020-02-14 | 北京浪潮数据技术有限公司 | Micro service equipment and data translation method, device and storage medium thereof |
| CN111078447B (en) * | 2019-11-24 | 2023-09-19 | 杭州安恒信息技术股份有限公司 | Abnormality positioning method, device, equipment and medium in micro-service architecture |
| CN111143286B (en) * | 2019-12-13 | 2024-01-09 | 无锡华云数据技术服务有限公司 | Cloud platform log management method and system |
| CN111651488B (en) * | 2020-05-29 | 2023-09-12 | 泰康保险集团股份有限公司 | Method, device, medium and electronic equipment for processing database business |
| CN111858007A (en) * | 2020-07-29 | 2020-10-30 | 广州海鹚网络科技有限公司 | Task scheduling method and device based on message middleware |
| CN114077619A (en) * | 2020-08-20 | 2022-02-22 | 北京字节跳动网络技术有限公司 | Data query method and device, electronic equipment and storage medium |
| CN112000564A (en) * | 2020-09-09 | 2020-11-27 | 中盈优创资讯科技有限公司 | Log visual presentation management method and device |
| CN113364775B (en) * | 2021-06-04 | 2023-04-18 | 北京京东振世信息技术有限公司 | Calling method and device of microservice and server |
| CN114817821B (en) * | 2022-06-30 | 2022-09-16 | 数字广东网络建设有限公司 | Display method and device of government affair information, electronic equipment and storage medium |
| CN117194176A (en) * | 2023-11-03 | 2023-12-08 | 中国电子科技集团公司第十五研究所 | Non-invasive operation monitoring method, device, electronic equipment and storage medium |
Family Cites Families (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20160103577A1 (en) * | 2014-10-09 | 2016-04-14 | Wrap Media, LLC | Creating and delivering a vehicle manual in the form of an interactive wrapped package of cards |
| CN107483221B (en) * | 2016-06-08 | 2021-04-27 | 阿里巴巴集团控股有限公司 | Cross-application problem troubleshooting method, device and system |
| CN107544984B (en) * | 2016-06-27 | 2020-07-31 | 北京京东尚科信息技术有限公司 | Data processing method and device |
| CN106850258A (en) * | 2016-12-22 | 2017-06-13 | 北京锐安科技有限公司 | A kind of Log Administration System, method and device |
| CN107018042B (en) * | 2017-04-13 | 2020-03-20 | 广东神马搜索科技有限公司 | Tracking method and tracking system for online service system |
| US20180308072A1 (en) * | 2017-04-21 | 2018-10-25 | Gem | Method and apparatus for blockchain management |
| AU2018280266B2 (en) * | 2017-06-09 | 2020-09-10 | Equinix, Inc. | Near real-time messaging service for data center infrastructure monitoring data |
| CN107357856B (en) * | 2017-06-29 | 2020-07-14 | 广西电网有限责任公司 | Method for realizing data integration and data service based on power grid panoramic service model |
| US10379934B2 (en) * | 2017-07-31 | 2019-08-13 | Oracle International Corporation | System and method of providing post error analysis for instances of applications in cloud service environments on a per user basis |
| CN107479990B (en) * | 2017-08-11 | 2020-12-18 | 恒丰银行股份有限公司 | Distributed software service system |
| CN107861859B (en) * | 2017-11-22 | 2021-04-02 | 北京汇通金财信息科技有限公司 | Log management method and system based on micro-service architecture |
| CN108038145A (en) * | 2017-11-23 | 2018-05-15 | 携程旅游网络技术（上海）有限公司 | Distributed Services tracking, system, storage medium and electronic equipment |
| CN107861823A (en) * | 2017-11-23 | 2018-03-30 | 国云科技股份有限公司 | A kind of method of the system security data final consistency based on micro services framework |
| CN108600012A (en) * | 2018-04-26 | 2018-09-28 | 深圳光华普惠科技有限公司 | Micro services framework monitoring system |
-
2019
- 2019-02-28 CN CN201910148823.1A patent/CN109902105B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN109902105A (en) | 2019-06-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109902105B (en) | Data query system, method, device and storage medium for micro-service architecture | |
| US10318595B2 (en) | Analytics based on pipes programming model | |
| CN108694221B (en) | Data real-time analysis method, module, equipment and device | |
| US20150089415A1 (en) | Method of processing big data, apparatus performing the same and storage media storing the same | |
| CN112231568B (en) | Hotel label automatic configuration method and device, electronic equipment and storage medium | |
| US20220214872A1 (en) | Dynamic review of software updates after pull requests | |
| CN110795697A (en) | Logic expression obtaining method and device, storage medium and electronic device | |
| CN115129753A (en) | Data blood relationship analysis method and device, electronic equipment and storage medium | |
| US20090106459A1 (en) | Configuration identification tool and methods | |
| CN109684198B (en) | Method, device, medium and electronic equipment for acquiring data to be tested | |
| CN113297287B (en) | Automatic user policy deployment method and device and electronic equipment | |
| CN113962597A (en) | Data analysis method and device, electronic equipment and storage medium | |
| CN112818070A (en) | Data query method and device based on global data dictionary and electronic equipment | |
| CN113157523A (en) | Service monitoring method and device, computer equipment and storage medium | |
| CN112380403A (en) | Type selection method, device, equipment and computer readable storage medium | |
| CN116450723A (en) | Data extraction method, device, computer equipment and storage medium | |
| CN102880927A (en) | A method and apparatus for enterprise intelligence ('ei') management in an ei framework | |
| CN113051171B (en) | Interface testing method, device, equipment and storage medium | |
| CN115080570A (en) | Graph database-based field blood relationship processing method and system | |
| CN115114297A (en) | Data lightweight storage and search method and device, electronic equipment and storage medium | |
| CN113138906A (en) | Call chain data acquisition method, device, equipment and storage medium | |
| US20140164397A1 (en) | Apparatus and method for searching information | |
| CN111045983A (en) | Nuclear power station electronic file management method and device, terminal equipment and medium | |
| CN115563183B (en) | Query method, query device and program product | |
| CN112988507B (en) | Service monitoring method, device, equipment, storage medium and computer program product |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |