CN113946491A - Microservice data processing method, microservice data processing device, computer equipment and storage medium - Google Patents
Microservice data processing method, microservice data processing device, computer equipment and storage medium Download PDFInfo
- Publication number
- CN113946491A CN113946491A CN202111216620.5A CN202111216620A CN113946491A CN 113946491 A CN113946491 A CN 113946491A CN 202111216620 A CN202111216620 A CN 202111216620A CN 113946491 A CN113946491 A CN 113946491A
- Authority
- CN
- China
- Prior art keywords
- service
- micro
- calling
- microservice
- registration
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012545 processing Methods 0.000 title claims abstract description 79
- 238000003672 processing method Methods 0.000 title claims description 19
- 238000003860 storage Methods 0.000 title claims description 14
- 238000000034 method Methods 0.000 claims abstract description 72
- 230000002159 abnormal effect Effects 0.000 claims abstract description 19
- 238000004364 calculation method Methods 0.000 claims abstract description 13
- 230000007246 mechanism Effects 0.000 claims abstract description 12
- 230000008569 process Effects 0.000 claims description 42
- 238000013507 mapping Methods 0.000 claims description 21
- 238000010606 normalization Methods 0.000 claims description 21
- 230000004044 response Effects 0.000 claims description 14
- 238000004590 computer program Methods 0.000 claims description 13
- 238000011084 recovery Methods 0.000 claims description 11
- 238000004458 analytical method Methods 0.000 claims description 10
- 101000771640 Homo sapiens WD repeat and coiled-coil-containing protein Proteins 0.000 claims description 7
- 102100029476 WD repeat and coiled-coil-containing protein Human genes 0.000 claims description 7
- 238000012544 monitoring process Methods 0.000 abstract description 5
- 230000002354 daily effect Effects 0.000 description 24
- 230000006870 function Effects 0.000 description 6
- 238000010586 diagram Methods 0.000 description 4
- 238000007726 management method Methods 0.000 description 4
- 230000002776 aggregation Effects 0.000 description 3
- 238000004220 aggregation Methods 0.000 description 3
- 238000004891 communication Methods 0.000 description 3
- 238000009826 distribution Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000001360 synchronised effect Effects 0.000 description 2
- 230000005856 abnormality Effects 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000003203 everyday effect Effects 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
- G06F11/3006—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1415—Saving, restoring, recovering or retrying at system level
- G06F11/1417—Boot up procedures
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Quality & Reliability (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention relates to the technical field of server monitoring, and discloses a method, a device, equipment and a medium for processing micro-service data, wherein the method comprises the following steps: acquiring registration information in the registration request; registering the micro-service into a service table of a registration center based on the registration information; elastic resource calculation is carried out on the historical calling record associated with the micro-service, and a standby server is set up; based on the registration information and the standby server, carrying out snapshot processing on the micro-service to obtain a mirror image micro-service corresponding to the micro-service; when the registration center detects that the micro-service calling duration is abnormal, starting mirror micro-services corresponding to the micro-services, and registering the mirror micro-services into a service table; and distributing the calling of the micro-service and the mirror image micro-service by using a shunting mechanism. Therefore, the invention realizes management only by recording the calling time length through the registration center, improves the service operation speed and improves the accuracy and efficiency of micro-service data processing.
Description
Technical Field
The invention relates to the technical field of server monitoring, in particular to a micro-service data processing method and device, computer equipment and a storage medium.
Background
In the cloud computing era, micro-service architectures are increasingly applied to development and deployment of distributed systems, the micro-service architecture divides an original huge single system into micro-service nodes with independent functions, and each micro-service node can be used for realizing one service. Meanwhile, a plurality of dependency relationships exist among a plurality of micro service nodes, one micro service node can be called by one or more other micro service nodes, once a certain micro service node in the micro service architecture loses response due to a fault and the like and can not return a request result after a delay, the micro service node calling the micro service node continuously tries to resend the request and waits, service resources are quickly consumed, the micro service nodes in the micro service architecture are blocked in a large area, and the communication efficiency and the service fluency among the micro service nodes in the micro service architecture are reduced.
Disclosure of Invention
The invention provides a micro-service data processing method and device, computer equipment and a storage medium, which realize management only by recording calling duration through a registration center, improve service running speed, directly recover service from a mirror image and have higher starting speed.
A micro-service data processing method comprises the following steps:
receiving a registration request of a micro service, and acquiring registration information in the registration request;
registering the micro-service into a service table of a registration center based on the registration information;
elastic resource calculation is carried out on the historical call records associated with the micro-service through the registration center, and a standby server is built;
based on the registration information and the standby server, carrying out snapshot processing on the micro-service to obtain a mirror image micro-service corresponding to the micro-service;
when the registration center detects that the micro-service calling duration is abnormal, starting the mirror micro-service corresponding to the micro-service, and registering the mirror micro-service into the service table;
and distributing the micro-service and the calling of the mirror image micro-service through the registration center by using a shunting mechanism.
A microservice data processing apparatus comprising:
the system comprises a receiving module, a registration module and a processing module, wherein the receiving module is used for receiving a registration request of the microservice and acquiring registration information in the registration request;
the registration module is used for registering the microservice into a service table of a registration center based on the registration information;
the building module is used for performing elastic resource calculation on the historical calling record associated with the micro-service through the registration center and building a standby server;
the snapshot module is used for carrying out snapshot processing on the micro-service based on the registration information and the standby server to obtain mirror image micro-service corresponding to the micro-service;
the starting module is used for starting the mirror image microservice corresponding to the microservice and registering the mirror image microservice into the service table when the registry detects that the microservice calling time length is abnormal;
and the calling module is used for distributing the calling of the micro-service and the mirror image micro-service through the registration center by using a shunting mechanism.
A computer device comprising a memory, a processor and a computer program stored in said memory and executable on said processor, said processor implementing the steps of the above-mentioned microservice data processing method when executing said computer program.
A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out the steps of the above-mentioned microservice data processing method.
The invention provides a micro-service data processing method, a device, computer equipment and a storage medium, wherein the method obtains registration information in a registration request by receiving the registration request of a micro-service; registering the micro-service into a service table of a registration center based on the registration information; elastic resource calculation is carried out on the historical call records associated with the micro-service through the registration center, and a standby server is built; based on the registration information and the standby server, carrying out snapshot processing on the micro-service to obtain a mirror image micro-service corresponding to the micro-service; when the registration center detects that the micro-service calling duration is abnormal, starting the mirror micro-service corresponding to the micro-service, and registering the mirror micro-service into the service table; the method and the system have the advantages that the shunting mechanism is used, the micro service and the mirror image micro service are distributed through the registration center, so that the micro service is registered through the registration center, the standby server is calculated and established through the elastic resources, the mirror image micro service is obtained through snapshot processing, when the micro service calling abnormity is detected, the mirror image micro service is automatically started, the shunting mechanism is used for automatically providing service together with the micro service, therefore, monitoring service without a container is achieved, only the calling time length is recorded through the registration center for management, the middlewares such as the container and the like are omitted, system resources are saved, the service operation speed is increased, the service is directly recovered from the mirror image, and the starting speed is higher.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings needed to be used in the description of the embodiments of the present invention will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art that other drawings can be obtained according to these drawings without inventive labor.
FIG. 1 is a schematic diagram of an application environment of a method for processing microservice data according to an embodiment of the present invention;
FIG. 2 is a flow chart of a method for processing microservice data in an embodiment of the present invention;
FIG. 3 is a flowchart illustrating step S30 of the method for processing microservice data in accordance with an embodiment of the present invention;
FIG. 4 is a flowchart illustrating step S301 of a method for processing microservice data according to an embodiment of the present invention;
FIG. 5 is a flowchart illustrating step S50 of the method for processing microservice data in accordance with an embodiment of the present invention;
FIG. 6 is a functional block diagram of a microservice data processing apparatus in accordance with an embodiment of the present invention;
FIG. 7 is a schematic diagram of a computer device in an embodiment of the invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The method for processing microservice data provided by the invention can be applied to the application environment shown in fig. 1, wherein a client (computer equipment or terminal) communicates with a server through a network. The client (computer device or terminal) includes, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers, and portable wearable devices. The server may be an independent server, or may be a cloud server that provides basic cloud computing services such as a cloud service, a cloud database, cloud computing, a cloud function, cloud storage, a Network service, cloud communication, a middleware service, a domain name service, a security service, a Content Delivery Network (CDN), a big data and artificial intelligence platform, and the like.
In an embodiment, as shown in fig. 2, a method for processing microservice data is provided, which mainly includes the following steps S10-S60:
s10, receiving the registration request of the micro service, and acquiring the registration information in the registration request.
Understandably, the registration information includes a micro service IP, a micro service Port and a micro service name, the registration request is a request sent to a registration center when the micro service needs to be enabled, the micro service IP is a unique Address corresponding to each micro service, i.e., an IP Address (Internet Protocol Address, Protocol Address) of the micro service, the micro service Port is a Protocol Port of each micro service, i.e., a Port (Port) through which the micro service provides communication, and the micro service name is a computer name given by the micro service.
And S20, registering the micro service into a service table of a registration center based on the registration information.
Understandably, the registry maintains the service table, the registry can manage all micro services which can be used or are used through the service table, the service table records all the micro services which can be served, the service table is divided into two layers, the first layer of the service table is used for recording micro service names, and the second layer of the service table is used for recording micro service IPs and micro service ports.
In an embodiment, before the step S20, that is, before the registering the microservice into the service table of the registry based on the registration information, the method includes:
and receiving a registration center starting request, and acquiring a configuration file in the registration center starting request.
Understandably, the registry start request is a request triggered by starting the operation or service of the registry, and the registry start request includes the configuration file, and the configuration file is used for configuring a relevant file of parameters for the registry.
And building the registration center corresponding to the configuration file based on a SpringCloud framework.
Understandably, the process of building the spring cloud framework is to build a spring boot project, sequentially run an environment dependent code file, a starting code file and an yml configuration file, and finally run the spring boot project, so as to build the registration center with the spring cloud framework.
The invention realizes that the configuration file in the registration center starting request is obtained by receiving the registration center starting request; and building the registration center corresponding to the configuration file based on the SpringCloud framework, so that the operation of the registration center can be facilitated by building the registration center of the SpringCloud framework, and the registration center is highly available.
And S30, performing elastic resource calculation on the history calling record associated with the micro service through the registration center, and building a standby server.
Understandably, the registry further records history call records associated with all micro services, the history call records are a record table of call logs in a preset history time period, the elastic resource calculation is to analyze the history call logs by applying a MapReduce method, so as to configure the micro services to draw an image, obtain a process of a standby configuration type corresponding to the micro services, obtain a configuration file corresponding to the standby configuration type, and construct an idle server according to the configuration file to obtain the standby server.
In an embodiment, the historical call records include a daily call record, a peak call record and a fault call record, and understandably, the daily call record is a record of the micro service being called every day, the peak call record is a time record of the number of times of calling in a unit second of the history being greater than a preset peak threshold, and the fault call record is a time record of the calling in a unit second of the history being faulty or not responding for a time greater than a preset time.
In an embodiment, as shown in fig. 3, in step S30, that is, performing, by the registry, elastic resource calculation on the history call record associated with the microservice, and building a standby server includes:
s301, call normalization processing is carried out on the daily call records by using a MapReduce method, and a plurality of time call records are obtained.
Understandably, the MapReduce method is a parallel computing model and method oriented to large-scale data processing, the process of calling the normalization processing is to apply the MapReduce method to carry out Map phase processing on the daily call record and divide the daily call record into a plurality of maps, namely, the calling records are divided into second level, the parallel reduce stage processing is carried out according to the Map per second, namely, the processing of duplicate removal and normalization, a plurality of call records with the same IP are merged into one, then the Map stage processing of minute level is carried out on each call record of second level, the call records of second level of the same minute are merged into one Map, the parallel reduce stage processing is carried out according to the Map of every minute, namely, the duration of the call records of the same IP is merged, and the processing of the Map stage and the reduce stage of the time level is carried out in the same way to obtain 24 call records of 24 hours a day, the time calling records are the number of times of calling the service in one hour and the duration of each IP calling.
In an embodiment, as shown in fig. 4, in step S301, that is, performing call normalization processing on the daily call record by using a MapReduce method, to obtain an immediate call record, includes:
s3011, performing second-level normalization processing on the daily call records to obtain a plurality of second call records.
Understandably, the second-level normalization processing is to divide the records in the daily call record by taking the second as a unit, perform duplicate removal on the same records in the same second after division, and only reserve one processing process, so as to obtain the second call record corresponding to each second.
In an embodiment, the performing second-level normalization processing on the daily call record to obtain a plurality of second call records includes:
and performing second-level division on the daily call record to obtain a plurality of second-level records.
Understandably, the second-level division divides the record in each second of the daily call records into a plurality of records in units of seconds, which are recorded as the second-level records.
And performing de-duplication and normalization processing on each second-level record to obtain a second calling record corresponding to each second-level record.
Understandably, the duplicate removal and aggregation processing is to perform duplicate removal on the same client or call records of the client in the records, only one record processing process is reserved, and the second-level record after the duplicate removal and aggregation processing is determined as the second call record.
The invention realizes the second-level division of the daily calling record to obtain a plurality of second-level records; and performing de-duplication normalization processing on each second-level record to obtain a second calling record corresponding to each second-level record, so that the influence of repeated invalid operations in the same second on the statistics of subsequent time length can be avoided, the time length statistics is not accurate, and the accuracy of outputting the second calling record is improved.
And S3012, performing hierarchical normalization processing on all the second call records to obtain a plurality of sub-call records.
Understandably, the hierarchical normalization process is to aggregate the second call records corresponding to all the second moments in the minute according to the unit of minute, combine the second call records of the same client or user side, add the time lengths according to the second level, and only reserve one operation process of the call records of the client or user side, thereby obtaining the sub-call records corresponding to each minute.
And S3013, performing time-level normalization processing on all the branch call records to obtain a plurality of the time-call records.
Understandably, the time-level normalization process is to aggregate the branch call records corresponding to all minute moments in the hour according to the hour as a unit, combine the branch call records of the same client or user side, add the time lengths according to the second level, convert the time lengths into a time length format of a minute level, and further only reserve an operation process of the call records of the client or user side, thereby obtaining the time call record corresponding to each hour.
The invention realizes that a plurality of second calling records are obtained by carrying out second-level normalization processing on the daily calling records; carrying out grading normalization processing on all the second calling records to obtain a plurality of sub-calling records; the branch call records are subjected to time-level normalization processing to obtain a plurality of time-level call records, so that massive daily call records can be subjected to parallel Map stage and reduce stage processing through second-level, grading and time-level normalization processing, the calculation workload is greatly reduced, and the processing efficiency of massive data processing is improved.
S302, daily operation drawing is conducted on all the time calling records to obtain a plurality of daily calling curve graphs, rule analysis is conducted on the peak value calling records to obtain a peak value rule graph, and fault calling duration analysis is conducted on the fault calling records to obtain a fault duration curve graph.
Understandably, the daily operation is drawn as a process of drawing curves of call times and call duration according to a horizontal axis of 24 hours a day through all the time call records, so as to obtain daily call graphs, the rule analysis is to find out a time rule and a call time rule of a peak value in the peak value call record, so as to obtain the peak value rule graph, the peak value rule graph reflects a predicted time rule of a future peak value and a frequency distribution of future calls, the fault call duration analysis is to count fault processing duration or non-response duration in the fault call record, so as to obtain the fault duration graph, and the fault duration graph reflects a graph of a fault starting rule and a rule of duration.
And S303, configuring the microservice to form an image according to the daily calling curve graph, the peak value rule graph and the fault duration curve graph, and obtaining a standby configuration type corresponding to the microservice.
Understandably, the configuration portrait is obtained by performing mean value aggregation on the daily calling curve graph, the peak value law graph and the fault duration curve graph, and classifying a machine learning model according to each aggregated mean value to obtain a standby configuration type suitable for the micro-service.
S304, building the standby server corresponding to the acquired standby configuration type.
Understandably, a server IP near to the micro-service IP is selected from an optional server list corresponding to the standby configuration type, an environment configuration file corresponding to the standby configuration type is obtained, and the environment configuration file is installed to the near server IP, so that the standby server is built.
The invention realizes that the daily call records are called and normalized by applying a MapReduce method to obtain a plurality of time call records; performing daily operation drawing on all the time calling records to obtain a plurality of daily calling curve graphs, performing rule analysis on the peak value calling records to obtain a peak value rule graph, and performing fault calling duration analysis on the fault calling records to obtain a fault duration curve graph; according to the daily calling curve graph, the peak value rule graph and the fault duration curve graph, carrying out configuration sketch on the micro service to obtain a standby configuration type corresponding to the micro service; and building the standby server corresponding to the obtained standby configuration type, so that the standby configuration type suitable for the micro service can be reasonably and scientifically and automatically identified, the suitable standby server is built, the standby server can be scientifically managed, and the effect of resource optimization is achieved.
And S40, based on the registration information and the standby server, carrying out snapshot processing on the micro service to obtain a mirror image micro service corresponding to the micro service.
Understandably, the snapshot processing process is a process of sending a notification instruction to the microservice, enabling the microservice to carry out full disk recovery, and carrying out snapshot processing on a full disk in the microservice through a snapshot generating tool after the full disk recovery to obtain the mirror image microservice.
The snapshot generating tool is a tool for performing physical copy or logical copy on data of all disks at the moment by using a snapshot technology.
In an embodiment, in step S40, that is, the snapshot processing is performed on the microservice based on the registration information and the standby server, so as to obtain a mirror microservice corresponding to the microservice, where the step S40 includes:
and sending a notification instruction to the micro service to enable the micro service to carry out full disk recovery.
Understandably, the notification command is a System command for calling a primary micro service, and notifies the garbage collector to perform a FULL GC process, where the FULL GC process is a process for collecting the entire heap, including all parts such as young gen, old gen, perm gen (if any), and recovering the garbage space.
And receiving a recovery finishing instruction returned by the micro service after the full disk recovery is executed, and performing snapshot processing on the full disk in the micro service through a snapshot generating tool to obtain the mirror image micro service.
Understandably, the process of obtaining the mirror image microservice is performed by performing snapshot processing on the whole disk in the microservice through a snapshot generating tool, wherein the snapshot generating tool is a tool for performing physical copy or logical copy on data of all disks at the moment by using a snapshot technology.
The invention realizes that the micro service is subjected to full disk recovery by sending the notification instruction to the micro service; and receiving a recovery finishing instruction returned by the micro service after the full disk recovery is executed, and performing snapshot processing on the full disk in the micro service through a snapshot generating tool to obtain the mirror image micro service.
And S50, when the registration center detects that the micro service calling time length is abnormal, starting the mirror image micro service corresponding to the micro service, and registering the mirror image micro service into the service table.
Understandably, the registry is responsible for managing all micro-service calls so as to record RPC call duration, when the current micro-service call duration is detected to be abnormal, the mirror micro-service corresponding to the micro-service is started, and the process of judging the abnormity of the micro-service call duration is to perform time interval analysis on a real-time call record corresponding to the micro-service through the registry record to obtain a first call attenuation value and a second call attenuation value; and performing response prediction on the first calling attenuation value and the second calling attenuation value to obtain whether an abnormal result exists.
The mirror image micro service starting process is to acquire memory data in the micro service through the snapshot generating tool; mapping the memory data by using an MMAP mapping mode to obtain mapping relation data; the mirror image micro service is stored in the mirror image micro service according to the current processing progress and the mapping relation data in the micro service, so that the mirror image micro service is started and registered in the registration center, the monitoring service of a container is not needed, the management is carried out through the self record of the registration center, the middleware such as the container and the like is omitted, the system resources are saved, the service running speed is improved, the service is directly recovered from the mirror image, and the starting speed is higher.
In an embodiment, before the step S50, that is, before the registry detects that there is an abnormality in the micro service invocation duration, the method includes:
and analyzing the real-time calling record corresponding to the micro service in time intervals through the registration center record to obtain a first calling attenuation value and a second calling attenuation value.
Understandably, the first call attenuation value is a call attenuation average of up to one hour and the second call attenuation value is a call attenuation average of up to five minutes.
And performing response prediction on the first calling attenuation value and the second calling attenuation value to obtain a prediction result.
Understandably, the response prediction is a process of predicting a probability value of the second call attenuation value greater than 80% of the first call attenuation value in the future, so as to obtain the prediction result, the prediction result represents whether the second call attenuation value is an abnormal result, the prediction result includes a response rate, and the response rate is the number of responses per unit time of the current micro service.
And if the prediction result is abnormal, determining that the micro-service calling duration is abnormal.
And if the prediction result is normal, throttling the call of the microservice through the registration center according to the response rate in the prediction result.
Understandably, the throttling process is a process of reducing the proportion of the traffic proportionally according to the response rate.
The invention realizes the time interval analysis of the real-time calling record corresponding to the micro-service through the registration center record to obtain a first calling attenuation value and a second calling attenuation value; performing response prediction on the first calling attenuation value and the second calling attenuation value to obtain a prediction result; if the prediction result is abnormal, determining that the micro-service calling duration is abnormal; and if the prediction result is normal, throttling the micro service calling through the registration center according to the response rate in the prediction result, so that whether the micro service calling is abnormal or not is automatically judged through the registration center without judging an intermediate container, system resources are saved, and the service running speed is increased.
In an embodiment, as shown in fig. 5, the step S50, that is, the starting the mirror microservice corresponding to the microservice and registering the mirror microservice in the service table includes:
s501, obtaining memory data in the microservice through the snapshot generating tool.
Understandably, the previous FULL disk reclamation processing (FULL GC processing) makes all current program counters stay in a safe area or a safe point, no memory address offset occurs at this time, and the memory data of the whole micro service process is acquired by the snapshot generating tool.
And S502, mapping the memory data by using an MMAP mapping mode to obtain mapping relation data.
Understandably, the memory data is mapped by using an MMAP memory mapping file mode, and one file or other objects are mapped to the address space of the process by the MMAP mapping mode, so that the one-to-one correspondence relationship between the file disk address and a section of virtual address in the virtual address space of the process is realized, and the mapping relationship data is obtained.
S503, generating a mirror image plaintext based on the JVM process data and the mapping relation data in the micro service, and storing the mirror image plaintext into the mirror image micro service.
Understandably, the binary data in the memory space of the JVM process is read through the address pointer stored in the mapping relation data, converted into the 16-system plaintext, and the converted plaintext is stored in the mirror plaintext (using Unicode encoding).
S504, the JVM process of the mirror image micro service is started, the started mirror image micro service is registered through the registry, and the service table is updated.
Understandably, the JVM process of the mirror image micro service is started, a system kernel malloc function is called to carry out memory allocation, suspended processes which are mirrored from all the mirror image micro service JVM processes are recovered, a registration center is called, and information (mirror image micro service IP, a mirror image micro service port and a mirror image micro service name) of the current service is registered in the registration center.
The invention realizes that the memory data in the microservice is obtained through the snapshot generating tool; mapping the memory data by using an MMAP mapping mode to obtain mapping relation data; generating a mirror image plaintext based on JVM process data and the mapping relation data in the microservice, and storing the mirror image plaintext into the mirror image microservice; the JVM process of the mirror image micro service is started, the started mirror image micro service is registered through the registry, the service table is updated, therefore, the JVM process in the micro service is automatically mirrored to the JVM process in the mirror image micro service by using a snapshot generating tool and an MMAP mapping mode, the memory data in the micro service is also stored in the mirror image micro service, the mirror image micro service can be quickly started to recover the JVM process suspended in the micro service in the mirror image micro service, the memory data is mirrored to the mirror image micro service, the service is quickly recovered, and the service operation speed is improved.
And S60, distributing the micro service and the calling of the mirror image micro service through the registry by using a shunting mechanism.
Understandably, the shunting mechanism is to use a balance strategy, distribute the number of calling processes between the microservice and the mirror microservice through a registry, so that the microservice and the mirror microservice are commonly called for service, the service can be smoother, the condition of long calling time does not exist, and the image service experience is further realized.
The invention realizes that the registration information in the registration request is obtained by receiving the registration request of the micro service; registering the micro-service into a service table of a registration center based on the registration information; elastic resource calculation is carried out on the historical call records associated with the micro-service through the registration center, and a standby server is built; based on the registration information and the standby server, carrying out snapshot processing on the micro-service to obtain a mirror image micro-service corresponding to the micro-service; when the registration center detects that the micro-service calling duration is abnormal, starting the mirror micro-service corresponding to the micro-service, and registering the mirror micro-service into the service table; the method and the system have the advantages that the shunting mechanism is used, the micro service and the mirror image micro service are distributed through the registration center, so that the micro service is registered through the registration center, the standby server is calculated and established through the elastic resources, the mirror image micro service is obtained through snapshot processing, when the micro service calling abnormity is detected, the mirror image micro service is automatically started, the shunting mechanism is used for automatically providing service together with the micro service, therefore, monitoring service without a container is achieved, only the calling time length is recorded through the registration center for management, the middlewares such as the container and the like are omitted, system resources are saved, the service operation speed is increased, the service is directly recovered from the mirror image, and the starting speed is higher.
It should be understood that, the sequence numbers of the steps in the foregoing embodiments do not imply an execution sequence, and the execution sequence of each process should be determined by its function and inherent logic, and should not constitute any limitation to the implementation process of the embodiments of the present invention.
In an embodiment, a micro service data processing apparatus is provided, and the micro service data processing apparatus corresponds to the micro service data processing methods in the above embodiments one to one. As shown in fig. 6, the microservice data processing apparatus includes a receiving module 11, a registering module 12, a building module 13, a snapshot module 14, a starting module 15, and a calling module 16. The functional modules are explained in detail as follows:
a receiving module 11, configured to receive a registration request of a microservice, and acquire registration information in the registration request;
a registration module 12, configured to register the microservice into a service table of a registry based on the registration information;
the building module 13 is used for performing elastic resource calculation on the historical calling record associated with the micro-service through the registration center and building a standby server;
a snapshot module 14, configured to perform snapshot processing on the micro service based on the registration information and the standby server, so as to obtain a mirror image micro service corresponding to the micro service;
a starting module 15, configured to start the mirror image microservice corresponding to the microservice when the registry detects that the microservice calling duration is abnormal, and register the mirror image microservice in the service table;
and the calling module 16 is configured to distribute the calling of the microservice and the mirror microservice through the registry by using a offloading mechanism.
For specific limitations of the micro service data processing apparatus, reference may be made to the above limitations of the micro service data processing method, which is not described herein again. The modules in the microservice data processing apparatus described above may be implemented in whole or in part by software, hardware, and combinations thereof. The modules can be embedded in a hardware form or independent from a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.
In one embodiment, a computer device is provided, which may be a client or a server, and its internal structure diagram may be as shown in fig. 7. The computer device includes a processor, a memory, a network interface, and a database connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a readable storage medium and an internal memory. The readable storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of an operating system and computer programs in the readable storage medium. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement a microservice data processing method.
In one embodiment, a computer device is provided, which includes a memory, a processor, and a computer program stored on the memory and executable on the processor, and when the processor executes the computer program, the micro service data processing method in the above embodiments is implemented.
In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the microservice data processing method in the above-described embodiments.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. Any reference to memory, storage, databases, or other media used in embodiments provided herein may include non-volatile and/or volatile memory. Non-volatile memory can include read-only memory (ROM), Programmable ROM (PROM), Electrically Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus Direct RAM (RDRAM), direct bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
It will be apparent to those skilled in the art that, for convenience and brevity of description, only the above-mentioned division of the functional units and modules is illustrated, and in practical applications, the above-mentioned function distribution may be performed by different functional units and modules according to needs, that is, the internal structure of the apparatus is divided into different functional units or modules to perform all or part of the above-mentioned functions.
The above-mentioned embodiments are only used for illustrating the technical solutions of the present invention, and not for limiting the same; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; such modifications and substitutions do not substantially depart from the spirit and scope of the embodiments of the present invention, and are intended to be included within the scope of the present invention.
Claims (10)
1. A method for processing microservice data, comprising:
receiving a registration request of a micro service, and acquiring registration information in the registration request;
registering the micro-service into a service table of a registration center based on the registration information;
elastic resource calculation is carried out on the historical call records associated with the micro-service through the registration center, and a standby server is built;
based on the registration information and the standby server, carrying out snapshot processing on the micro-service to obtain a mirror image micro-service corresponding to the micro-service;
when the registration center detects that the micro-service calling duration is abnormal, starting the mirror micro-service corresponding to the micro-service, and registering the mirror micro-service into the service table;
and distributing the micro-service and the calling of the mirror image micro-service through the registration center by using a shunting mechanism.
2. The micro-service data processing method of claim 1, wherein the historical call records include daily call records, peak call records, and fault call records;
through the registration center, elastic resource calculation is carried out on the historical call records associated with the micro-service, and a standby server is built, and the method comprises the following steps:
calling and normalizing the daily calling records by using a MapReduce method to obtain a plurality of time calling records;
performing daily operation drawing on all the time calling records to obtain a plurality of daily calling curve graphs, performing rule analysis on the peak value calling records to obtain a peak value rule graph, and performing fault calling duration analysis on the fault calling records to obtain a fault duration curve graph;
according to the daily calling curve graph, the peak value rule graph and the fault duration curve graph, carrying out configuration sketch on the micro service to obtain a standby configuration type corresponding to the micro service;
and building the standby server corresponding to the obtained standby configuration type.
3. The method for processing microservice data according to claim 2, wherein the performing call normalization processing on the daily call record by using a MapReduce method to obtain an immediate call record comprises:
carrying out second-level normalization processing on the daily call records to obtain a plurality of second call records;
carrying out grading normalization processing on all the second calling records to obtain a plurality of sub-calling records;
and performing time-level normalization processing on all the branch call records to obtain a plurality of time-call records.
4. The micro-service data processing method of claim 3, wherein prior to registering the micro-service into a service table of a registry based on the registration information, comprising:
receiving a registration center starting request, and acquiring a configuration file in the registration center starting request;
and building the registration center corresponding to the configuration file based on a SpringCloud framework.
5. The method for processing microservice data according to claim 1, wherein the snapshot processing of the microservice based on the registration information and the backup server to obtain a mirror microservice corresponding to the microservice comprises:
sending a notification instruction to the micro service to enable the micro service to carry out full disk recovery;
and receiving a recovery finishing instruction returned by the micro service after the full disk recovery is executed, and performing snapshot processing on the full disk in the micro service through a snapshot generating tool to obtain the mirror image micro service.
6. The micro-service data processing method of claim 1, wherein before the registry detects that there is an exception in the micro-service invocation duration, the method comprises:
analyzing the real-time calling record corresponding to the micro service in time intervals through the registration center record to obtain a first calling attenuation value and a second calling attenuation value;
performing response prediction on the first calling attenuation value and the second calling attenuation value to obtain a prediction result;
if the prediction result is abnormal, determining that the micro-service calling duration is abnormal;
and if the prediction result is normal, throttling the call of the microservice through the registration center according to the response rate in the prediction result.
7. The micro-service data processing method of claim 1, wherein the starting the mirror micro-service corresponding to the micro-service and registering the mirror micro-service in the service table comprises:
acquiring memory data in the microservice through the snapshot generating tool;
mapping the memory data by using an MMAP mapping mode to obtain mapping relation data;
generating a mirror image plaintext based on JVM process data and the mapping relation data in the microservice, and storing the mirror image plaintext into the mirror image microservice;
and starting the JVM process of the mirror image micro service, registering the started mirror image micro service through the registry, and updating the service table.
8. A microservice data processing apparatus, comprising:
the system comprises a receiving module, a registration module and a processing module, wherein the receiving module is used for receiving a registration request of the microservice and acquiring registration information in the registration request;
the registration module is used for registering the microservice into a service table of a registration center based on the registration information;
the building module is used for performing elastic resource calculation on the historical calling record associated with the micro-service through the registration center and building a standby server;
the snapshot module is used for carrying out snapshot processing on the micro-service based on the registration information and the standby server to obtain mirror image micro-service corresponding to the micro-service;
the starting module is used for starting the mirror image microservice corresponding to the microservice and registering the mirror image microservice into the service table when the registry detects that the microservice calling time length is abnormal;
and the calling module is used for distributing the calling of the micro-service and the mirror image micro-service through the registration center by using a shunting mechanism.
9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that the processor implements the microservice data processing method according to any of claims 1 to 7 when executing the computer program.
10. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out a microservice data processing method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111216620.5A CN113946491A (en) | 2021-10-19 | 2021-10-19 | Microservice data processing method, microservice data processing device, computer equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111216620.5A CN113946491A (en) | 2021-10-19 | 2021-10-19 | Microservice data processing method, microservice data processing device, computer equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113946491A true CN113946491A (en) | 2022-01-18 |
Family
ID=79331369
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111216620.5A Pending CN113946491A (en) | 2021-10-19 | 2021-10-19 | Microservice data processing method, microservice data processing device, computer equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113946491A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115033477A (en) * | 2022-06-08 | 2022-09-09 | 山东省计算中心（国家超级计算济南中心） | Large-scale micro-service-oriented active performance anomaly detection and processing method and system |
| CN115242863A (en) * | 2022-04-20 | 2022-10-25 | 广东数夫软件有限公司 | Micro-service application method and micro-service architecture system |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20170242784A1 (en) * | 2016-02-19 | 2017-08-24 | International Business Machines Corporation | Failure recovery testing framework for microservice-based applications |
| US20190028496A1 (en) * | 2017-07-19 | 2019-01-24 | Cisco Technology, Inc. | Anomaly detection for micro-service communications |
| US10684940B1 (en) * | 2018-09-18 | 2020-06-16 | Amazon Technologies, Inc. | Microservice failure modeling and testing |
| CN111615066A (en) * | 2020-02-07 | 2020-09-01 | 中国海洋大学 | Distributed micro-service registration and calling method based on broadcast |
| CN112540838A (en) * | 2020-12-18 | 2021-03-23 | 南京中兴力维软件有限公司 | System resource scheduling method and device and computer storage medium |
| CN112788112A (en) * | 2020-12-30 | 2021-05-11 | 北京航天测控技术有限公司 | Automatic publishing method, device and platform for equipment health management micro-service |
-
2021
- 2021-10-19 CN CN202111216620.5A patent/CN113946491A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20170242784A1 (en) * | 2016-02-19 | 2017-08-24 | International Business Machines Corporation | Failure recovery testing framework for microservice-based applications |
| US20190028496A1 (en) * | 2017-07-19 | 2019-01-24 | Cisco Technology, Inc. | Anomaly detection for micro-service communications |
| US10684940B1 (en) * | 2018-09-18 | 2020-06-16 | Amazon Technologies, Inc. | Microservice failure modeling and testing |
| CN111615066A (en) * | 2020-02-07 | 2020-09-01 | 中国海洋大学 | Distributed micro-service registration and calling method based on broadcast |
| CN112540838A (en) * | 2020-12-18 | 2021-03-23 | 南京中兴力维软件有限公司 | System resource scheduling method and device and computer storage medium |
| CN112788112A (en) * | 2020-12-30 | 2021-05-11 | 北京航天测控技术有限公司 | Automatic publishing method, device and platform for equipment health management micro-service |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115242863A (en) * | 2022-04-20 | 2022-10-25 | 广东数夫软件有限公司 | Micro-service application method and micro-service architecture system |
| CN115242863B (en) * | 2022-04-20 | 2024-03-26 | 广东数夫软件有限公司 | Micro-service application method and micro-service architecture system |
| CN115033477A (en) * | 2022-06-08 | 2022-09-09 | 山东省计算中心（国家超级计算济南中心） | Large-scale micro-service-oriented active performance anomaly detection and processing method and system |
| CN115033477B (en) * | 2022-06-08 | 2023-06-27 | 山东省计算中心（国家超级计算济南中心） | Performance abnormality active detection and processing method and system for large-scale micro-service |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10776174B2 (en) | Managing hosted resources across different virtualization platforms | |
| US11550630B2 (en) | Monitoring and automatic scaling of data volumes | |
| US10713080B1 (en) | Request-based virtual machine memory transitioning in an on-demand network code execution system | |
| US10795905B2 (en) | Data stream ingestion and persistence techniques | |
| US10691716B2 (en) | Dynamic partitioning techniques for data streams | |
| CN111386516B (en) | Automatically scaling hosted machine learning models to make generative inferences | |
| JP6381801B2 (en) | Dynamic scaling of storage volume for storage client file system | |
| US10635644B2 (en) | Partition-based data stream processing framework | |
| US9276959B2 (en) | Client-configurable security options for data streams | |
| US10754741B1 (en) | Event-driven replication for migrating computing resources | |
| US20150134797A1 (en) | Managed service for acquisition, storage and consumption of large-scale data streams | |
| CN112910945A (en) | Request link tracking method and service request processing method | |
| US9535754B1 (en) | Dynamic provisioning of computing resources | |
| CN105025095A (en) | Cluster framework capable of realizing cloud computing flexible service | |
| TWI725744B (en) | Method for establishing system resource prediction and resource management model through multi-layer correlations | |
| CN113946491A (en) | Microservice data processing method, microservice data processing device, computer equipment and storage medium | |
| CN109614227A (en) | Task resource concocting method, device, electronic equipment and computer-readable medium | |
| US20220358106A1 (en) | Proxy-based database scaling | |
| US20180150365A1 (en) | Disaster Recover of Managed Systems | |
| CN111464589A (en) | Intelligent contract processing method, computer equipment and storage medium | |
| CN113762906A (en) | Method, device and equipment for alarming task period delay and storage medium | |
| CN111949442B (en) | System and method for scalable backup service | |
| CN112631994A (en) | Data migration method and system | |
| US20220164228A1 (en) | Fine-grained virtualization resource provisioning for in-place database scaling | |
| WO2022121387A1 (en) | Data storage method and apparatus, server, and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |