CN112100152A

CN112100152A - Service data processing method, system, server and readable storage medium

Info

Publication number: CN112100152A
Application number: CN202010959886.8A
Authority: CN
Inventors: 陈智铭; 高翔; 郭荣洁; 林锦培
Original assignee: Guangzhou Huaduo Network Technology Co Ltd
Current assignee: Guangzhou Overseas shoulder sub network technology Co.,Ltd.
Priority date: 2020-09-14
Filing date: 2020-09-14
Publication date: 2020-12-18

Abstract

The application relates to a business data processing method, a business data processing system, a server and a readable storage medium. The method comprises the following steps: the index server acquires original incremental business data and performs data fragmentation on the original incremental business data to obtain incremental fragmentation business data; the index server processes the incremental fragment service data to obtain incremental service data with a preset data structure, and stores the incremental service data in a distributed file system (HDFS); the index server updates the data version information in the target server based on the incremental business data and sends a data version updating notice to the target server; and the target server pulls the incremental business data corresponding to the updated data version information from the HDFS according to the updated data version information, and loads the incremental business data to the memory for the client to access. By adopting the method, the automation capability and the expandability of the service data processing system can be improved, and the waste of memory resources can be reduced.

Description

Service data processing method, system, server and readable storage medium

Technical Field

The present application relates to the field of computer technologies, and in particular, to a method, a system, a server, and a readable storage medium for processing service data.

Background

With the continuous development of services, more and more service data are required to be processed by a service data processing system. In an existing business data processing system, business data of an offline data source needs to be manually imported into a storage engine such as a Remote Dictionary server (Redis) to provide online services.

Taking Redis as an example, Redis is a memory DataBase based on a memory, data of Redis is stored in the memory in a centralized manner, and in order to avoid data loss caused by sudden downtime, RDB (Redis DataBase, a persistence mode default to Redis) can copy the data to the memory periodically, and then persist the copied data.

However, the automation capability of the business data processing system needs to be improved, the expandability of the memory database is poor, and the persistence of the Redis storage engine also causes the waste of memory resources, which is not favorable for the business data processing system to provide stable online services.

Disclosure of Invention

In view of the foregoing, it is desirable to provide a service data processing method, a system, a server and a readable storage medium, which can improve the automation capability and the expandability of a service data processing system and reduce the waste of memory resources.

In a first aspect, a service data processing method is provided, which is applied to a service data processing system, where the service data processing system includes an index server and multiple service servers, and the method includes:

the index server acquires original incremental business data and performs data fragmentation on the original incremental business data to obtain incremental fragmentation business data;

the index server processes the incremental fragment service data to obtain incremental service data with a preset data structure, and stores the incremental service data in a distributed file system (HDFS);

the index server updates data version information in a target server based on the incremental business data and sends a data version updating notice to the target server, wherein the target server is one of the business servers;

and the target server pulls the incremental business data corresponding to the updated data version information from the HDFS according to the updated data version information, and loads the incremental business data to a memory for a client to access.

In one embodiment, the method further comprises:

the index server generates first data basic information according to the updated data version information of the incremental business data and a first storage address of the incremental business data in the HDFS;

correspondingly, the target server pulls the incremental service data corresponding to the updated data version information from the HDFS according to the updated data version information, including:

the target server acquires the first data basic information from the index server according to the updated data version information;

and the target server pulls the incremental business data corresponding to the updated data version information from the HDFS storage position indicated by the first storage address according to the first storage address included in the first data basic information.

In one embodiment, the processing, by the index server, the incremental fragment service data to obtain incremental service data with a preset data structure includes:

and the index server processes the incremental fragment service data by adopting a zipper method to obtain the incremental service data with a preset data structure.

In one embodiment, the service data processing system further includes a data source server, and the method further includes:

the data source server acquires the original incremental business data, stores the original incremental business data to the HDFS, and sends second data basic information of the original incremental business data to the index server, wherein the second data basic information at least comprises a second storage address of the original incremental business data in the HDFS;

correspondingly, the index server acquires original incremental service data, including:

and the index server pulls the original incremental service data from the HDFS storage position indicated by the second storage address according to the second storage address included in the second data basic information.

In one embodiment, the method further comprises:

the index server analyzes the incremental fragment service data according to a preset data format to obtain the analyzed incremental fragment service data;

correspondingly, the index server processes the incremental fragment service data to obtain incremental service data with a preset data structure, including:

and the index server processes the analyzed incremental fragment service data to obtain the incremental service data with a preset data structure.

In one embodiment, the method further comprises:

the target server acquires full data version information, and pulls full service data corresponding to the full data version information from the HDFS according to the full data version information, wherein the full data version information is updated in the target server by the index server based on the full service data;

and the target server loads the full service data to a memory for a client to access.

In one embodiment, the method further comprises:

if the target server detects a restart trigger event, pulling the incremental service data from the HDFS according to the updated data version information, and pulling the full service data from the HDFS according to the full data version information;

and the target server loads the pulled incremental business data and the full business data to a memory for a client to access.

In one embodiment, the method further comprises:

the index server acquires full-volume fragment service data from the HDFS, wherein the full-volume fragment service data is obtained by performing data fragmentation on original full-volume service data;

the index server processes the full-scale fragment service data by adopting an open addressing method to generate the full-scale service data with a preset data structure;

the index server stores the full service data to the HDFS, and generates third data basic information according to full data version information of the full service data and a third storage address;

correspondingly, the target server pulls the full service data corresponding to the full data version information from the HDFS according to the full data version information, and the method includes:

the target server acquires the third data basic information from the index server according to the full data version information;

and the target server pulls the full service data corresponding to the full data version information from the HDFS storage location indicated by the third storage address according to the third storage address included in the third data basic information.

In one embodiment, the method further comprises:

the index server acquires fragmentation mode information of the original full service data and detects whether the fragmentation mode information is the same as local fragmentation mode information, wherein the local fragmentation mode information is related to the number of the plurality of service servers;

if the fragmentation mode information is the same as the local fragmentation mode information, the index server executes the step that the index server acquires full fragmentation service data from the HDFS;

and if the fragmentation mode information is different from the local fragmentation mode information, the index server performs data fragmentation on the original full volume service data according to the local fragmentation mode information to obtain the full volume fragmentation service data, and executes the step that the index server acquires the full volume fragmentation service data from the HDFS.

In one embodiment, the target server includes a first fragmented data copy and a second fragmented data copy, where the first fragmented data copy and the second fragmented data copy are used to provide the same data access service, and the target server loads the incremental business data to the memory, including:

the target server updates the data of the first fragment data copy according to the incremental business data, and in the process of updating the data of the first fragment data copy, the target server provides data access service through the second fragment data copy and forbids the first fragment data copy in the process of updating the data to provide the data access service;

and after the data of the first fragment data copy is updated, the target server updates the data of the second fragment data copy according to the incremental service data, and in the process of updating the data of the second fragment data copy, the target server provides data access service through the first fragment data copy after data updating and forbids the second fragment data copy in the process of updating the data to provide the data access service.

In one embodiment, the method further comprises:

if the index server receives a data address acquisition request sent by the client, the index server determines the service address of the target server according to the data address acquisition request;

the index server returns the service address to the client;

the client sends an access request aiming at the incremental business data to the target server according to the service address;

and the target server receives the access request and responds to the access request based on the incremental business data loaded in the memory.

In one embodiment, the determining, by the index server, the service address of the target server according to the data address obtaining request includes:

the index server analyzes the data address acquisition request to obtain a key code corresponding to the data address acquisition request;

and the index server determines the service address of the target server corresponding to the key code from the plurality of service servers according to the key code.

In a second aspect, a service data processing system is provided, where the service data processing system includes an index server and a plurality of service servers, and a target server is one of the plurality of service servers;

the index server is used for acquiring original incremental business data, performing data slicing on the original incremental business data to obtain incremental slicing business data, processing the incremental slicing business data to obtain incremental business data with a preset data structure, and storing the incremental business data in a distributed file system (HDFS);

the index server is further used for updating data version information in a target server based on the incremental business data and sending a data version updating notice to the target server;

and the target server is used for pulling the incremental business data corresponding to the updated data version information from the HDFS according to the updated data version information and loading the incremental business data to a memory for a client to access.

In a third aspect, a server is provided, which includes a memory and a processor, where the memory stores a computer program, and the processor implements the service data processing method according to the first aspect when executing the computer program.

In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored, which, when being executed by a processor, is adapted to perform the method of processing service data according to the first aspect.

The beneficial effects brought by the technical scheme provided by the embodiment of the application at least comprise:

in the service data processing system of the embodiment of the application, the index server performs data fragmentation on original incremental service data to obtain incremental fragmentation service data, the index server processes the incremental fragmentation service data to obtain incremental service data with a preset data structure, and stores the incremental service data in the HDFS, so that automatic storage and updating of the incremental service data are realized, and distributed storage on the HDFS is performed after the fragmented incremental fragmentation service data is processed through data fragmentation, so that distributed management on the original incremental service data can be realized, even if the service data is increased rapidly, distributed management on the service data can be realized through the data fragmentation, and the expandability of the service data processing system is enhanced; then, the index server updates the data version information in the target server based on the incremental business data, and sends a data version update notification to the target server, the target server pulls incremental service data corresponding to the updated data version information from the HDFS according to the updated data version information, and loading the incremental service data into a memory for the client to access, so that the target server directly and automatically pulls the incremental service data from the HDFS according to the updated data version information, and loads the pulled incremental business data into the memory for the client to access, thereby realizing the automatic loading of the incremental business data, improving the automation capability of the business data processing system, therefore, the problem that in the traditional technology, the service data needs to be manually imported into Redis in the memory to provide online access, so that the automation capability of the service data processing system is poor is solved. In addition, in the embodiment of the application, the incremental service data is stored in the HDFS, the target server automatically pulls the incremental service data and loads the incremental service data to the memory, and the incremental service data does not need to be manually guided into the Redis, so that the RDB based on the Redis does not need to be used for realizing persistence, thereby avoiding the memory waste caused by the RDB needing to copy the service data in the memory to the memory and then persistence the copied data, and further reducing the waste of memory resources of the target server, thereby being beneficial to providing stable online service.

Drawings

FIG. 1 is a diagram of an application environment of a method for processing service data in one embodiment;

FIG. 2 is a flow diagram of a method for business data processing in one embodiment;

FIG. 3 is a diagram of an exemplary service data fragmentation process;

FIG. 4 is a flow diagram of an index server for service discovery in one embodiment;

FIG. 5 is a diagram of an exemplary index server for service discovery;

FIG. 6 is a flow diagram of a method for business data processing in one embodiment;

FIG. 7 is a flow diagram of a method for business data processing in one embodiment;

FIG. 8 is a flow diagram of index server pooling in one embodiment;

FIG. 9 is a flow diagram that illustrates the warehousing of the index server during service scale-up and expansion in one embodiment;

FIG. 10 is a flowchart illustrating loading of incremental business data into memory by a target server according to an embodiment;

FIG. 11 is a pictorial diagram of exemplary sliced data;

FIG. 12 is a flow diagram of a method for business data processing in one embodiment;

FIG. 13 is a schematic diagram of an exemplary component-based business data processing system;

fig. 14 is an internal configuration diagram of a server in one embodiment.

Detailed Description

In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.

In the following, a brief description will be given of an implementation environment related to the service data processing method provided in the embodiment of the present application.

The implementation environment includes a business data processing System, which may include a data source server 101, an index server 102, a plurality of business servers, and an HDFS (Hadoop Distributed File System) 104 as shown in fig. 1, it should be noted that fig. 1 only exemplarily shows a target server 103, and the target server 103 is one of the plurality of business servers.

The data source server 101 may communicate with the index server 102 and the HDFS104 via a network, the index server 102 may communicate with the data source server 101, the target server 103, the HDFS104, and the client 105 via a network, and the target server 103 may communicate with the index server 102, the HDFS104, and the client 105 via a network.

The data source server 101, the index server 102, and the target server 103 may be tower servers, rack servers, blade servers, high-density servers, single-path servers, two-path servers, or multi-path servers, and the client 105 may be a personal computer, a notebook computer, a media player, a smart television, a smart phone, a tablet computer, a portable wearable device, and the like, which is not specifically limited in this embodiment of the present application.

In an embodiment, as shown in fig. 2, a service data processing method is provided, which is described by taking an example that the method is applied to the service data processing system shown in fig. 1, the service data processing system may include an index server and a plurality of service servers, and the target server is one of the plurality of service servers. The method comprises the following steps of 201, 202, 203 and 204:

step 201, the index server obtains original incremental business data, and performs data slicing on the original incremental business data to obtain incremental slicing business data.

In a service data processing System, a large amount of off-line service data exist, an index server performs corresponding processing on the service data, then a library is built for the processed service data, and the processed service data is stored in an HDFS (Hadoop Distributed File System) in a Distributed manner. The target server can directly pull the processed service data from the HDFS and load the service data into a memory for the client to access.

With the lapse of time, the business data processing system will continuously generate new business data, i.e. original incremental business data, the index server obtains the original incremental business data, and performs data slicing on the original incremental business data to obtain incremental sliced business data. In a possible implementation manner, the index server may divide the original incremental service data into a plurality of fragments through a map-reduce task, so that each fragment may be stored in the HDFS in a distributed manner after being processed by the index server, thereby implementing distributed management of the original incremental service data.

Referring to fig. 3, fig. 3 shows an exemplary traffic data fragmentation process. As shown in fig. 3, the index server divides the original incremental service data into two segments by a map-reduce task, where each segment is an incremental segment service data.

Step 202, the index server processes the incremental fragment service data to obtain incremental service data with a preset data structure, and stores the incremental service data in the distributed file system HDFS.

In the embodiment of the application, the index server processes and stores the incremental fragment service data, and processes the incremental fragment service data to obtain the incremental service data with a preset data structure, wherein the preset data structure can be a hash table, for example, so that the quick search of the incremental service data is facilitated.

In a possible implementation manner of step 202, the index server may process the incremental fragment service data by using a zipper method to obtain incremental service data with a preset data structure.

The zipper method is also called a chain address method, and the zipper method is to place key word (synonym) values with the same hash address in the same single linked list (synonym linked list). And the index server performs Hash processing on the incremental fragment service data by adopting a zipper method to generate incremental service data corresponding to the incremental fragment service data. The incremental business data generated by the zipper method supports change insertion, so that automatic data updating of the incremental business data can be realized.

The index server processes the incremental fragment service data to obtain incremental service data with a preset data structure, and then stores the incremental service data in the distributed file system HDFS, so that automatic storage and updating of the incremental service data are realized, and the processed incremental fragment service data are stored in the HDFS in a distributed manner through data fragmentation, so that distributed management of the original incremental service data can be realized, even if the service data are increased rapidly, the distributed management of the service data can be realized through the data fragmentation, and the expandability of the service data processing system is enhanced.

Step 203, the index server updates the data version information in the target server based on the incremental business data and sends a data version update notification to the target server.

The target server is one of a plurality of business servers.

After the index server stores the incremental business data in the HDFS, the index server updates the data version information of the incremental business data to the target server and sends a data version update notification to the target server to prompt the target server to pull the incremental business data from the HDFS according to the data version information of the incremental business data.

In this embodiment of the application, the data version information may represent version information of the service data, and for example, the data version information of the incremental service data may be a version number of the incremental service data.

And step 204, the target server pulls the incremental business data corresponding to the updated data version information from the HDFS according to the updated data version information, and loads the incremental business data to a memory for the client to access.

After receiving the data version update notification sent by the index server, the target server pulls the incremental service data corresponding to the updated data version information from the HDFS according to the updated data version information (i.e., the data version information of the incremental service data updated to the target server by the index server).

In a possible implementation manner, after the index server stores the incremental service data in the HDFS, the index server may further generate first data basic information according to updated data version information of the incremental service data and a first storage address of the incremental service data in the HDFS, where the updated data version information is used by the target server to obtain the first data basic information according to the updated data version information, and the first data basic information is used by the target server to pull the incremental service data according to the storage location of the HDFS indicated by the first storage address. The target server may implement the process of step 204 by performing step A11 and step A12 as follows:

step A11, the target server obtains the first data basic information from the index server according to the updated data version information.

Step A12, the target server pulls the incremental business data corresponding to the updated data version information from the HDFS storage location indicated by the first storage address according to the first storage address included in the first data basic information.

In this way, when the target server pulls the incremental service data corresponding to the updated data version information from the HDFS according to the updated data version information, the target server may obtain the first data base information from the index server according to the updated data version information, and then pull the incremental service data corresponding to the updated data version information from the HDFS storage location indicated by the first storage address included in the first data base information.

After the target server pulls the incremental business data from the HDFS, the incremental business data are automatically loaded to the memory of the target server, so that the automatic loading of the incremental business data is realized, and the automation capability of the business data processing system is improved.

If the target server receives an access request aiming at the incremental business data sent by the client, the target server responds to the data access request based on the incremental business data in the memory, and the client accesses the incremental business data loaded in the memory, so that the offline business data is automatically converted into the online service, and the business data processing efficiency is improved.

Optionally, the target server in the embodiment of the present application may be deployed in a container cluster such as kubernets. Taking kubernets as an example, a target server serves as a service node in the kubernets, data access service is provided for a client by starting a container deployed in a pod in the target server, and the container can provide services such as reading and writing of incremental business data for the client when the container runs.

Optionally, the index server may also be deployed in kubernets, and the index server performs service discovery through a consul, which is an open source tool developed based on the GO language and mainly provides functions of service registration, service discovery, and configuration management for a distributed and servitized system.

In one possible implementation, referring to fig. 4, the process of the index server performing service discovery through consul may include

steps

401, 402, 403, and 404 shown in fig. 4:

in step 401, if the index server receives a data address acquisition request sent by the client, the index server determines a service address of the target server according to the data address acquisition request.

The client sends a data address acquisition request to the index server to acquire the address of the accessible service server, and the index server determines a target server for providing service for the client from the plurality of service servers through consul.

Referring to fig. 5, fig. 5 is a diagram illustrating service discovery performed by an exemplary index server. As shown in fig. 5, servers may be service servers, and a consul periodically performs health check on each server, detects whether each server operates normally, adds a server that operates normally to a server list, and removes a server that does not operate normally from the server list, so as to ensure that a service server in the server list can provide services normally.

The index server obtains the service address of the server which normally runs from the consul, and the service address can be an IP: in the Port form, the index server can establish the mapping relationship between each key code and each service address, different key codes correspond to different service addresses, and the key codes can be in the form of character strings.

In a possible implementation manner, the index server may analyze the data address acquisition request to obtain a key code corresponding to the data address acquisition request, and the index server determines a service address of a target server corresponding to the key code from the plurality of service servers according to the key code and the established mapping relationship, so as to implement a process of determining the target server for providing a service to the client from the plurality of service servers by consul. The index server in the embodiment of the application discovers the service through the consul, so that the decoupling of the client and the service server is realized, and the service processing efficiency is improved.

At step 402, the index server returns the service address to the client.

And step 403, the client sends an access request aiming at the incremental business data to the target server according to the service address.

In step 404, the target server receives the access request and responds to the access request based on the incremental service data loaded in the memory.

And after the index server determines the service address of the target server from the plurality of business servers, the index server returns the service address to the client. The client sends an access request aiming at the incremental business data to the service address, and after the target server receives the access request, the target server responds to the data access request based on the incremental business data loaded in the memory, so that the client can access the incremental business data. In one embodiment, the client may access the service data from the target server through an http-proxy interface.

In the embodiment of the application, the client can send a data address acquisition request including the key code, and acquire the returned service address of the target server from the index server, so that the client can access the incremental business data in the target server based on the service address, thereby realizing the k-v (key-value) query access of the client.

In other embodiments, the index server may also establish a vector type database when establishing the database, so that the client may implement the vector-based service data query access, which is not limited herein.

In the service data processing system of the embodiment of the application, the index server performs data fragmentation on original incremental service data to obtain incremental fragmentation service data, the index server processes the incremental fragmentation service data to obtain incremental service data with a preset data structure, and stores the incremental service data in the HDFS, so that automatic storage of the incremental service data is realized, and distributed management of the original incremental service data can be realized through data fragmentation and distributed storage in the HDFS after processing the fragmented incremental fragmentation service data through the data fragmentation, even if the service data is increased rapidly, distributed management of the service data can be realized through the data fragmentation, and the expandability of the service data processing system is enhanced; then, the index server updates the data version information in the target server based on the incremental business data, and sends a data version update notification to the target server, the target server pulls incremental service data corresponding to the updated data version information from the HDFS according to the updated data version information, and loading the incremental service data into a memory for the client to access, so that the target server directly and automatically pulls the incremental service data from the HDFS according to the updated data version information, and loads the pulled incremental business data into the memory for the client to access, thereby realizing the automatic loading of the incremental business data, improving the automation capability of the business data processing system, therefore, the problem that in the traditional technology, the service data needs to be manually imported into Redis in the memory to provide online access, so that the automation capability of the service data processing system is poor is solved. In addition, in the embodiment of the application, the incremental service data is stored in the HDFS, the target server automatically pulls the incremental service data and loads the incremental service data to the memory, and the incremental service data does not need to be manually guided into the Redis, so that the RDB based on the Redis does not need to be used for realizing persistence, thereby avoiding the memory waste caused by the RDB needing to copy the service data in the memory to the memory and then persistence the copied data, and further reducing the waste of memory resources of the target server, thereby being beneficial to providing stable online service.

In an embodiment, based on the above embodiment shown in fig. 2, this embodiment relates to a process of how the index server obtains original incremental business data and how the index server performs data normalization on the incremental business data. As shown in fig. 6, the service data processing system of this embodiment further includes a data source server, and the service data processing method provided in this embodiment further includes step 205:

step 205, the data source server obtains the original incremental service data, stores the original incremental service data to the HDFS, and sends the second data basic information of the original incremental service data to the index server.

And the second data basic information at least comprises a second storage address of the original incremental service data in the HDFS.

Correspondingly, step 201 may include step 2011 shown in fig. 6:

in step 2011, the index server pulls the original incremental service data from the HDFS storage location indicated by the second storage address according to the second storage address included in the second data basic information, and performs data slicing on the original incremental service data to obtain the incremental slicing service data.

Referring to fig. 6, after step 2011, the method for processing service data according to the present embodiment may further include step 206:

and step 206, the index server analyzes the incremental fragmentation service data according to a preset data format to obtain the analyzed incremental fragmentation service data.

In the embodiment of the application, after the index server performs data fragmentation on the original incremental business data to obtain the incremental fragmentation business data, the index server may analyze the incremental fragmentation business data according to a preset data format to obtain the analyzed incremental fragmentation business data. The preset data format may be, for example, a Protobuf format.

Correspondingly, step 202 may include step 2021 as shown in fig. 6:

step 2021, the index server processes the analyzed incremental fragment service data to obtain incremental service data with a preset data structure, and stores the incremental service data in the HDFS.

The index server may perform the process of step 202 by hashing the parsed incremental fragmentation service data by using a zipper method to generate incremental service data. Therefore, the analyzed incremental fragment service data are all in a uniform data format, and the user can obtain structured data, so that the purpose of normalizing data is achieved. The problem of the traditional technology that the data maintenance difficulty is large due to the fact that the data source is complex and the data definition is various is solved. The embodiment reduces the maintenance difficulty of the incremental business data.

In one embodiment, based on the embodiment shown in fig. 2, the present embodiment relates to a process of how the target server provides the client access based on the full traffic data, and relates to a process of how the target server recovers the traffic data in case of the restart of the target server. In the embodiment of the application, the full service data may be historical service data, and the incremental service data is newly added service data.

Referring to fig. 7, the target server may implement a process of providing client access based on full traffic data by performing

steps

701 and 702 as shown in fig. 7:

step 701, the target server obtains the full data version information, and pulls the full service data corresponding to the full data version information from the HDFS according to the full data version information.

Wherein the full data version information is updated in the target server by the index server based on the full traffic data.

In this embodiment of the application, the full service data may be obtained by performing corresponding processing, such as fragmentation processing, hash processing, and the like, on the original full service data by the index server. And after the index server correspondingly processes the original full service data, storing the obtained full service data to the HDFS.

And the index server updates the full data version information of the full service data to the target server and informs the target server to pull the full service data, and the target server pulls the full service data corresponding to the full data version information from the HDFS according to the full data version information.

In step 702, the target server loads the full service data into a memory for the client to access.

The target server loads the full service data to the memory, and when the client accesses the full service data and/or the incremental service data, the target server can provide service for the client based on the service data loaded in the memory. Therefore, offline service data are automatically converted into online services, and the efficiency of service processing is improved.

With continued reference to fig. 7, the target server may implement the process of recovering the service data in case of restarting the target server by executing step 703 and step 704 shown in fig. 7:

in step 703, if the target server detects a restart trigger event, the target server pulls incremental service data from the HDFS according to the updated data version information, and pulls full service data from the HDFS according to the full data version information.

In step 704, the target server loads the pulled incremental service data and the full service data to a memory for the client to access.

In the embodiment of the application, if the target server is restarted due to conditions such as unexpected downtime and the like of the target server, the target server acquires the full data version information and the updated data version information from the index server again, and the target server pulls the incremental service data from the HDFS according to the updated data version information and pulls the full service data from the HDFS according to the full data version information.

The target server loads the pulled incremental business data and the full business data to the memory, and when the client accesses the full business data and/or the incremental business data, the target server can provide services for the client based on the business data loaded in the memory. Thus, even if the target server is restarted, data loss can not be caused.

In the conventional technology, in order to avoid loss of service data when a server is restarted, data persistence is generally performed through RDB of Redis, and a problem of double memory occupation exists in a data persistence process, which causes waste of memory resources. In the embodiment of the application, after the target server is restarted, the service data can be pulled from the HDFS according to the updated data version information and the full data version information, so that the additional memory of the target server is not occupied for data persistence, and the memory resource of the target server is saved.

In one embodiment, based on the embodiment shown in fig. 7, referring to fig. 8, this embodiment relates to a process of the index server for creating a database of the full traffic data. As shown in fig. 8, the process may include

steps

801, 802, and 803:

step 801, the index server obtains full-volume slicing service data from the HDFS.

In an embodiment of the present application, the service data processing system further includes a data source server, and in a possible implementation manner, the data source server derives full-volume fragmented service data, where the full-volume fragmented service data is obtained by performing data fragmentation on original full-volume service data, and the data source server pushes the full-volume fragmented service data to the HDFS.

In another possible implementation, the data source server derives the original full volume service data, and the data source server may push the original full volume service data to the HDFS, and then the index server performs data fragmentation on the original full volume service data to obtain full volume fragmented service data, and stores the full volume service data in the HDFS.

Thus, the index server can obtain the full amount of fragmented service data from the HDFS.

Step 802, the index server processes the full-volume fragment service data by using an open addressing method, and generates full-volume service data with a preset data structure.

The predetermined data structure may be, for example, in the form of a hash table, and the open addressing method is a method of processing a conflict, in which a free storage unit is searched from the hash table in a certain order from a unit in which a conflict occurs, and an element to be inserted in which the conflict occurs is stored in the unit.

In the embodiment of the application, the index server performs hash processing on the full-scale fragmented service data by using an open addressing method to generate the full-scale service data corresponding to the full-scale fragmented service data. The full service data generated by adopting the open addressing method is compact, and the storage occupation space of the full service data is favorably reduced.

Step 803, the index server stores the full service data to the HDFS, and generates third data basic information according to the full data version information of the full service data and the third storage address.

The full data version information is used for the target server to acquire third data basic information according to the full data version information, and the third data basic information is used for the target server to pull full service data corresponding to the full data version information according to the HDFS storage position indicated by the third storage address, so that the database building of the full service data is realized.

In one possible implementation, after step 803, the target server may implement the process of step 701 by performing step a21 and step a22 as follows:

and step A21, the target server acquires the basic information of the third data from the index server according to the version information of the full data.

Step A22, the target server pulls the full service data corresponding to the full data version information from the HDFS storage location indicated by the third storage address according to the third storage address included in the third data basic information.

In the embodiment of the application, after the index server stores the full service data to the HDFS, the index server generates third data basic information according to the full data version information of the full service data and a third storage address, and the index server updates the data version information in the target server and notifies the target server of pulling the data. And the target server acquires the full data version information, acquires third data basic information corresponding to the full data version information from the index server according to the full data version information, and then pulls the full service data corresponding to the full data version information from the HDFS storage position indicated by the third storage address according to a third storage address included by the third data basic information.

Therefore, the offline full service data is built in the HDFS, the target server automatically pulls the full service data from the HDFS and loads the full service data to the memory to provide data access service, the offline automatic building of the service data is realized, and the effect of automatically converting the offline service data into the online service is achieved.

Optionally, after step 801, the service data processing method according to this embodiment may further include step a 31:

step A31, the index server analyzes the full-scale slicing service data according to a preset data format, and obtains the analyzed full-scale slicing service data.

In the embodiment of the application, after the index server acquires the full-scale fragment service data from the HDFS, the index server analyzes the full-scale fragment service data according to a preset data format to obtain the analyzed full-scale fragment service data. The preset data format may be, for example, Protocol (google Protocol buffer).

Correspondingly, the index server may perform hash processing on the parsed full volume fragmented service data by using an open addressing method to generate full volume service data, so as to implement the process of step 802. Therefore, the analyzed full-scale fragment service data are all in a uniform data format, and the users can obtain structured data, so that the purpose of normalizing data is achieved. The problem of the traditional technology that the data maintenance difficulty is large due to the fact that the data source is complex and the data definition is various is solved. The embodiment reduces the maintenance difficulty of the full service data.

In an embodiment, based on the above-mentioned embodiment shown in fig. 8, referring to fig. 9, this embodiment relates to a process how a target server builds a database for offline service data in a service scaling process. As shown in fig. 9, the process may include the steps of:

step 901, the index server obtains the fragmentation mode information of the original full service data, and detects whether the fragmentation mode information is the same as the stored local fragmentation mode information.

In an actual application scenario, with a change of a service, the number of service servers often needs to be expanded or reduced, and after the expansion or reduction occurs, the number of service servers in a service data processing system will change.

In view of the above situation of the extended capacity, in this embodiment of the present application, before the index server creates the database, the index server first obtains the fragmentation mode information of the original full service data, and detects whether the fragmentation mode information is the same as the stored local fragmentation mode information, where the local fragmentation mode information is related to the number of the plurality of service servers, for example, the local fragmentation mode information may be the number of the service servers in the service data processing system after the extended capacity.

Taking a capacity expansion scenario as an example, for example, if the index server fragments the original full volume service data according to 2 fragments, then the fragment mode information of the original full volume service data obtained by the index server is 2, the number of service servers in the service data processing system after capacity expansion is 4, and the index server detects whether 2 and 4 are the same.

If the fragmentation mode information is the same as the local fragmentation mode information, the fragmentation mode representing the original full volume service data is matched with the number of the service servers after expansion and contraction, and the index server performs the

steps

801, 802 and 803 as above to build a library for the offline service data.

If the fragmentation mode information is different from the local fragmentation mode information, the fragmentation mode representing the original full volume service data is fragmented according to the number of the service servers before the capacity expansion and reduction, so that the original full volume service data needs to be re-fragmented to match the number of the service servers after the capacity expansion and reduction, and the index server executes step 902:

and step 902, the index server performs data fragmentation on the original full volume service data according to the local fragmentation mode information to obtain full volume fragmentation service data.

The server performs data fragmentation on the original full volume service data according to the local fragmentation mode information, namely performs data fragmentation on the original full volume service data again according to the number of machines after capacity expansion to obtain full volume fragmentation service data, and the index server stores the full volume fragmentation service data in the HDFS.

After re-fragmentation, on the basis of step 902, the index server performs the

steps

801, 802 and 803 as above to build a library for the offline service data.

In the embodiment of the application, the index server can automatically build the library according to the condition of combining the expansion and contraction capacity when building the library, so that the offline service data is automatically managed, and the requirements of actual services on the expansion and contraction capacity are better met.

In an embodiment, based on the embodiment shown in fig. 2, referring to fig. 10, this embodiment relates to a process in which a target server loads incremental service data into a memory. In the embodiment of the application, the target server includes a first fragmented data copy and a second fragmented data copy, and the first fragmented data copy and the second fragmented data copy are used for providing the same data access service. As shown in fig. 10, the process may include step 2041 and step 2042:

step 2041, the target server performs data updating on the first fragmented data copy according to the incremental service data, and during the data updating process of the first fragmented data copy, the target server provides data access service through the second fragmented data copy and prohibits the first fragmented data copy during the data updating process from providing data access service.

In the embodiment of the application, each node has multiple copies, and the node is a service server. The first fragmented data copy and the second fragmented data copy may be two copies of the last version of incremental business data (assumed to be referred to as historical incremental business data), and both copies externally provide access service for the historical incremental business data.

If the historical incremental service data is updated, the updated data is the incremental service data of the embodiment of the application, and after the target server acquires the incremental service data, the incremental service data needs to be loaded into the memory, so that the historical incremental service data is updated by adopting the incremental service data.

In the embodiment of the application, the target server updates the fragment data copies one by one, so that at least one fragment data copy can provide services.

In one embodiment, the target server is deployed in kubernets, and the first fragmented data copy and the second fragmented data copy provide services to the outside through different pods. When the target server updates the first fragment data copy according to the incremental business data, the target server changes the pod state corresponding to the first fragment data copy from running to running stop so as to prohibit the first fragment data copy in the data updating process from providing data access service. And if the access request of the client exists, the target server provides data access service for the client through the pod corresponding to the second fragment data copy.

Step 2042, after the data of the first fragmented data copy is updated, the target server performs data updating on the second fragmented data copy according to the incremental business data, and in the process of updating the data of the second fragmented data copy, the target server provides data access service through the first fragmented data copy after data updating and prohibits the second fragmented data copy in the process of updating the data from providing data access service.

After the target server finishes updating the data of the first fragment data copy according to the incremental service data, the target server updates the data of the second fragment data copy according to the incremental service data,

similarly, the target server changes the state of the pod corresponding to the second fragmented data copy from running to stopped running so as to prohibit the second fragmented data copy in the data updating process from providing data access service. And if the access request of the client exists, the target server provides data access service for the client through the pod corresponding to the first fragmented data copy after data updating. Therefore, the online service is not affected in the process of loading the data into the memory, the service quality of the online service is ensured, and the problem of unstable service caused by the fact that the service is provided to the outside while the service data is updated in the traditional technology is solved.

Referring to fig. 11, fig. 11 is a schematic diagram of exemplary sliced data. As shown in fig. 11, segment one and segment two may be different service segment data, for example, segment one and segment two may be different incremental service data. And loading the first fragment into a memory of the first service server by the first service server, loading the second fragment into a memory of the second service server by the second service server, wherein pod:0 and pod:1 are two fragment data copies of the first fragment, and pod:2 and pod:3 are two fragment data copies of the second fragment.

In the data updating process, the first business server updates pod:0 and provides service through pod:1, the second business server updates pod:2 and provides service through pod:3, and the timestamps of pod:1 and pod:3 are consistent, that is, the data version information is consistent, for example, 20200401.

After pod:0 and pod:2 update is completed, the data version information of pod:0 and pod:2 are both updated to the latest version information, for example, 20200402, the first business server updates pod:1 and the second business server updates pod:3, so that the updated pod:0 and pod:2 provide services.

Therefore, for the data access of the same batch, the business data processing system is based on version control, and each accessed business server provides the fragment data copy with the same timestamp (data version information) for the client to access, so that the data access of the same batch is ensured, the business data of the same version are accessed, and the consistency of the data access is ensured.

In one embodiment, referring to fig. 12, a flow chart of a business data processing method is provided, which can be applied in the implementation environment shown in fig. 1. As shown in fig. 12, the method may include the steps of:

step 1001, the data source server exports the full amount of the service data, and the data source server pushes the full amount of the service data to the HDFS.

The full-volume slicing service data is obtained by slicing the original full-volume service data.

Step 1002, the data source server obtains original incremental service data, stores the original incremental service data to the HDFS, and sends second data basic information of the original incremental service data to the index server.

The second data basic information at least comprises a second storage address of the original incremental business data; the data source server can continuously acquire the original incremental service data.

Step 1003, the index server pulls the original incremental business data from the HDFS storage location indicated by the second storage address according to the second storage address included in the second data basic information, and performs data slicing on the original incremental business data to obtain incremental sliced business data.

Step 1004, the index server obtains the fragmentation mode information of the original full volume service data and the original incremental service data, and detects whether the fragmentation mode information is the same as the local fragmentation mode information.

Step 1005, if the data are the same, the index server analyzes the full fragmentation service data according to a preset data format to obtain the analyzed full fragmentation service data.

Step 1006, the index server processes the parsed full volume fragment service data by using an open addressing method, and generates full volume service data having a preset data structure.

Step 1007, the index server stores the full service data to the HDFS, generates third data basic information according to the full data version information of the full service data and the third storage address, and notifies the target server to pull the data.

Step 1008, the target server obtains the full data version information, and obtains third data basic information corresponding to the full data version information from the index server according to the full data version information.

Step 1009, the target server pulls the full service data corresponding to the full data version information from the HDFS storage location indicated by the third storage address according to the third storage address included in the third data basic information.

Step 1010, the index server analyzes the incremental fragmentation service data according to a preset data format to obtain the analyzed incremental fragmentation service data.

And step 1011, the index server processes the analyzed incremental fragment service data by adopting a zipper method to generate incremental service data with a preset data structure.

Step 1012, the index server stores the incremental service data in the HDFS, and generates first data basic information according to the updated data version information of the incremental service data and the first storage address of the incremental service data in the HDFS.

Step 1013, the index server updates the data version information in the target server based on the incremental business data and sends a data version update notification to the target server.

In step 1014, if the target server receives the data version update notification sent by the index server, the target server obtains the first data basic information from the index server according to the updated data version information.

Step 1015, the target server pulls the incremental service data corresponding to the updated data version information from the HDFS storage location indicated by the first storage address according to the first storage address included in the first data basic information.

In step 1016, the target server loads the full traffic data and the incremental traffic data into the memory for the client to access.

In the process that the target server loads the incremental business data into the memory, the target server updates the data of the first fragment data copy according to the incremental business data, in the process of updating the data of the first fragment data copy, the target server provides data access service through the second fragment data copy, and forbids the first fragment data copy in the process of updating the data to provide the data access service; the target server comprises a first fragment data copy and a second fragment data copy, and the first fragment data copy and the second fragment data copy are used for providing the same data access service.

And after the data of the first fragment data copy is updated, the target server updates the data of the second fragment data copy according to the incremental service data, and in the process of updating the data of the second fragment data copy, the target server provides data access service through the first fragment data copy after the data is updated and forbids the second fragment data copy in the process of updating the data to provide the data access service.

Step 1017, if the index server receives the data address acquisition request sent by the client, the index server analyzes the data address acquisition request to obtain a key code corresponding to the data address acquisition request, the index server determines a service address of a target server corresponding to the key code from the plurality of service servers according to the key code, and the index server returns the service address to the client.

In step 1018, the client sends an access request of the service data to the target server according to the service address.

Step 1019, if the target server receives the access request of the service data sent by the client, the target server responds to the data access request based on the full service data and/or the incremental service data in the memory.

Step 1020, if the target server detects a restart trigger event, pulling incremental service data from the HDFS according to the updated data version information, and pulling full service data from the HDFS according to the full data version information.

Step 1021, the target server loads the pulled incremental business data and the full business data to the memory for the client to access.

In this embodiment of the present application, as an implementation manner, the index server may include an incremental data slice delta sharing component and an update peta update component, and the index server executes the steps of the index server based on the components. The target server may include a control component and a controller component, and the target server performs the steps of the target server based on the plurality of components.

The index server, the target server, the data source server, the HDFS and the various components can be accessed through interfaces. Referring to fig. 13, fig. 13 is a schematic diagram of an exemplary component-based service data processing system, in which a client accesses a server through a peta client. The data interaction process based on the interface among the index server, the target server, the data source server, the HDFS and each component is as follows:

1) the data source server ts derives the full fragmentation service data base, and the ts pushes the full fragmentation service data to the HDFS.

2) And ts acquires the original incremental service data delta, stores the original incremental service data to the HDFS, and calls an add _ delta _ file interface of the index server to send second data basic information of the original incremental service data to the index server.

The second data basic information at least comprises a second storage address of the original incremental service data, and ts can continuously acquire the original incremental service data.

3) The delta sharing is used for accessing the index server at regular time, obtaining second data basic information through a get _ delta _ sharing _ file interface and a set _ delta _ sharing _ file _ work interface, performing data fragmentation on original incremental business data through a Map/Reduce task, obtaining incremental fragmentation business data after fragmentation, storing the incremental fragmentation business data in the HDFS by the delta sharing, calling add _ delta _ shared _ file to push file information of the incremental fragmentation business data to the index server, and enabling the file information of the incremental fragmentation business data to comprise data version information and storage path information of the incremental fragmentation business data.

4) Before the peta update is built, an add _ and _ get _ board _ count interface of the index server is called to obtain the fragmentation mode information of the original full volume service data and the original incremental service data, and whether the fragmentation mode information is the same as the local fragmentation mode information is detected.

5) And if the data is the same as the data, analyzing the full-scale fragment service data according to a preset data format to obtain the analyzed full-scale fragment service data. And the peta update processes the analyzed full-volume fragment service data by adopting an open addressing method to generate full-volume service data with a preset data structure. And the peta update stores the full service data to the HDFS, and generates third data basic information according to the full data version information of the full service data and a third storage address.

6) And analyzing the incremental fragment service data according to a preset data format to obtain the analyzed incremental fragment service data. And the peta update processes the analyzed incremental fragment service data by adopting a zipper method to generate incremental service data with a preset data structure. The peta update stores the incremental business data to the HDFS, and generates first data basic information according to the updated data version information of the incremental business data and the first storage address.

7) The peta update updates the data version information on the control and notifies the controller to pull the data.

8) And the controller calls get _ base _ and _ delta _ file _ list of the index server to acquire the full data version information, and acquires third data basic information corresponding to the full data version information from the index server according to the full data version information. And the controller pulls the full service data corresponding to the full data version information from the HDFS storage position indicated by the third storage address according to the third storage address included in the third data basic information.

9) And the controller calls get _ base _ and _ delta _ file _ list of the index server to acquire updated data version information. And the controller acquires first data basic information corresponding to the updated data version information from the index server according to the updated data version information. And the controller pulls the incremental business data corresponding to the updated data version information from the HDFS storage position indicated by the first storage address according to the first storage address included in the first data basic information.

10) And triggering the server to load data by the controller, and loading the full service data and the incremental service data to the memory by the server for the client to access.

In the process that the server loads the incremental business data into the memory, the server updates the data of the first fragment data copy according to the incremental business data, in the process of updating the data of the first fragment data copy, the server provides data access service through the second fragment data copy, and forbids the first fragment data copy in the process of updating the data to provide the data access service; the server comprises a first fragment data copy and a second fragment data copy, and the first fragment data copy and the second fragment data copy are used for providing the same data access service.

After the data updating of the first fragment data copy is completed, the server updates the data of the second fragment data copy according to the incremental service data, and in the data updating process of the second fragment data copy, the server provides data access service through the first fragment data copy after the data updating and forbids the second fragment data copy in the data updating process to provide the data access service.

11) if the index server receives a data address acquisition request sent by the client, the index server analyzes the data address acquisition request to obtain a key code corresponding to the data address acquisition request, and the index server determines a service address of a target server corresponding to the key code from a plurality of service servers according to the key code.

12) index server returns the service address to the client.

13) And the client sends an access request aiming at the incremental business data to the server according to the service address.

14) And if the server receives the access request of the service data sent by the client, responding to the data access request based on the full service data and/or the incremental service data in the memory.

15) And if the controller detects a restart trigger event, pulling incremental service data from the HDFS according to the updated data version information, and pulling full service data from the HDFS according to the full data version information.

16) And the controller triggers the server to reload the data, and the server loads the pulled incremental business data and the full business data to a memory for the client to access.

It should be understood that, although the steps in the above-described flowcharts are shown in order as indicated by the arrows, the steps are not necessarily performed in order as indicated by the arrows. The steps are not performed in the exact order shown and described, and may be performed in other orders, unless explicitly stated otherwise. Moreover, at least a part of the steps in the above-mentioned flowcharts may include a plurality of steps or a plurality of stages, which are not necessarily performed at the same time, but may be performed at different times, and the order of performing the steps or the stages is not necessarily performed in sequence, but may be performed alternately or alternately with other steps or at least a part of the steps or the stages in other steps.

In one embodiment, a business data processing system is provided, the business data processing system comprises an index server and a plurality of business servers, and a target server is one of the business servers;

In one embodiment, the index server is further configured to generate first data basic information according to the updated data version information of the incremental business data and a first storage address of the incremental business data in the HDFS;

correspondingly, the target server is specifically configured to obtain the first data basic information from the index server according to the updated data version information; and pulling the incremental business data corresponding to the updated data version information from the HDFS storage location indicated by the first storage address according to the first storage address included in the first data basic information.

In an embodiment, the index server is specifically configured to process the incremental fragment service data by using a zipper method to obtain the incremental service data with a preset data structure.

In one embodiment, the service data processing system further includes a data source server, where the data source server is configured to obtain the original incremental service data, store the original incremental service data in the HDFS, and send second data basic information of the original incremental service data to the index server, where the second data basic information at least includes a second storage address of the original incremental service data in the HDFS;

correspondingly, the index server is specifically configured to pull the original incremental service data from the HDFS storage location indicated by the second storage address according to the second storage address included in the second data basic information.

In an embodiment, the index server is specifically configured to analyze the incremental fragmentation service data according to a preset data format to obtain analyzed incremental fragmentation service data; and processing the analyzed incremental fragment service data to obtain the incremental service data with a preset data structure.

In one embodiment, the target server is further configured to obtain full data version information, and pull full service data corresponding to the full data version information from the HDFS according to the full data version information, where the full data version information is updated in the target server by the index server based on the full service data; the target server is further used for loading the full service data to a memory for a client to access.

In one embodiment, the target server is further configured to, if a restart trigger event is detected, pull the incremental service data from the HDFS according to the updated data version information, and pull the full service data from the HDFS according to the full data version information; and loading the pulled incremental business data and the full business data to a memory for a client to access.

In an embodiment, the index server is further configured to obtain full volume fragmentation service data from the HDFS, where the full volume fragmentation service data is obtained by performing data fragmentation on original full volume service data; processing the full-volume fragment service data by adopting an open addressing method to generate the full-volume service data with a preset data structure; storing the full service data to the HDFS, and generating third data basic information according to the full data version information of the full service data and a third storage address;

correspondingly, the target server is specifically configured to obtain the third data basic information from the index server according to the full data version information; and pulling the full service data corresponding to the full data version information from the HDFS storage location indicated by the third storage address according to the third storage address included in the third data basic information.

In an embodiment, the index server is further configured to obtain fragmentation mode information of the original full volume service data, and detect whether the fragmentation mode information is the same as local fragmentation mode information, where the local fragmentation mode information is related to the number of the plurality of service servers; if the fragmentation mode information is the same as the local fragmentation mode information, the index server executes the step that the index server acquires full fragmentation service data from the HDFS; and if the fragmentation mode information is different from the local fragmentation mode information, the index server performs data fragmentation on the original full volume service data according to the local fragmentation mode information to obtain the full volume fragmentation service data, and executes the step that the index server acquires the full volume fragmentation service data from the HDFS.

In an embodiment, the target server includes a first fragmented data copy and a second fragmented data copy, where the first fragmented data copy and the second fragmented data copy are used to provide the same data access service, and the target server is specifically used to perform data update on the first fragmented data copy according to the incremental service data, and in a process of updating data of the first fragmented data copy, the target server provides the data access service through the second fragmented data copy and prohibits the first fragmented data copy in a data update process from providing the data access service; and after the data of the first fragment data copy is updated, the target server updates the data of the second fragment data copy according to the incremental service data, and in the process of updating the data of the second fragment data copy, the target server provides data access service through the first fragment data copy after data updating and forbids the second fragment data copy in the process of updating the data to provide the data access service.

In an embodiment, the index server is further configured to determine, if a data address acquisition request sent by the client is received, a service address of the target server according to the data address acquisition request; the index server returns the service address to the client; the client sends an access request aiming at the incremental business data to the target server according to the service address;

the target server is further configured to receive the access request, and respond to the access request based on the incremental service data loaded in the memory.

In one embodiment, the index server is specifically configured to parse the data address obtaining request to obtain a key corresponding to the data address obtaining request; and determining the service address of the target server corresponding to the key code from the plurality of service servers according to the key code.

For the specific definition of the service data processing system, reference may be made to the above definition of the service data processing method, which is not described herein again. The modules in the business data processing system can be wholly or partially implemented by software, hardware and a combination thereof. The modules can be embedded in a hardware form or independent of a processor in the server, and can also be stored in a memory in the server in a software form, so that the processor can call and execute operations corresponding to the modules.

In one embodiment, a server is provided, the internal structure of which may be as shown in fig. 14. The server includes a processor, a memory, and a network interface connected by a system bus. Wherein the processor of the server is configured to provide computing and control capabilities. The memory of the server comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The database of the server is used for storing the service data processing data. The network interface of the server is used for communicating with an external terminal through network connection. The computer program is executed by a processor to implement a business data processing method.

Those skilled in the art will appreciate that the architecture shown in fig. 14 is a block diagram of only a portion of the architecture associated with the subject application, and does not constitute a limitation on the servers to which the subject application applies, as a particular server may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.

In one embodiment, there is provided a server comprising a memory and a processor, the memory having a computer program stored therein, the processor when executing the computer program implementing the steps of:

acquiring original incremental business data, and performing data slicing on the original incremental business data to obtain incremental slicing business data;

processing the incremental fragment service data to obtain incremental service data with a preset data structure, and storing the incremental service data in a distributed file system (HDFS);

updating data version information in a target server based on the incremental business data, and sending a data version updating notice to the target server;

the target server is one of the plurality of service servers, the data version update notification is used for notifying the target server, the incremental service data corresponding to the updated data version information is pulled from the HDFS according to the updated data version information, and the incremental service data is loaded to a memory for a client to access.

In one embodiment, the processor, when executing the computer program, further performs the following steps;

generating first data basic information according to the updated data version information of the incremental business data and a first storage address of the incremental business data in the HDFS; the first data basic information is used for the target server to acquire the first data basic information from the index server according to the updated data version information; the first data basic information is used for the target server to pull the incremental service data corresponding to the updated data version information from the HDFS storage location indicated by the first storage address according to the first storage address included in the first data basic information.

In one embodiment, the processor, when executing the computer program, further performs the steps of:

and processing the incremental fragment service data by adopting a zipper method to obtain the incremental service data with a preset data structure.

correspondingly, the processor, when executing the computer program, further implements the following steps:

and pulling the original incremental service data from the HDFS storage position indicated by the second storage address according to the second storage address included in the second data basic information.

analyzing the incremental fragment service data according to a preset data format to obtain the analyzed incremental fragment service data;

and processing the analyzed incremental fragment service data to obtain the incremental service data with a preset data structure.

In one embodiment, the target server is configured to obtain full data version information, and pull full service data corresponding to the full data version information from the HDFS according to the full data version information, where the full data version information is updated in the target server by the index server based on the full service data; and the target server loads the full service data to a memory for a client to access.

In one embodiment, the target server is configured to, if a restart trigger event is detected, pull the incremental service data from the HDFS according to the updated data version information, and pull the full service data from the HDFS according to the full data version information;

acquiring full-volume fragment service data from the HDFS, wherein the full-volume fragment service data is obtained by performing data fragmentation on original full-volume service data;

processing the full-volume fragment service data by adopting an open addressing method to generate the full-volume service data with a preset data structure;

storing the full service data to the HDFS, and generating third data basic information according to the full data version information of the full service data and a third storage address;

the third data basic information is used for the target server to acquire the third data basic information from the index server according to the full data version information; and the target server pulls the full service data corresponding to the full data version information from the HDFS storage location indicated by the third storage address according to the third storage address included in the third data basic information.

acquiring fragmentation mode information of the original full service data, and detecting whether the fragmentation mode information is the same as local fragmentation mode information, wherein the local fragmentation mode information is related to the number of the plurality of service servers;

if the fragmentation mode information is the same as the local fragmentation mode information, executing the step that the index server acquires full fragmentation service data from the HDFS;

and if the fragmentation mode information is different from the local fragmentation mode information, performing data fragmentation on the original full volume service data according to the local fragmentation mode information to obtain the full volume fragmentation service data, and executing the step of acquiring the full volume fragmentation service data from the HDFS by the index server.

if a data address acquisition request sent by the client is received, determining a service address of the target server according to the data address acquisition request;

returning the service address to the client; the service address is used for the client to send an access request aiming at the incremental business data to the target server according to the service address; and the target server receives the access request and responds to the access request based on the incremental business data loaded in the memory.

analyzing the data address acquisition request to obtain a key code corresponding to the data address acquisition request;

and determining the service address of the target server corresponding to the key code from the plurality of service servers according to the key code.

In one embodiment, a computer-readable storage medium is provided, having a computer program stored thereon, which when executed by a processor, performs the steps of:

In one embodiment, the computer program when executed by a processor further performs the following steps;

In one embodiment, the computer program when executed by the processor further performs the steps of:

correspondingly, the computer program, when executed by the processor, further implements the steps of:

It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. Any reference to memory, storage, database or other medium used in the embodiments provided herein can include at least one of non-volatile and volatile memory. Non-volatile Memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash Memory, optical storage, or the like. Volatile Memory can include Random Access Memory (RAM) or external cache Memory. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM), among others.

The technical features of the above embodiments can be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the above embodiments are not described, but should be considered as the scope of the present specification as long as there is no contradiction between the combinations of the technical features.

The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present patent shall be subject to the appended claims.

Claims

1. A service data processing method is applied to a service data processing system, the service data processing system comprises an index server and a plurality of service servers, and the method comprises the following steps:

2. The method of claim 1, further comprising:

3. The method according to claim 1, wherein the index server processes the incremental fragmentation service data to obtain incremental service data with a preset data structure, and the method comprises:

4. The method of claim 1, wherein the business data processing system further comprises a data origin server, the method further comprising:

5. The method of claim 1, further comprising:

6. The method of claim 1, further comprising:

7. The method of claim 6, further comprising:

8. The method of claim 6, further comprising:

9. The method of claim 8, further comprising:

10. The method according to claim 1, wherein the target server includes a first fragmented data copy and a second fragmented data copy, the first fragmented data copy and the second fragmented data copy are used for providing the same data access service, and the target server loads the incremental business data into the memory, and includes:

11. The method of claim 1, further comprising:

the index server returns the service address to the client;

12. The method of claim 11, wherein the index server determines the service address of the target server according to the data address obtaining request, comprising:

13. A business data processing system is characterized by comprising an index server and a plurality of business servers, wherein a target server is one of the business servers;

14. A server comprising a memory and a processor, the memory storing a computer program, characterized in that the processor realizes the steps of the method of any one of claims 1 to 12 when executing the computer program.

15. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 12.