CN116225729A

CN116225729A - Method, system, equipment and storage medium for data processing

Info

Publication number: CN116225729A
Application number: CN202111481691.8A
Authority: CN
Inventors: 黄增荣; 陈月华; 高世雄
Original assignee: Jingdong Technology Holding Co Ltd
Current assignee: Jingdong Technology Holding Co Ltd
Priority date: 2021-12-03
Filing date: 2021-12-03
Publication date: 2023-06-06

Abstract

The present disclosure relates to a method and system for data processing, a device and a storage medium, wherein the method includes: inquiring various label data for specialized operation required locally; judging whether the plurality of tag data for specialized operation are loaded or not: when the various label data for the specialized operation is not loaded, an interface calling request is sent to load and store the various label data for the specialized operation through a calling interface; in response to the received tag identification request, according to various tag data for specialized operation and tag identification information in the tag identification request, an operation strategy corresponding to the tag identification information is determined, and various tag data for specialized operation is loaded to the local, so that the data verification process can be changed from the remote interface calling process to the local process, and the problem that the data verification process is influenced by overlarge interface pressure caused by network jitter, interface jitter and large concurrency is avoided.

Description

Method, system, equipment and storage medium for data processing

Technical Field

The present disclosure relates to the field of big data technologies, and in particular, to a method, a system, an apparatus, and a storage medium for data processing.

Background

In internet business, the implementation of each transaction, presentation and payment is very complex. For example, in the electronic commerce industry, the user sees the commodity, the price of the commodity and the coupon corresponding to the commodity on the commodity detailed page, and the presentation of these information requires complex business logic. There may be multiple RPC (Remote Procedure Call ) calls per presentation and per request, and there may be numerous uncertainties in each RPC call, such as network conditions, GC (Garbage Collection ) at the service provider, performance of the service provider interface itself, performance and jitter of the interface on which the service provider depends, and each factor affects the performance of the interface when there is a subtle problem.

In the prior art, when data acquisition and judgment are needed, remote RPC call is directly carried out, data acquisition and judgment steps are directly carried out on a downstream interface, if the performance of the downstream interface cannot be supported, the system fuses the downstream interface, or the downstream interface performs degradation or current limiting on a request, and service availability is ensured through the means. However, the prior art has the following drawbacks: firstly, when an interface is directly called through the RPC, the downstream interface is affected by various factors to cause unstable performance; second, in the case of large concurrency, the pressure on the dependent interfaces may be very high, and the service provider may be damaged by either limiting current or degrading it, or blowing it by the upstream service, which may seriously affect the user experience.

Disclosure of Invention

To solve or at least partially solve the above technical problems, embodiments of the present disclosure provide a method, a system, an apparatus, and a storage medium for data processing.

In a first aspect, embodiments of the present disclosure provide a data processing method, applied to a service server, the method including:

inquiring various label data for specialized operation required locally;

judging whether the plurality of tag data for specialized operation are loaded or not:

when the various label data for the specialized operation is not loaded, an interface calling request is sent to load and store the various label data for the specialized operation through a calling interface;

and responding to the received tag identification request, and determining an operation strategy corresponding to the tag identification information according to various tag data for specialized operation and the tag identification information in the tag identification request.

In one possible implementation manner, the determining, according to the tag data and the tag identification information in the tag identification request for specialized operation, an operation policy corresponding to the tag identification information includes:

comparing the tag identification information in the tag identification request with various tag data for specialized operation, and determining the tag data type to which the tag identification information belongs;

and determining the operation strategy corresponding to the tag identification information according to the tag data type to which the tag identification information belongs based on the corresponding relation between the tag data type and the operation strategy.

In one possible embodiment, the method further comprises:

responding to the received update information of various label data for specialized operation, and sending an interface calling request;

and loading and storing the updated various label data for specialized operation through the calling interface so as to respond to the received label identification request, and determining an operation strategy corresponding to the label identification information in the label identification request by utilizing the updated various label data for specialized operation.

In one possible implementation, the plurality of tag data for the specialized operations is compressed data.

In a second aspect, embodiments of the present disclosure provide a method of data processing, applied to a remote server, including:

responding to a received interface call request, judging whether various tag data for specialized operation corresponding to the interface call request are compressed data or not:

and when the various label data for the specialized operation corresponding to the data loading request is not compressed data, compressing the various label data for the specialized operation, and sending the compressed various label data for the specialized operation to a service server through an interface corresponding to the interface calling request.

In one possible embodiment, the method further comprises:

when various label data for specialized operation are updated, a data update message is sent to the service server to trigger the service server to send a data loading request.

In a third aspect, embodiments of the present disclosure provide a system for data processing, comprising:

a service server for querying a plurality of tag data for specialized operations required locally; judging whether the plurality of tag data for specialized operation are loaded or not: when the various label data for the specialized operation is not loaded, an interface calling request is sent to load and store the various label data for the specialized operation through a calling interface; and responding to the received tag identification request, and determining an operation strategy corresponding to the tag identification information according to various tag data for specialized operation and the tag identification information in the tag identification request.

In one possible embodiment, the system further comprises:

the remote server is used for responding to the received interface call request and judging whether various tag data for specialized operation corresponding to the interface call request are compressed data or not: and when the various label data for the specialized operation corresponding to the data loading request is not compressed data, compressing the various label data for the specialized operation, and sending the compressed various label data for the specialized operation to a service server through an interface corresponding to the interface calling request. In a fourth aspect, embodiments of the present disclosure provide an electronic device including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory complete communication with each other through the communication bus;

a memory for storing a computer program;

and the processor is used for realizing the data processing method when executing the program stored in the memory.

In a fifth aspect, embodiments of the present disclosure provide a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the method of data processing described above.

Compared with the prior art, the technical scheme provided by the embodiment of the disclosure has at least part or all of the following advantages:

the data processing method and system provided by the embodiment of the disclosure inquire various tag data for specialized operation locally; judging whether the plurality of tag data for specialized operation are loaded or not: when the various label data for the specialized operation is not loaded, an interface calling request is sent to load and store the various label data for the specialized operation through a calling interface; in response to the received tag identification request, according to various tag data for specialized operation and tag identification information in the tag identification request, an operation strategy corresponding to the tag identification information is determined, and various tag data for specialized operation is loaded to the local, so that the data verification process can be changed from the remote interface calling process to the local process, and the problem that the data verification process is influenced by overlarge interface pressure caused by network jitter, interface jitter and large concurrency is avoided.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the disclosure and together with the description, serve to explain the principles of the disclosure.

In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings that are required to be used in the description of the embodiments or the related art will be briefly described below, and it will be apparent to those skilled in the art that other drawings can be obtained from these drawings without inventive effort.

FIG. 1 schematically illustrates a flow diagram of a method of data processing according to an embodiment of the present disclosure;

fig. 2 schematically illustrates a detailed flow diagram of step S14 according to an embodiment of the present disclosure;

FIG. 3 shows a flow diagram schematically illustrating a method of data processing according to another embodiment of the present disclosure;

FIG. 4 schematically illustrates a flow diagram of a method of data processing according to yet another embodiment of the present disclosure;

FIG. 5 schematically illustrates a flow diagram of a method of data processing according to an embodiment of the present disclosure;

FIG. 6 schematically illustrates a block diagram of a prior art system for data processing;

FIG. 7 schematically illustrates a block diagram of a system for data processing according to an embodiment of the present disclosure; and

fig. 8 schematically shows a block diagram of an electronic device according to an embodiment of the disclosure.

Detailed Description

For the purposes of making the objects, technical solutions and advantages of the embodiments of the present disclosure more apparent, the technical solutions of the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present disclosure, and it is apparent that the described embodiments are some, but not all, embodiments of the present disclosure. All other embodiments, which can be made by one of ordinary skill in the art without undue burden from the disclosure, are within the scope of the disclosure.

Referring to fig. 1, a first exemplary embodiment of the present disclosure provides a data processing method applied to a service server, the method including:

s11, inquiring various tag data for specialized operation required locally;

in practical applications, the specialized operation may be that the user group is divided into multiple tag groups, and the corresponding operation policy is executed for each tag group, for example, the user group is divided into 4 tag groups of a (20-40 years old), B (female), C (engineer post), D (beijing user), the operation policy for the a (20-40 years old) tag group is a policy for operating the interest point of the 20-40 years old, the operation policy for the B (female) tag group is a policy for operating the interest point of the female, the operation policy for the C (engineer post) tag group is a policy for operating the interest point of the engineer, and the operation policy for the D (beijing user) tag group is a policy for operating the interest point of the beijing user. The plurality of tag data is user data for each of a plurality of tag groups.

S12, judging whether the plurality of label data for specialized operation are loaded or not:

if yes, go to step S14;

if not, executing step S13;

s13, sending an interface call request to load and store various tag data for specialized operation through a call interface;

in practical application, the loaded various tag data for specialized operation are compressed data.

S14, responding to the received tag identification request, and determining an operation strategy corresponding to the tag identification information according to various tag data used for specialized operation and the tag identification information in the tag identification request.

Referring to fig. 2, in step S14, the determining, according to the tag data and the tag identification information in the tag identification request for the specialized operation, an operation policy corresponding to the tag identification information includes:

s21, comparing the tag identification information in the tag identification request with various tag data for specialized operation, and determining the tag data type of the tag identification information;

s22, based on the corresponding relation between the label data type and the operation strategy, determining the operation strategy corresponding to the label identification information according to the label data type to which the label identification information belongs.

In practical application, inputting tag identification information in a tag identification request into various tag data for specialized operation, and determining a tag group in the tag data to which the tag identification information belongs; and determining the operation strategy corresponding to the tag identification information according to the tag group to which the tag identification information belongs based on the corresponding relation between the preset tag group and the operation strategy.

Referring to fig. 3, the method for data processing according to the embodiment of the present disclosure includes, in addition to steps S11 to S14:

s31, responding to the received update messages of various label data for specialized operation, and sending an interface calling request;

s32, loading and storing the updated various label data for specialized operation through the calling interface so as to respond to the received label identification request, and determining an operation strategy corresponding to the label identification information in the label identification request by utilizing the updated various label data for specialized operation.

Referring to fig. 4, a second exemplary embodiment of the present disclosure provides a data processing method applied to a remote server, including:

s41, responding to a received interface call request, and judging whether various tag data for specialized operation corresponding to the interface call request are compressed data or not:

if yes, go to step S43;

if not, executing step S42;

s42, compressing the plurality of label data for specialized operation;

s43, the interface corresponding to the interface call request is used for sending the compressed various label data for specialized operation to the service server.

Referring to fig. 5, the method for data processing according to the embodiment of the present disclosure includes, in addition to steps S41 to S43:

s51, monitoring whether various label data for specialized operation are updated in real time:

if yes, go to step S52;

if not, not responding;

s52, sending a data update message to the service server to trigger the service server to send a data loading request.

In practical application, when any one of the tag data is newly added, deleted, modified and the like, various tag data for specialized operation is confirmed to be updated.

According to the data processing method disclosed by the invention, the process of carrying out data verification by the remote call interface is converted into the process of pre-pulling and storing the required remote call data to the local and carrying out data verification locally, so that the call times of the remote RPC are effectively reduced in a huge call request, the pressure of a service provider is greatly reduced, and the overall performance of the interface is improved.

Based on the same inventive concept, a data processing system provided by a third exemplary embodiment of the present disclosure includes a service server and a remote server:

a service server for querying a plurality of tag data for specialized operations required locally; judging whether the plurality of tag data for specialized operation are loaded or not: when the various label data for the specialized operation is not loaded, an interface calling request is sent to load and store the various label data for the specialized operation through a calling interface; responding to the received tag identification request, and determining an operation strategy corresponding to the tag identification information according to various tag data for specialized operation and the tag identification information in the tag identification request; and

the remote server is used for responding to the received interface call request and judging whether various tag data for specialized operation corresponding to the interface call request are compressed data or not: and when the various label data for the specialized operation corresponding to the data loading request is not compressed data, compressing the various label data for the specialized operation, and sending the compressed various label data for the specialized operation to a service server through an interface corresponding to the interface calling request.

In this embodiment, referring to fig. 6 and fig. 7, the workflow of the system of the data in the prior art and the system of the data in the present disclosure are compared in an application scenario where a remote interface needs to be called to determine whether a certain data exists.

Application scene: when the special operation is carried out on the user group, the user group is divided into 4 labels of A (20-40 years old), B (female), C (engineer post) and D (Beijing user), and each group has a certain operation policy. For a requesting user on a line, it is necessary to identify which tag the user is within. The RPC interface needs to be invoked to determine whether the current user is within the A, B, C, D tag, respectively.

Referring to fig. 6, under the use scenario of the prior art, the service system (service server) and the dependent interface (data call interface) are divided, and the interaction flow of the two is as follows:

when the service system receives a user request, firstly, service logic processing is performed, and when legitimacy judgment needs to be performed on data, for example, whether a user 'zhangsan' hits one of 4 labels of A (20-40 years old), B (female), C (engineer post) and D (Beijing user) needs to be performed, an interface needs to be called to request a 'dependent interface' for identification, and a result is returned after the dependent interface is judged.

The system shown in fig. 6 has the following problems: in high concurrency, the pressure on the dependent interfaces can be large, and the performance cannot be guaranteed.

Thus, for a new data processing center to be added to the system, see fig. 7, the system is divided into three parts: the interaction flow of the service system (service server), the data processing center of the service system and the dependent interface (data calling interface) is as follows.

(1) Service system-service start, constructing label to be loaded

And the service system initializes the data when the server is started, sorts the data, and identifies the data to be loaded. For example, 4 tags a (20-40 years old), B (female), C (engineer's post), D (beijing user) are identified at service start-up.

(2) Data processing center-scheduler (executing timing task, system executing code once every fixed time) process, calling dependent interface

The data processing center uses the scheduler to process at regular time, and queries the data needed to be loaded from the service system interface at regular time. For example, A, B, C, D tags are acquired and need to be processed. Next, it is determined whether the data has been loaded, and if not, a load operation is performed. The interfaces that are relied upon need to be invoked when loading is performed.

(3) Relying on interface-receiving request to generate bitmap data

After receiving the call request of the data processing center, the dependent interface judges whether the data is compressed or not, and if not, the data is compressed. In this embodiment, a bitmap (a data structure for storing specific data by a Bit array) is used, and since Bit is the minimum unit of data, the data structure often saves very much memory space, and Bit is used as the unit to store data, so that a great saving in memory space can be achieved. And the method is beneficial to data storage and network transmission in the memory.

For example, for a (20-40 years old) tag, all users corresponding to the tag are stored together with a bitmap, and about 30M is required for conversion to bitmap storage through on-line verification of 1000 ten thousand users.

When the user tag is updated, the system generates a new bitmap and sends an update MQ (Message queue).

(4) Data processing center-download bitmap data

The data processing center obtains the bitmap data to be processed from the dependent interface and then puts the bitmap data into the memory.

When the label generates data update, the update MQ notice is received, and then the corresponding label bitmap is pulled again. Thus, the data center can store the bitmap object of the latest data tag in real time. While writing updates to the ZooKeeper (ZooKeeper is a typical distributed data consistency solution), the distributed application may implement functions such as data publish/subscribe, load balancing, naming service, distributed coordination/notification, cluster management, master election, distributed lock and distributed queue, abbreviated ZK, etc., based on the ZooKeeper. The ZK is used to notify the individual service servers.

(5) Service system-starting loading bitmap data

When the service system is started, recognizing that the values of 4 labels of A (20-40 years old), B (female), C (engineering post) and D (Beijing user) need to be loaded, requesting a data processing center, loading the values to bitmap objects corresponding to the labels, and storing the bitmap objects in a server memory.

When the data updating notification of the ZK is monitored, the latest data bitmap is pulled from the data processing center and stored in the memory.

After the processing of the steps (1) - (5), when the service system application server is started, the data of the interface needing RPC call is stored into the server memory through the format of bitmap.

(6) Service system-data verification

And executing the service logic after the service system receives the front-end request. When the current user needs to be verified in which tag, the remote RPC interface is directly called from the prior art, and memory verification is directly performed according to the bitmap object instead. The method provided by Bitmap can determine whether a certain data is in the Bitmap object, which is not described in detail.

When the remote RPC interface is required to be called for data verification, the data of the RPC interface is compressed and then stored in the service server memory, the logic which is originally required to be called for verification by the RPC can be changed into the local memory verification through a series of processes, so that the remote call cost is reduced, the problem of poor performance is solved due to good memory verification stability and high speed, the problem of overlarge pressure of the remote interface is solved without calling the remote interface, and the data set with the size of tens of megabytes is processed by the bitmap, so that the system memory can be completely born.

The data processing system disclosed by the invention identifies the data to be loaded through the dispatcher of the data processing center, and then invokes the dependent interface to download the data so as to prepare the data required by the service server in advance before the service server sends the data acquisition request, thereby being capable of accelerating the data acquisition speed of the service server.

In the data processing system disclosed by the invention, the data set is integrated into a compressed data format, such as a bitmap format, by means of the interface support, remote call downloading can be supported, and an update message can be sent out when the data set is updated, so that the remote call data of the data processing center and the service server can be ensured to be up to date and the memory occupation amount is small.

In the data processing system, the data processing center monitors information during data updating, downloads the latest data and writes the update into the ZK, so that the data processing center can send data update information to a plurality of service servers through the ZK at the same time.

After the system service server for data processing receives the notification of the ZK, the latest data is downloaded from the data processing center, so that the latest remote call data of the service server can be ensured.

The system of the present embodiment may be a computer software platform or the like.

The implementation process of the functions and roles of each part in the above system is specifically shown in the implementation process of the corresponding steps in the above method, and will not be described herein again.

For system embodiments, reference is made to the description of method embodiments for the relevant points, since they essentially correspond to the method embodiments. The system embodiments described above are merely illustrative, wherein the elements illustrated as separate elements may or may not be physically separate, and the elements shown as elements may or may not be physical elements, may be located in one place, or may be distributed over a plurality of network elements. Some or all of the modules may be selected according to actual needs to achieve the objectives of the disclosed solution. Those of ordinary skill in the art will understand and implement the present invention without undue burden.

Based on the same inventive concept, referring to fig. 8, an electronic device provided in a fourth exemplary embodiment of the present disclosure includes a processor 1110, a communication interface 1120, a memory 1130, and a communication bus 1140, where the processor 1110, the communication interface 1120, and the memory 1130 perform communication with each other through the communication bus 1140;

a memory 1130 for storing a computer program;

processor 1110, when executing programs stored in memory 1130, implements the following data processing method:

responding to a received server starting signal, and determining remote call data to be loaded;

sending a data acquisition request, and acquiring and storing the remote call data;

and responding to the received data verification request, and carrying out local verification on the data to be verified in the data verification request according to the stored remote call data.

The communication bus 1140 may be a peripheral component interconnect standard (Peripheral Component Interconnect, PCI) bus or an extended industry standard architecture (Extended Industry Standard Architecture, EISA) bus, among others. The communication bus 1140 may be divided into an address bus, a data bus, a control bus, and the like. For ease of illustration, the figures are shown with only one bold line, but not with only one bus or one type of bus.

The communication interface 1120 is used for communication between the electronic device and other devices described above.

The memory 1130 may include random access memory (Random Access Memory, simply RAM) or may include non-volatile memory (non-volatile memory), such as at least one magnetic disk memory. Optionally, the memory 1130 may also be at least one storage device located remotely from the processor 1110.

The processor 1110 may be a general-purpose processor, including a central processing unit (Central Processing Unit, CPU for short), a network processor (Network Processor, NP for short), etc.; but also digital signal processors (Digital Signal Processing, DSP for short), application specific integrated circuits (Application Specific Integrated Circuit, ASIC for short), field-programmable gate arrays (Field-Programmable Gate Array, FPGA for short) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

Based on the same inventive concept, a fifth exemplary embodiment of the present disclosure also provides a computer-readable storage medium. The computer readable storage medium has stored thereon a computer program which, when executed by a processor, implements a method of data processing as described above.

The computer-readable storage medium may be embodied in the apparatus/means described in the above embodiments; or may exist alone without being assembled into the apparatus/device. The computer-readable storage medium carries one or more programs that when executed implement methods of data processing according to embodiments of the present disclosure.

According to embodiments of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, which may include, for example, but is not limited to: a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this disclosure, a computer-readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.

It should be noted that in this document, relational terms such as "first" and "second" and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises an element.

The foregoing is merely a specific embodiment of the disclosure to enable one skilled in the art to understand or practice the disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the disclosure. Thus, the present disclosure is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method of data processing, for application to a service server, the method comprising:

inquiring various label data for specialized operation required locally;

2. The method according to claim 1, wherein the determining an operation policy corresponding to the tag identification information according to the tag identification information in the tag identification request and the plurality of tag data for specialized operations includes:

3. The method according to claim 1, wherein the method further comprises:

4. The method of claim 1, wherein the plurality of tag data for the specialized operations is compressed data.

5. A method of data processing, for use with a remote server, comprising:

6. The method of claim 5, wherein the method further comprises:

7. A system for data processing, comprising:

8. The system of claim 7, wherein the system further comprises:

9. The electronic equipment is characterized by comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory are communicated with each other through the communication bus;

a memory for storing a computer program;

a processor for implementing the method of data processing according to any one of claims 1-4 or 5-6 when executing a program stored on a memory.

10. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the method of data processing according to any of claims 1-4 or 5-6.