CN116521399A

CN116521399A - Data processing method and system

Info

Publication number: CN116521399A
Application number: CN202310580224.3A
Authority: CN
Inventors: 熊自新; 徐钟豪
Original assignee: Shanghai Douxiang Information Technology Co ltd
Current assignee: Shanghai Douxiang Information Technology Co ltd
Priority date: 2023-05-22
Filing date: 2023-05-22
Publication date: 2023-08-01

Abstract

The application provides a data processing method and a system, wherein the method comprises the following steps: pre-distributing shared memory; the shared memory is used for caching exchange data between the client and the server; when data exchange with a server is needed, acquiring data to be transmitted; converting the data to be transmitted according to a preset data format to obtain target data; and writing the target data into the shared memory so that the server reads the target data from the shared memory. Therefore, the method can quickly and accurately realize the data exchange between the client and the server, thereby reducing the load of the CPU, reducing the memory consumption and being more beneficial to improving the data processing efficiency.

Description

Data processing method and system

Technical Field

The present application relates to the field of data processing technologies, and in particular, to a data processing method and system.

Background

Big data refers to information which is huge in data size and cannot be extracted, managed, processed and tidied in a reasonable time through the current mainstream software tool, so that the information is more positive in helping business operation decision. In the existing big data processing framework, log data produced by a client side is required to be sent to a server through a network, and the server processes and stores the data. In practice, it is found that in the existing method, the client needs to convert data into JSON data, then the server converts the JSON data, processes and stores the JSON data, so that the conversion between the character string and other data types easily increases the load of the CPU, and meanwhile, the memory consumption is increased, and meanwhile, the precision of certain data types (floating point numbers) is lost due to the use of JSON type data. Therefore, the existing method has the disadvantages of large CPU burden, large memory consumption and low data precision, thereby reducing the data processing efficiency.

Disclosure of Invention

An object of the embodiments of the present application is to provide a data processing method and system, which can quickly and accurately implement data exchange between a client and a server, thereby reducing the load of a CPU, reducing the memory consumption, and being more beneficial to improving the data processing efficiency.

An embodiment of the present application provides a data processing method, applied to a client, including:

pre-distributing shared memory; the shared memory is used for caching exchange data between the client and the server;

when data exchange is needed with the server, acquiring data to be sent;

converting the data to be sent according to a preset data format to obtain target data;

and writing the target data into the shared memory so that the server reads the target data from the shared memory.

In the implementation process, the method can allocate the shared memory in advance preferentially; and when data exchange with the server is needed, acquiring data to be transmitted. Then, the method converts the data to be transmitted according to a preset data format to obtain target data. Finally, the method writes the target data into the shared memory, so that the server reads the target data from the shared memory. Therefore, the method can quickly and accurately realize the data exchange between the client and the server, thereby reducing the load of the CPU, reducing the memory consumption and being more beneficial to improving the data processing efficiency.

Further, after the pre-allocating the shared memory, the method further includes:

a preset data format is agreed with the server side; the data format comprises a data format which represents tab data in the shared memory.

Further, the basic unit of the data format representing the data in the shared memory is an array, and the array is composed of one or more parts of a data type part, a buffer part, an array length part, a null value number and dictionary coding.

Further, the converting the data to be sent according to a preset data format to obtain target data includes:

converting the data to be transmitted into a plurality of groups of data according to a preset data format;

carrying out serialization summarization on the array data to obtain an ordered batch set;

performing binary byte stream conversion processing on the ordered batch set to obtain target data; wherein the target data is serialized binary stream data.

A second aspect of the embodiments of the present application provides a data processing method, which is applied to a server, and includes:

when data exchange with a client is needed, determining a preset shared memory between the client and the client;

reading target data written by the client from the shared memory according to a preset data format;

and carrying out corresponding service processing on the target data.

In the implementation process, the method can determine the preset shared memory between the client and the client preferentially when the data exchange with the client is needed; reading target data written by the client from the shared memory according to a preset data format; and finally, carrying out corresponding service processing on the target data. Therefore, the method can quickly and accurately realize the data exchange between the client and the server, thereby reducing the load of the CPU, reducing the memory consumption and being more beneficial to improving the data processing efficiency.

Further, the method further comprises:

and releasing the memory for storing the target data.

A third aspect of embodiments of the present application provides a data processing system, the data processing system including a client and a server, wherein,

the client is used for pre-distributing the shared memory; the shared memory is used for caching exchange data between the client and the server;

the client is further used for acquiring data to be sent when data exchange with the server is needed;

the client is further configured to perform conversion processing on the data to be sent according to a preset data format, so as to obtain target data;

the client is further configured to write the target data into the shared memory, so that the server reads the target data from the shared memory;

the server is used for determining a preset shared memory between the server and the client when data exchange with the client is needed;

the server is further configured to read, according to a preset data format, target data written by the client from the shared memory;

the server is further configured to perform corresponding service processing on the target data.

In the implementation process, the system can quickly and accurately realize data exchange between the client and the server, so that the CPU burden is reduced, the memory consumption is reduced, and the data processing efficiency is improved.

Further, the client is further configured to agree with a preset data format with the server after the shared memory is pre-allocated; the data format comprises a data format which represents tab data in the shared memory.

Further, the client is specifically configured to convert the data to be sent into a plurality of sets of data according to a preset data format;

the client is specifically further configured to perform serialization summarization on the array data to obtain an ordered batch set;

the client is specifically further configured to perform binary byte stream conversion processing on the ordered batch set to obtain target data; wherein the target data is serialized binary stream data.

Further, the server is specifically further configured to release the memory storing the target data.

A third aspect of the embodiments of the present application provides an electronic device, including a memory and a processor, where the memory is configured to store a computer program, and the processor is configured to execute the computer program to cause the electronic device to perform the data processing method according to any one of the first aspect of the embodiments of the present application.

A fourth aspect of the embodiments of the present application provides a computer readable storage medium storing computer program instructions which, when read and executed by a processor, perform the data processing method according to any one of the first aspect of the embodiments of the present application.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the embodiments of the present application will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present application and should not be considered as limiting the scope, and other related drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

Fig. 1 is a schematic flow chart of a data processing method according to an embodiment of the present application;

FIG. 2 is a flowchart of another data processing method according to an embodiment of the present disclosure;

FIG. 3 is a flowchart illustrating another data processing method according to an embodiment of the present disclosure;

FIG. 4 is a schematic diagram of a data processing system according to an embodiment of the present application;

fig. 5 is an exemplary diagram of an int32 array provided in an embodiment of the present application;

FIG. 6 is an exemplary diagram of a result set provided by an embodiment of the present application;

fig. 7 is an overall logic diagram of a data processing method according to an embodiment of the present application.

Detailed Description

The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application.

It should be noted that: like reference numerals and letters denote like items in the following figures, and thus once an item is defined in one figure, no further definition or explanation thereof is necessary in the following figures. Meanwhile, in the description of the present application, the terms "first", "second", and the like are used only to distinguish the description, and are not to be construed as indicating or implying relative importance.

Example 1

Referring to fig. 1, fig. 1 is a flowchart of a data processing method according to the present embodiment. The data processing method is applied to the client, and specifically comprises the following steps:

s101, pre-distributing a shared memory.

In this embodiment, the shared memory is used to cache the exchange data between the client and the server.

S102, when data exchange with the server is needed, acquiring data to be transmitted.

S103, converting the data to be transmitted according to a preset data format to obtain target data.

S104, writing the target data into the shared memory so that the server side reads the target data from the shared memory.

In this embodiment, the method describes a high-efficiency data processing method for large traffic. It can be seen that the method can support receiving logs from sensors and processing them quickly into a warehouse without losing events when the number of eps events per second exceeds a million under high flow conditions.

In this embodiment, the execution subject of the method may be a computing system such as a computer or a server, which is not limited in this embodiment.

In this embodiment, the execution body of the method may be an intelligent device such as a smart phone or a tablet computer, which is not limited in this embodiment.

Therefore, by implementing the data processing method described in the embodiment, data exchange between the client and the server can be quickly and accurately realized, so that the load of a CPU (central processing unit) is reduced, the memory consumption is reduced, and the data processing efficiency is improved.

Example 2

Referring to fig. 2, fig. 2 is a flow chart of a data processing method according to the present embodiment. The data processing method is applied to the client, and specifically comprises the following steps:

s201, pre-distributing the shared memory.

In this embodiment, a zero copy technology is adopted in advance between the Sensor system and the log processing module to realize data transmission and reception. The memory pool is pre-allocated at the beginning of system startup, and dynamic application and release of memory are not needed once. Therefore, the method can save system overhead and improve the performance of data transmission.

S202, agreeing with a preset data format with the server side.

In this embodiment, the data format includes a data format representing tab data in the shared memory.

In this embodiment, the data format represents that the basic unit of data in the shared memory is an array, and the array is composed of one or more parts of a data type part, a buffer part, an array length part, a number of null values, and dictionary coding.

In this embodiment, the basic unit of representing data in memory in this data format is an array, which represents a series of data of known length and the same type. And multiple arrays of the same length, the same type, or different types may be used to represent the result set (or a portion of the result set).

For example, each array consists of several parts:

(1) Logically data types (record array types, such as int 32);

(2) A list of buffers (storing specific data and null values);

(3) A length of 64 bits signed integer (record array length, also can be 32 bits);

(4) Another signed integer of 64 bits in length (number of record null values);

(5) (optional) dictionary (array for dictionary coding).

Further by way of example, an int32 array [1, null,2,4,8] as follows, will be presented as shown in FIG. 5. This expression in memory is typically a result set handling during actual program processing, and fig. 6 shows a result set (or table).

The left side of fig. 6 shows a result set with floating point type and character string type, which can be represented by conversion into an ordered set with size of 2 on the right, and the array (double array and string array) in the set has length of 3. It can be seen that this data format limits the maximum length of the array, and when the size of the result set (or table) exceeds the maximum length of the array, it is necessary to split the result set horizontally into multiple ordered sets.

In this embodiment, the log is not packaged in json format, but a data format representing tab data in the memory is defined.

In this embodiment, this format of representation of tab data is optimized specifically for data analysis type operations. For example, the advantages of modern cpu can be fully utilized to perform vectorization calculation in a column format; also, for example, the IPC format, can serialize data in memory, perform network transmission, or persist data in a file.

S203, when data exchange with the server is needed, acquiring data to be transmitted.

S204, converting the data to be transmitted into a plurality of groups of data according to a preset data format.

And S205, carrying out serialization summarization on the group data to obtain an ordered batch set.

In this embodiment, the method may form a batch set from an ordered set of multiple array components with the same length. Wherein this batch set is a serialized base unit. Multiple ordered sets can be serialized into a binary byte stream, allowing data to be exchanged between different processes.

S206, performing binary byte stream conversion processing on the ordered batch set to obtain target data.

In this embodiment, the target data is serialized binary stream data.

S207, writing the target data into the shared memory so that the server reads the target data from the shared memory.

In this embodiment, the method may take batch data serialized into binary stream from the shared memory, even all subsequent service streams take the memory-based format as the standard data exchange format, so that most of CPU cycles are omitted in the serialization and anti-serialization of data, which is better in time and memory consumption than the previous JSON-based exchange method, and is beneficial to realizing seamless sharing of data between different systems or system components.

In this embodiment, the method can optimize the data exchange manner, and can read the data only by using the memory mapping function, so that the method shows more excellent performance under the condition of a large data volume set.

Example 3

Referring to fig. 3, fig. 3 is a flowchart of a data processing method according to the present embodiment. The data processing method is applied to a server and specifically comprises the following steps:

s301, when data exchange with the client is needed, determining a preset shared memory between the client and the shared memory.

S302, reading target data written by the client from the shared memory according to a preset data format.

S303, performing corresponding service processing on the target data.

S304, releasing the memory for storing the target data.

Referring to fig. 7, in this embodiment, the Sensor shown in fig. 7 is used as a client for sending data, the log processing module shown in fig. 7 is used as a server for receiving data, and a specific data exchange process between the client and the server is described by this, which is specifically as follows:

firstly, the client allocates a shared memory for caching exchanged data when a program is started;

then, the original data to be sent by the client are converted into a final result set in batches according to a contracted memory-based column arrangement mode, and the final result set is written into the shared memory;

secondly, the server reads a batch result set from the shared memory, reads the result set according to a agreed format and releases a memory space;

and finally, carrying out batch data processing in the server according to the service requirement.

Example 4

Referring to fig. 4, fig. 4 is a schematic diagram of a data processing system according to the present embodiment. As shown in fig. 4, the data processing system includes a client and a server, wherein,

a client 400, configured to pre-allocate a shared memory; the shared memory is used for caching exchange data between the client 400 and the server 500;

the client 400 is further configured to obtain data to be sent when data exchange with the server 500 is required;

the client 400 is further configured to perform conversion processing on data to be sent according to a preset data format, so as to obtain target data;

the client 400 is further configured to write the target data into the shared memory, so that the server 500 reads the target data from the shared memory;

the server 500 is configured to determine a preset shared memory with the client 400 when data exchange with the client 400 is required;

the server 500 is further configured to read, from the shared memory, the target data written by the client 400 according to a preset data format;

the server 500 is further configured to perform corresponding service processing on the target data.

In this embodiment, the basic unit of the data format representing data in the shared memory is an array, and the array is composed of one or more parts of a data type part, a buffer part, an array length part, a number of null values, and dictionary coding.

In this embodiment, the explanation of the data processing system may refer to the description in embodiment 1, embodiment 2 or embodiment 3, and further description is omitted in this embodiment.

Therefore, the data processing system described in the embodiment can quickly and accurately realize data exchange between the client and the server, so that the load of a CPU (Central processing Unit) is reduced, the memory consumption is reduced, and the data processing efficiency is improved.

Example 5

With continued reference to fig. 4, the data processing system includes a client and a server, wherein,

As an optional implementation manner, the client 400 is further configured to, after pre-allocating the shared memory, agree with a preset data format with the server 500; the data format comprises a data format which represents the tab data in the shared memory.

As an optional implementation manner, the client 400 is specifically configured to convert data to be sent into data sets according to a preset data format;

the client 400 is specifically further configured to perform serialized summarization on the array data, so as to obtain an ordered batch set;

the client 400 is specifically further configured to perform binary byte stream conversion processing on the ordered batch set to obtain target data; wherein the target data is serialized binary stream data.

As an optional implementation manner, the server 500 is specifically further configured to perform a release process on a memory storing the target data.

An embodiment of the present application provides an electronic device, including a memory and a processor, where the memory is configured to store a computer program, and the processor is configured to execute the computer program to cause the electronic device to execute a data processing method in embodiment 1, embodiment 2, or embodiment 3 of the present application.

The present embodiment provides a computer readable storage medium storing computer program instructions that, when read and executed by a processor, perform the data processing method of embodiment 1, embodiment 2, or embodiment 3 of the present application.

In the several embodiments provided in this application, it should be understood that the disclosed systems and methods may be implemented in other ways as well. The system embodiments described above are merely illustrative, for example, of the flowcharts and block diagrams in the figures that illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

In addition, the functional modules in the embodiments of the present application may be integrated together to form a single part, or each module may exist alone, or two or more modules may be integrated to form a single part.

The functions, if implemented in the form of software functional modules and sold or used as a stand-alone product, may be stored in a computer-readable storage medium. Based on such understanding, the technical solution of the present application may be embodied essentially or in a part contributing to the prior art or in a part of the technical solution, in the form of a software product stored in a storage medium, including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes.

The foregoing is merely exemplary embodiments of the present application and is not intended to limit the scope of the present application, and various modifications and variations may be suggested to one skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application. It should be noted that: like reference numerals and letters denote like items in the following figures, and thus once an item is defined in one figure, no further definition or explanation thereof is necessary in the following figures.

The foregoing is merely specific embodiments of the present application, but the scope of the present application is not limited thereto, and any person skilled in the art can easily think about changes or substitutions within the technical scope of the present application, and the changes and substitutions are intended to be covered by the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

It is noted that relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.

Claims

1. A data processing method, applied to a client, comprising:

when data exchange is needed with the server, acquiring data to be sent;

2. The data processing method of claim 1, wherein after the pre-allocating shared memory, the method further comprises:

3. The method of claim 2, wherein the data format represents that the basic unit of data in the shared memory is an array, and the array is composed of one or more of a data type part, a buffer part, an array length part, a number of null values, and dictionary coding.

4. The data processing method according to claim 1, wherein the converting the data to be sent according to a preset data format to obtain target data includes:

5. The data processing method is characterized by being applied to a server and comprising the following steps:

and carrying out corresponding service processing on the target data.

6. The data processing method of claim 5, wherein the method further comprises:

and releasing the memory for storing the target data.

7. A data processing system, characterized in that the data processing system comprises a client and a server, wherein,

8. The data processing system of claim 7, wherein the client is further configured to agree with the server with a preset data format after the pre-allocation of the shared memory; the data format comprises a data format which represents tab data in the shared memory.

9. An electronic device comprising a memory for storing a computer program and a processor that runs the computer program to cause the electronic device to perform the data processing method of any one of claims 1 to 6.

10. A readable storage medium, characterized in that the readable storage medium has stored therein computer program instructions, which when read and executed by a processor, perform the data processing method of any of claims 1 to 6.