CN116774910A

CN116774910A - Network data processing method, device, equipment, storage medium and program product

Info

Publication number: CN116774910A
Application number: CN202210239272.1A
Authority: CN
Inventors: 廖阿敏
Original assignee: Tencent Technology Shenzhen Co Ltd
Current assignee: Tencent Technology Shenzhen Co Ltd
Priority date: 2022-03-11
Filing date: 2022-03-11
Publication date: 2023-09-19

Abstract

The application provides a network data processing method, a network data processing device, an electronic device, a computer readable storage medium and a computer program product; the method comprises the following steps: acquiring a network byte stream for deserialization; when the data type of the network byte stream is a table type, performing table element statistical processing on the network byte stream to obtain the element number of the table to be deserialized; performing analysis processing of a triplet message format based on the element number of the table to be reverse-sequenced to obtain triplet message information of the table to be reverse-sequenced; performing pre-allocation processing of the storage space based on the element number included in the triplet message information; and performing deserialization processing on the table to be deserialized based on the storage space obtained by pre-allocation to obtain a deserialized table. According to the application, the deserialization performance can be improved.

Description

Network data processing method, device, equipment, storage medium and program product

Technical Field

The present application relates to computer data processing technology, and in particular, to a network data processing method, apparatus, electronic device, computer readable storage medium, and computer program product.

Background

With the development of computer technology, applications (apps) run by electronic devices are becoming more and more diversified, and the advent of various apps can meet various demands of users, for example, using game apps can enable users to relieve life pressure in the process of games; the use of a live App enables users to interact more effectively.

In order to ensure the integrity and transitability of an object, when the object (e.g., table) is transferred and saved, the object needs to be converted into an ordered byte stream through serialization in order to transmit data over a network or save the data into a local file. When the object is rebuilt by unpacking, the ordered byte stream is converted into the object by deserializing according to the object state and the description information stored in the byte stream.

In the related art, in the process of deserialization, each table is constructed by first creating an empty table, then gradually inserting data, and continuously triggering a rehash operation (rehash) to expand the capacity, so that deserialization performance is poor, and a large amount of storage space and computing resources are wasted.

Disclosure of Invention

Embodiments of the present application provide a network data processing method, apparatus, electronic device, computer readable storage medium, and computer program product, which can improve deserialization performance.

The technical scheme of the embodiment of the application is realized as follows:

the embodiment of the application provides a network data processing method, which comprises the following steps:

acquiring a network byte stream for deserialization;

when the data type of the network byte stream is a table type, performing table element statistical processing on the network byte stream to obtain the element number of a table to be inversely sequenced;

performing analysis processing of a triplet message format based on the element number of the table to be de-sequenced to obtain triplet message information of the table to be de-sequenced;

performing pre-allocation processing of storage space based on the element number included in the triplet message information;

and performing deserialization processing on the table to be deserialized based on the storage space obtained by pre-allocation to obtain a deserialized table.

An embodiment of the present application provides a network data processing apparatus, including:

the acquisition module is used for acquiring a network byte stream for deserialization;

the statistics module is used for carrying out table element statistics processing on the network byte stream when the data type of the network byte stream is a table type, so as to obtain the element number of the table to be inversely sequenced;

The analysis module is used for carrying out analysis processing of a triplet message format based on the element number of the table to be deserialized to obtain triplet message information of the table to be deserialized;

the distribution module is used for carrying out pre-distribution processing on the storage space based on the element number included in the triplet message information;

and the deserialization module is used for carrying out deserialization processing on the table to be deserialized based on the storage space obtained by pre-allocation to obtain a deserialized table.

In the above technical solution, the obtaining module is further configured to receive, through an application layer, a network byte stream for deserialization;

before the pre-allocation processing of the storage space is performed based on the element number included in the triplet message information, the parsing module is further configured to create an intermediate buffer layer in the application layer;

caching the triple message information of the table to be reverse-sequenced in the middle buffer layer;

the allocation module is further configured to combine the element number included in the triplet message information of the table to be deserialized, and perform pre-allocation processing on the storage space for the table to be deserialized through a virtual machine layer, so as to obtain the storage space allocated by the table to be deserialized.

In the above technical solution, the statistics module is further configured to start performing table element statistics processing on a data entry after an arbitrary data entry when a type of the arbitrary data entry in the network byte stream is a start identifier of a table type;

and when the type of the data entry after any data entry is the end identification of the table type, ending the table element statistical processing, and taking the corresponding element number when the table element statistical processing is ended as the element number of the table to be deserialized.

In the above technical solution, the statistics module is further configured to perform the following processing for a data entry after the arbitrary data entry:

when continuously analyzing to a key field entry and a value field entry, adding 1 to the element number corresponding to the table element statistical processing.

In the above technical solution, the allocation module is further configured to execute, by using the virtual machine layer, the following processing:

receiving a buffer pointer sent by the application layer, wherein the buffer pointer represents the address of the triple message information in the middle buffer layer;

determining the element number in the triple message information through the buffer pointer;

And performing pre-allocation processing on the storage space based on the element number included in the triple message information to obtain the storage space allocated by the table to be de-sequenced.

In the above technical solution, the parsing module is further configured to perform a conversion process on the network byte stream in a triplet message format, so as to obtain the triplet message format of the network byte stream;

wherein, when the data structure of the data entry of the table type is an array or a hash table, the triplet message format includes a type field, a length field, and a value field, the length field includes a first byte for characterizing the data structure, and a second byte for the number of elements of the table type;

setting the value of the type field to be a value representing the type of the table, setting the value of the value field to be a data entry of the table to be deserialized, setting the value of the first byte to be a value representing the data type of the data entry of the table to be deserialized, and setting the value of the second byte to be the number of elements of the table to be deserialized, wherein the data type of the data entry of the table to be deserialized is the array or the hash table.

In the above technical solution, when the data types of the data entries of the table to be deserialized include an array and a hash table, the deserializing module is further configured to perform deserializing processing on the data entries of the first data type included in the table to be deserialized based on the pre-allocated storage space, to obtain a partial table;

the first data type is a first data type in the table to be deserialized, and the first data type is any one of the array and the hash table;

and performing anti-serialization processing based on re-hash on the data items of a second data type included in the table to be anti-serialized based on the part table to obtain the table after anti-serialization, wherein the second data type is the other one of the array and the hash table.

In the above technical solution, when the data structure of the table type data entry includes an array and a hash table, the length field includes a first sub-length field and a second sub-length field;

wherein the first sub-length field includes a third byte for characterizing the array and a fourth byte for the number of elements of the array, and the second sub-length field includes a fifth byte for characterizing the hash table and a sixth byte for the number of elements of the hash table;

The parsing module is further configured to set the value of the fourth byte to the number of elements of an array included in the table to be deserialized, and set the value of the sixth byte to the number of elements of a hash table included in the table to be deserialized.

In the above technical solution, the element number included in the triplet message information includes the element number of the array included in the table to be deserialized and the element number of the hash table included in the table to be deserialized; the allocation module is further used for performing pre-allocation processing on the storage space based on the element number of the array included in the table to be de-sequenced to obtain a first storage space;

performing pre-allocation processing on the storage space based on the element number of the hash table included in the table to be de-serialized to obtain a second storage space;

the anti-serialization module is further configured to perform anti-serialization processing on an array included in the table to be anti-serialized based on the first storage space, so as to obtain a first part table;

performing deserialization processing on the hash table included in the table to be deserialized based on the second storage space to obtain a second partial table;

and combining the first part table and the second part table to obtain the anti-sequenced table.

In the above technical solution, when the data type of the network byte stream is a non-table type, the anti-serialization module is further configured to perform analysis processing of a triplet message format on the network byte stream to obtain triplet message information of the table to be anti-serialized;

performing deserialization processing on the triplet message information to obtain the deserialized table;

wherein the non-table type comprises Boolean type, integer type, character string type and numerical value type.

In the above technical solution, the anti-serialization module is further configured to create the table to be anti-serialized based on the storage space obtained by the pre-allocation, so as to obtain an empty table;

and carrying out assignment processing on the empty table based on key data items and value data items included in the triple message information to obtain the table after reverse serialization.

An embodiment of the present application provides an electronic device for network data processing, including:

a memory for storing executable instructions;

and the processor is used for realizing the network data processing method provided by the embodiment of the application when executing the executable instructions stored in the memory.

The embodiment of the application provides a computer readable storage medium which stores executable instructions for causing a processor to execute, thereby realizing the network data processing method provided by the embodiment of the application.

An embodiment of the present application provides a computer program product, including a computer program or an instruction, which is characterized in that the computer program or the instruction implement the network data processing method provided by the embodiment of the present application when being executed by a processor.

The embodiment of the application has the following beneficial effects:

by counting the element number of the table to be deserialized in advance and pre-distributing the storage space based on the element number included in the triplet message information obtained by analysis, deserialization processing is performed based on the pre-distributed storage space, and a re-hash operation is not required to be triggered, so that the deserialization performance is improved, and compared with a scheme of continuously triggering the re-hash operation, the related storage space and calculation resources are saved.

Drawings

Fig. 1 is a schematic diagram of an application mode of a network data processing method according to an embodiment of the present application;

fig. 2 is a schematic structural diagram of an electronic device for network data processing according to an embodiment of the present application;

fig. 3 to fig. 5 are schematic flow diagrams of a network data processing method according to an embodiment of the present application;

Fig. 6 is a flowchart of a packet receiving and transmitting and packet opening operation provided in the related art;

FIG. 7 is a schematic diagram of a table construction flow provided by the related art;

FIG. 8 is a block diagram of an optimization algorithm provided by an embodiment of the present application;

fig. 9 is a schematic diagram of a triplet message (TLV) format provided by an embodiment of the present application;

FIG. 10 is a schematic diagram of a data structure of an intermediate buffer layer according to an embodiment of the present application;

FIG. 11 is a schematic flow chart of the lua_closed batch creation provided by the embodiment of the application;

FIG. 12 is a schematic diagram of a table data structure provided by an embodiment of the present application;

FIG. 13 is a graph showing a comparison of performance provided by an embodiment of the present application.

Detailed Description

The present application will be further described in detail with reference to the accompanying drawings, for the purpose of making the objects, technical solutions and advantages of the present application more apparent, and the described embodiments should not be construed as limiting the present application, and all other embodiments obtained by those skilled in the art without making any inventive effort are within the scope of the present application.

In the following description, the terms "first", "second", and the like are merely used to distinguish between similar objects and do not represent a particular ordering of the objects, it being understood that the "first", "second", or the like may be interchanged with one another, if permitted, to enable embodiments of the application described herein to be practiced otherwise than as illustrated or described herein.

Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of the application only and is not intended to be limiting of the application.

Before describing embodiments of the present application in further detail, the terms and terminology involved in the embodiments of the present application will be described, and the terms and terminology involved in the embodiments of the present application will be used in the following explanation.

1) In response to: for representing a condition or state upon which an operation is performed, one or more operations performed may be in real-time or with a set delay when the condition or state upon which the operation is dependent is satisfied; without being specifically described, there is no limitation in the execution sequence of the plurality of operations performed.

2) Client side: applications running in the terminal for providing various services, such as a video play client, a game client, etc.

3) Virtual scene: the game program displays (or provides) a virtual game scene when running on the terminal. The virtual scene may be a simulation environment for the real world, a semi-simulation and semi-fictional virtual environment, or a pure fictional virtual environment. The virtual scene may be any one of a two-dimensional virtual scene, a 2.5-dimensional virtual scene or a three-dimensional virtual scene, and the dimension of the virtual scene is not limited in the embodiment of the present application. For example, a virtual scene may include sky, land, sea, etc., the land may include environmental elements of a desert, city, etc., and a user may control a virtual object to move in the virtual scene.

4) Virtual object: the avatars of various people and objects in the virtual scene that can interact with, or movable objects in the virtual scene. The movable object may be a virtual character, a virtual animal, a cartoon character, etc., such as a character, an animal, etc., displayed in a virtual scene. The virtual object may be a virtual avatar in a virtual scene for representing a user. A virtual scene may include a plurality of virtual objects, each virtual object having its own shape and volume in the virtual scene, occupying a portion of space in the virtual scene.

5) Scene data: the feature data representing the virtual scene may be, for example, an area of a building area in the virtual scene, a building style in which the virtual scene is currently located, and the like; and may also include the location of the virtual building in the virtual scene, the footprint of the virtual building, etc.

6) Lua: a compact scripting language is used for server-side logic function development. Lua is written in standard C and can be compiled and run on almost all operating systems and platforms. LUA provides only one general type of table (table) with which arrays (array), hash tables (hash), collections, objects can be implemented. Some key mechanisms required by object-oriented programming, such as data abstraction, virtual functions, inheritance, reloading, etc., can be conveniently supported through closures and tables.

7) Remote procedure call (RPC, remote Procedure Call): the embodiment of the application is a set of remote procedure call mechanism based on Lua, and the bottom layer automatically performs function parameters, serialization (converting a table into a character string) operation and deserialization (converting the character string into the table) operation. Wherein the serialization functions as follows: when delivering and saving an object (table), the integrity and transitability of the object are guaranteed, and the object is converted into an ordered byte stream so as to transmit data on a network or save the data into a local file. The effect of the anti-serialization is as follows: and reconstructing the object through deserialization according to the object state and the description information stored in the byte stream.

8) Protocol Buffering (PB): a data description language developed by Google corporation, similar to the extensible markup language (XML, extensible Markup Language), can serialize structured data, can be used in data storage, communication protocols and the like, and supports three programming languages of C++, JAVA, python and the like.

9) Triple format (TLV, type-Length-Value): a data coding mode adopts a triplet of Type Length Value and an entry in a triplet format comprises Type, length, value.

10 Re-hash (rehash): a dynamic capacity expansion mode of a table memory in a Lua virtual machine is used for carrying out capacity expansion based on 2 times of the current memory each time.

The table in Lua includes an array portion storing data indexed as integer and a hash table portion storing key-value pairs indexed by other types. The hash table uses a hash algorithm to calculate key values (keys) of key-value pairs, and uses an open-chain method to handle hash collisions. When a table t= { } is declared, the size of the array part and the hash table part of t are both 0, and when key pairs are continuously inserted into the array part, the following is true: 1. inserting a [1] =1, wherein the array part is not enough in size and needs to be expanded, and the size of the array part after expansion is 1; 2. inserting a [2] =2, wherein the array part is not enough in size and needs to be expanded, and the size of the array part after expansion is 2; 3. inserting a [3] =3, the array part is not enough in size, expansion is needed, and the array size after expansion is 4. The capacity expansion process is rehash, each rehash traverses the array part and the hash table part of the whole table, and the effective key value pairs are counted, when the size is insufficient, the capacity is expanded, the expanded size is the integer power of 2, and the utilization rate of the whole array part or the hash table part after rehash operation is ensured to be more than 50%. Each rehash is time consuming and should be minimized.

It should be noted that the embodiment of the present application may be applied to various application scenarios, such as a game scenario, a live broadcast scenario, a short video scenario, an online shopping scenario, etc. The following will take a game scene as an example:

embodiments of the present application provide a network data processing method, apparatus, electronic device, computer readable storage medium, and computer program product, which can improve deserialization performance. In order to facilitate easier understanding of the network data processing method provided by the embodiment of the present application, first, an exemplary implementation scenario of the network data processing method provided by the embodiment of the present application is described, where a virtual scenario (implemented by deserializing network data) in the network data processing method provided by the embodiment of the present application may be output cooperatively based on a terminal and a server, or may be output cooperatively based on a server and a server.

In some embodiments, the virtual scene may be an environment for interaction of game characters, for example, the game characters may fight in the virtual scene, and both parties may interact in the virtual scene by controlling actions of the game characters, so that a user can relax life pressure in the game process.

In an implementation scenario, referring to fig. 1, fig. 1 is a schematic diagram of an application mode of a network data processing method according to an embodiment of the present application, which is applied to a terminal 400 and a server 200, and is suitable for an application mode that completes virtual scene calculation depending on a computing capability of the server 200 and outputs a virtual scene at the terminal 400.

Taking the example of forming the visual perception of the virtual scene 100, the server 200 performs calculation of virtual scene related display data (such as a network byte stream of scene data and the like) and sends the calculated display data to the terminal 400 through the network 300, the terminal 400 finishes loading, analyzing and rendering of the calculated display data depending on graphic calculation hardware, and outputs the virtual scene depending on graphic output hardware to form the visual perception, for example, a two-dimensional video frame can be presented on a display screen of a smart phone, or a video frame for realizing a three-dimensional display effect can be projected on a lens of an augmented reality/virtual reality glasses; as regards the perception of the form of the virtual scene, it will be appreciated that it is possible to form an auditory perception by means of the corresponding hardware output of the terminal 400, for example using a microphone, a tactile perception by means of a vibrator, etc.

As an example, the terminal 400 is provided with a client 410 (e.g., a web-based game application) running thereon, and performs game interaction with other users through the connection server 200 (e.g., a game server), the terminal 400 outputs a virtual scene of the client 410 (by performing deserialization on a network byte stream such as scene data), and the virtual scene 100 is displayed in a first-person perspective, and the virtual object 110 is displayed in the virtual scene 100, and the virtual object 110 may be artificial intelligence set in the virtual scene through training, and the virtual object 110 may walk in the virtual scene, and may also remain stationary, jump in place, and use various functions (such as skills and props); virtual object 110 may be a game character under the control of a user (or player) that will operate in a virtual scene in response to manipulation of buttons (including rocker buttons, attack buttons, defense buttons, etc.) by a real user, e.g., when the real user moves the rocker buttons to the left, the virtual object will move to the left in the virtual scene, and may remain stationary in place, jump, and use various functions (e.g., skills and props); virtual object 110 may also be a non-user character set in the virtual scene interaction.

For example, taking the first-person view to display the virtual scene 100 as an example, the terminal 400 outputs a virtual scene of the client 410, where the virtual scene is obtained by obtaining a network byte stream in a transmission process, and when a data type of the network byte stream is a table type, performing table element statistics processing on the network byte stream to obtain the number of elements of a table to be de-sequenced, performing parsing processing in a triplet message format based on the number of elements of the table to be de-sequenced, obtaining triplet message information of the table to be de-sequenced, performing pre-allocation processing of a storage space based on the number of elements included in the triplet message information, performing de-sequencing processing on the table to be de-sequenced based on the pre-allocated storage space to obtain a de-sequenced table, where the de-sequenced table includes various data in the virtual scene, for example, for a game gate card system, where the de-sequenced table includes various data (for example, a jump position, a virtual object position, a jump speed, etc.), so that the virtual object 110 is displayed in the virtual scene 100, and the jump speed of the virtual object 110 is controlled to be performed to a jump to a high difficulty level 120.

In some embodiments, the terminal 400 may implement the service processing method provided by the embodiment of the present application by running a computer program, for example, the computer program may be a native program or a software module in an operating system; may be a local (Native) application (APP, APPlication), i.e., a program that needs to be installed in an operating system to run, such as a reload game APP (i.e., client 410 described above); the method can also be an applet, namely a program which can be run only by being downloaded into a browser environment; but also a game applet that can be embedded in any APP. In general, the computer programs described above may be any form of application, module or plug-in.

Taking a computer program as an application program as an example, in actual implementation, the terminal 400 installs and runs an application program supporting a virtual scene. The application may be any one of a First person shooter game (FPS), a third person shooter game, a virtual reality application, a three-dimensional map program, or a multiplayer gunfight survival game. The user uses the terminal 400 to operate virtual objects located in a virtual scene to perform activities including, but not limited to: at least one of body posture adjustment, crawling, walking, running, riding, jumping, driving, picking up, shooting, attacking, throwing, building a virtual building. Illustratively, the virtual object may be a virtual character, such as an emulated persona or a cartoon persona, or the like.

In some embodiments, the embodiments of the present application may also be implemented by means of Cloud Technology (Cloud Technology), which refers to a hosting Technology that unifies serial resources such as hardware, software, networks, etc. in a wide area network or a local area network, so as to implement calculation, storage, processing, and sharing of data.

The cloud technology is a generic term of network technology, information technology, integration technology, management platform technology, application technology and the like based on cloud computing business model application, can form a resource pool, and is flexible and convenient as required. Cloud computing technology will become an important support. Background services of technical network systems require a large amount of computing and storage resources.

For example, the server 200 in fig. 1 may be an independent physical server, a server cluster or a distributed system formed by a plurality of physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDNs, and basic cloud computing services such as big data and artificial intelligence platforms. The terminal 400 may be, but is not limited to, a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, etc. The terminal 400 and the server 200 may be directly or indirectly connected through wired or wireless communication, which is not limited in the embodiment of the present application.

Referring to fig. 2, fig. 2 is a schematic structural diagram of an electronic device for network data processing according to an embodiment of the present application, and the electronic device is taken as a server 200 for illustration, and the electronic device 400 shown in fig. 2 includes: at least one processor 420, a memory 460, at least one network interface 430, and a user interface 440. The various components in terminal 400 are coupled together by a bus system 450. It is understood that bus system 450 is used to implement the connected communications between these components. The bus system 450 includes a power bus, a control bus, and a status signal bus in addition to a data bus. But for clarity of illustration the various buses are labeled as bus system 450 in fig. 2.

The processor 420 may be an integrated circuit chip with signal processing capabilities such as a general purpose processor, such as a microprocessor or any conventional processor, or the like, a digital signal processor (DSP, digital Signal Processor), or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or the like.

The user interface 440 includes one or more output devices 441 that enable presentation of media content, including one or more speakers and/or one or more visual displays. The user interface 440 also includes one or more input devices 442, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

Memory 460 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid state memory, hard drives, optical drives, and the like. Memory 460 optionally includes one or more storage devices physically remote from processor 420.

Memory 460 includes volatile memory or nonvolatile memory, and may also include both volatile and nonvolatile memory. The nonvolatile Memory may be a Read Only Memory (ROM), and the volatile Memory may be a random access Memory (RAM, random Access Memory). The memory 460 described in embodiments of the present application is intended to comprise any suitable type of memory.

In some embodiments, memory 460 is capable of storing data to support various operations, examples of which include programs, modules and data structures, or subsets or supersets thereof, as exemplified below.

An operating system 461 including system programs for handling various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and handling hardware-based tasks;

network communication module 462 for reaching other computing devices via one or more (wired or wireless) network interfaces 430, the exemplary network interfaces 430 comprising: bluetooth, wireless compatibility authentication (WiFi), and universal serial bus (USB, universal Serial Bus), etc.;

A presentation module 463 for enabling presentation of information (e.g., a user interface for operating peripheral devices and displaying content and information) via one or more output devices 441 (e.g., a display screen, speakers, etc.) associated with the user interface 440;

an input processing module 464 for detecting one or more user inputs or interactions from one of the one or more input devices 442 and translating the detected inputs or interactions.

In some embodiments, the network data processing apparatus provided by the embodiments of the present application may be implemented in software, and fig. 2 shows the network data processing apparatus 465 stored in the memory 460, which may be software in the form of programs and plug-ins, including the following software modules: the acquisition module 4651, statistics module 4652, parsing module 4653, assignment module 4654, and de-serialization module 4655 are logical, and thus may be arbitrarily combined or further split depending on the functions implemented.

In other embodiments, the network data processing apparatus provided by the embodiments of the present application may be implemented in hardware, and by way of example, the network data processing apparatus provided by the embodiments of the present application may be a processor in the form of a hardware decoding processor that is programmed to perform the network data processing method provided by the embodiments of the present application, for example, the processor in the form of a hardware decoding processor may employ one or more application specific integrated circuits (ASIC, application Specific Integrated Circuit), DSP, programmable logic device (PLD, programmable Logic Device), complex programmable logic device (CPLD, complex Programmable Logic Device), field programmable gate array (FPGA, field-Programmable Gate Array), or other electronic components.

The network data processing method provided by the embodiment of the application is specifically described below with reference to the accompanying drawings. The network data processing method provided by the embodiment of the present application may be cooperatively executed by the terminal 400 and the server 200 in fig. 1. Referring to fig. 3, fig. 3 is a flowchart of a network data processing method according to an embodiment of the present application, and will be described with reference to the steps shown in fig. 3.

It should be noted that the method shown in fig. 3 may be executed by various computer programs running on the terminal 400, and is not limited to the above client 410, but may also be the operating system 461, software modules and scripts described above, and therefore the client should not be considered as limiting the embodiments of the present application.

In step 101, a network byte stream for deserialization is acquired.

It should be noted that, during the process of running an application program, a large amount of network data needs to be transferred, and when a table (table) is transferred and saved, in order to ensure the integrity and transitability of an object, the table needs to be converted into a network byte stream through serialization so as to transmit the data on a network or save the data into a local file. When the table is rebuilt by unpacking, the network byte stream is converted into the table by deserializing according to the object state and the description information stored in the network byte stream.

In the embodiment of the application, when the table is rebuilt by unpacking, the network byte stream for deserialization is received through an application layer (realized through a C++ layer) to perform deserialization processing on the network byte stream in 1011 steps shown in fig. 4.

In step 102, when the data type of the network byte stream is a table type, performing table element statistics processing on the network byte stream to obtain the element number of the table to be deserialized.

It should be noted that, in the related art in the c++ layer reverse serialization process, when the data type of the network byte stream is a table type, immediately calling the lua_new function to create a blank table, and then gradually inserting a continuous key value parameter (key) and a value parameter (value) (simply referred to as a key value pair, i.e., KV) into the blank table, so as to continuously trigger a rehash (rehash) operation to expand the capacity, wherein the rehash operation is to reallocate a larger memory, copy old data to the newly allocated memory, then release the old memory, and the background service unpacking operation is a high-frequency action, which results in extremely high rehash calling frequency and low deserialization performance.

In order to solve the problems of the related art, the embodiment of the application suspends the process of creating the empty table, but performs table element statistics processing on the network byte stream to obtain the element number of the table to be deserialized, so that the memory of the table to be deserialized is allocated in advance based on the element number of the table to be deserialized, thereby avoiding rehash operation and improving deserialization performance.

In some embodiments, the process of table element statistical processing is as follows: when the type of any data entry in the network byte stream is a start identification of a table type, starting to perform table element statistical processing on the data entry after any data entry; and when the type of the data entry after any data entry is the end identification of the table type, ending the table element statistical processing, and taking the corresponding element number when ending the table element statistical processing as the element number of the table to be deserialized.

For example, in the process of parsing table type data in a network byte stream, there are table_head type data (which characterizes the start of a table, i.e. the start identifier of a table type) and table_tail type data (which characterizes the end of a table, i.e. the end identifier of a table type), when parsing to a table_head (i.e. any data entry (i.e. field) in the network byte stream is the start identifier of a table type), counting is performed by a variable num until parsing to a table_tail (i.e. the type of a data entry following any data entry is the end identifier of a table type), and then counting is finished, and then num at the end of counting represents the actual element number ElementNum of the table to be de-sequenced.

In some embodiments, starting table element statistical processing of data entries subsequent to any data entry includes: the following processing is performed for a data entry following any data entry: when continuously resolving to a key field entry and a value field entry, adding 1 to the element number corresponding to the table element statistical processing.

For example, in the counting process, by counting through a variable num, where a pair of key data (key field entry) and value data (value field entry)) is continuously parsed, i.e., a key field entry is parsed first, and then a value field entry is parsed, the num count is incremented by 1 until the table_tail is parsed.

In step 103, the analysis processing of the triplet message format is performed based on the element number of the table to be de-serialized, so as to obtain the triplet message information of the table to be de-serialized.

For example, after performing table element statistics processing on the network byte stream, obtaining the element number of the table to be inversely sequenced, writing the element number of the table to be inversely sequenced into a triplet message format (TLV) to obtain triplet message information of the table to be inversely sequenced, and when a table is subsequently created, pre-distributing memory based on the element number in the triplet message information, so that table rehash operation is avoided, and the anti-serialization performance is improved.

As shown in fig. 4, before the pre-allocation processing of the storage space based on the number of elements included in the triplet message information, steps 106 to 107 are further included: in step 106, an intermediate buffer layer is created in the application layer; in step 107, the triple message information of the table to be de-serialized is buffered in the intermediate buffer layer.

For example, by creating an intermediate buffer layer (i.e. a buffer area different from the virtual machine layer), triple message information of a table to be deserialized is temporarily stored in the intermediate buffer layer, so that a table is prevented from being directly created by the LUA virtual machine, a table rehash operation is avoided, and deserialization performance is improved.

Referring to fig. 5, fig. 5 is a schematic flow chart of an alternative network data processing method according to an embodiment of the present application, and fig. 5 shows that step 103 in fig. 3 may be implemented by steps 1031 to 1032: in step 1031, performing a conversion process on the network byte stream to obtain a triplet message format of the network byte stream, where when the data structure of the data entry of the table type is an array or a hash table, the triplet message format includes a type field, a length field, and a value field, and the length field includes a first byte for representing the data structure and a second byte for representing the number of elements of the table type; in step 1032, the value of the type field is set to a value characterizing the type of the table, the value of the value field is set to a data entry of the table to be de-serialized, the value of the first byte is set to a value characterizing the data type of the data entry of the table to be de-serialized, the value of the second byte is set to the number of elements of the table to be de-serialized, the data type of the data entry of the table to be de-serialized is an array or a hash table.

For example, in the embodiment of the present application, format conversion is performed by creating an intermediate buffer layer, which is a continuous memory, and different types of data are stored continuously. For different types of data, TLV mode encoding is adopted, and meanwhile, some optimization is carried out on the table type so as to facilitate the identification of the Lua virtual machine.

It should be noted that, although the table may include two parts of an array (array) and a hash table (hash), the table during data transmission generally includes only the array or the hash, and there are few cases where the array and the hash coexist, so only a 1-bit tag type (i.e., the first byte is used to characterize the array or the hash table) is required.

For example, as shown in fig. 9, the value of the type field (type) in the TLV is set to a value (e.g. "1" for characterizing a table type), the value of the value field (value) in the TLV is set to a data entry (containing elem_num key value pairs as shown in fig. 9) of the table to be deserialized, the value of the first byte is set to a value (e.g. "1" for characterizing an array type, "0" for characterizing a hash table type) for characterizing a data entry of the table to be deserialized, and the value of the second byte is set to the number of elements of the table to be deserialized (i.e. the actual number of elements ElementNum of the table to be deserialized), thereby constructing the triplet message information of the table to be deserialized.

In some embodiments, when the data types of the data items of the table to be deserialized include an array and a hash table, performing pre-allocation processing on the storage space based on the number of elements included in the triple message information, and performing deserialization processing on the data items of the first data type included in the table to be deserialized based on the storage space obtained by pre-allocation to obtain a part of the table; the first data type is the first data type in a table to be deserialized, and the first data type is any one of an array and a hash table; and performing anti-serialization processing based on the rehashed data items of a second data type included in the table to be deserialized based on the partial table to obtain a deserialized table, wherein the second data type is the other one of the array and the hash table.

For example, when the first data type in the table to be deserialized is an array, performing table element statistics processing on the array in the network stream to obtain the element number of the array in the table to be deserialized, performing ternary message format analysis processing on the basis of the element number of the array in the table to be deserialized to obtain ternary message information of the table to be deserialized, wherein the value of the first byte in the ternary message information is a value representing the array, the value of the second byte is the element number of the array, performing pre-allocation processing on the basis of the element number included in the ternary message information, performing deserialization processing on the data items (including key fields and value fields) of the array included in the table to be deserialized based on the pre-allocation obtained storage space, obtaining a partial table, performing hash processing on the data items of the hash table included in the table to be deserialized based on the partial table to obtain the deserialized table, and performing a restore hash operation in the hash processing on the data items of the table based on the hash table to be deserialized. Therefore, for the situation that the table to be deserialized comprises an array and a hash table, the number of elements of the array or the hash table which appears for the first time can be recorded by one byte, memory is pre-allocated based on the recorded number of elements, a table is created based on the pre-allocated memory, and when the memory of the table is small in the deserialization process, the capacity expansion is performed in a rehash mode to continue deserialization.

In some embodiments, when the data structure of the data entry of the table type includes an array and a hash table, the length field includes a first sub-length field and a second sub-length field; wherein the first sub-length field includes a third byte for characterizing the array and a fourth byte for the number of elements of the array, and the second sub-length field includes a fifth byte for characterizing the hash table and a sixth byte for the number of elements of the hash table; the value of the fourth byte is set to the number of elements of the array comprised by the table to be de-serialized, and the value of the sixth byte is set to the number of elements of the hash table comprised by the table to be de-serialized.

For example, for the extreme case that the table to be deserialized includes an array and a hash table, two fields are set for the length field in the TLV format, namely, a first sub-length field including a third byte for characterizing the array and a fourth byte for characterizing the number of elements of the array, and a second sub-length field including a fifth byte for characterizing the hash table and a sixth byte for characterizing the number of elements of the hash table, namely, the number of elements included in the triplet message information includes the number of elements of the array included in the table to be deserialized and the number of elements of the hash table included in the table to be deserialized. And pre-allocating the storage space of the array based on the fourth byte in the first sub-length field to obtain a first storage space, pre-allocating the storage space of the hash table based on the sixth byte in the second sub-length field to obtain a second storage space, performing anti-serialization processing on the array included in the table to be anti-serialized based on the first storage space to obtain a first part table, performing anti-serialization processing on the hash table included in the table to be anti-serialized based on the second storage space to obtain a second part table, and combining the first part table and the second part table to obtain the table after anti-serialization.

In step 104, a pre-allocation process of the storage space is performed based on the number of elements included in the triplet message information.

For example, the element number of the table to be deserialized is written into a triplet message format (TLV), after the triplet message information of the table to be deserialized is obtained, when the table is subsequently created, memory is pre-allocated based on the element number in the triplet message information, so that table rehash operation is avoided, and deserialization performance is improved. The storage space obtained by pre-allocation is larger than or equal to the element number included in the triple message information.

In some embodiments, the pre-allocation processing of the storage space is performed on the table to be de-serialized through the virtual machine layer in combination with the element number included in the triple message information of the table to be de-serialized, so as to obtain the storage space allocated by the table to be de-serialized.

For example, by creating the intermediate buffer layer to temporarily store the triplet message information, the direct creation of a table through the Lua virtual machine is avoided, the table rehash operation is avoided, and the deserialization performance is improved.

In some embodiments, in combination with the number of elements in the triple message information of the table to be deserialized, performing, by the virtual machine layer, pre-allocation processing of a storage space for the table to be deserialized, to obtain the storage space allocated by the table to be deserialized, including: the following processing is performed by the virtual machine layer: receiving a buffer pointer sent by an application layer, wherein the buffer pointer represents the address of the triple message information in an intermediate buffer layer; determining the element number in the triple message information through a buffer pointer; and performing pre-allocation processing on the storage space based on the element number included in the triplet message information to obtain the storage space allocated by the table to be de-sequenced.

For example, the intermediate buffer layer data is pushed to the lua virtual machine layer through a pointer (i.e. a buffer pointer) of a block of data, based on the buffer pointer triplet message information, the pre-allocation processing of the storage space is performed based on the element number included in the triplet message information, and the storage space allocated by the table to be de-sequenced is obtained. The data pushing is realized only through the pointer, no consumption of data copying is caused, and the storage space is saved.

In step 105, the table to be deserialized is deserialized based on the memory space obtained by the pre-allocation, and the deserialized table is obtained.

For example, after storage space is pre-allocated based on the element number included in the triplet message information, when a table is subsequently created, a table to be deserialized is deserialized based on the storage space obtained by pre-allocation, so that table rehash operation is avoided, and deserialization performance is improved.

In some embodiments, when the data type of the network byte stream is a non-table type, performing analysis processing of a triple message format on the network byte stream to obtain triple message information of a table to be de-sequenced; performing deserialization processing on the triplet message information to obtain a deserialized table; the non-table type comprises Boolean type, integer type, character string type and numerical value type.

For example, since the space occupation of data types such as boolean types, integer types, character string types, numerical types and the like is small, excessive rehash operation is not needed, pre-allocation of space is not needed, and the calculation resource consumption in the statistics process of the intermediate buffer layer is avoided.

In some embodiments, performing deserialization processing on a table to be deserialized based on the memory space obtained by pre-allocation to obtain a deserialized table, including: creating a table to be deserialized based on the storage space obtained by pre-allocation to obtain an empty table; and carrying out assignment processing on the empty table based on key data items and value data items included in the triple message information to obtain a deserialized table.

For example, a table is created through a lua_createtable (lua_state_l, int narry, int nrec) and memory is pre-allocated, wherein the lua_createtable is an application program interface (API, application Programming Interface) newly added at the bottom layer of the Lua virtual machine, the lua_createtable can analyze TLV formats of intermediate buffer layers and create the table in batches, when the table is created, the number of elements in the TLV formats is utilized to perform memory pre-allocation, the size of the created empty table is the storage space obtained by pre-allocation, assignment processing is performed on the empty table based on key data entries and value data entries included in the triplet message information, and a table after deserialization is obtained, namely, the key data entries (key data) in the triplet message information are analyzed through a decode_key function, and the value data entries (value data) in the triplet message information are analyzed through a decode_value function.

In the following, an exemplary application of an embodiment of the present application in a practical game application scenario will be described.

In the related art, the background process framework is a combined mode of C++ and Lua. The C++ layer is responsible for network packet receiving and transmitting and packet opening operation, and the specific flow is shown in figure 6, and the 1, C++ layer receives network byte streams; 2. the c++ layer calls remote procedure call (RPC, remote Procedure Call) functions, deserializing network packets (network byte streams); 3. if the parameter is a table type, calling a lua_new function in the C++ layer to construct a table in the Lua virtual machine; 4. the Lua layer RPC function interface is called by lua_pcall.

In the reverse serialization process of the c++ layer, as shown in fig. 7, when the parameter is a table, the lua_new function is called to create a blank table, and every two pieces of data are the key and the value of the table respectively, and then the data are gradually inserted into the lua blank table. Therefore, each table is constructed by creating an empty table and then gradually inserting a single piece of data, and continuously triggering a table re-hash (rehash) operation to expand the capacity. the step of table rehash is to reallocate a larger block of memory, copy old data to the newly allocated memory, and then release the old memory. Therefore, the rehash is a very time-consuming operation, and the background service unpacking operation is a high-frequency action, so that the table rehash calling frequency is extremely high, and the program performance is low.

In order to solve the above problems, the embodiment of the application provides an unpacking optimization algorithm for reducing Lua table rehash. Through the optimization algorithm, the table rehash operation at the bottom layer of the Lua virtual machine can be greatly reduced when the network data packet is received in reverse sequence, and the program performance is greatly optimized. When the C++ layer receives the network byte stream, the optimization algorithm does not call the lua_newtable & lua_setable function immediately to construct a table in the Lua virtual machine (a large number of table rehash operations can be generated in the step), but inserts an intermediate buffer layer in the C++ layer, temporarily stores the pre-unpacked intermediate data in a TLV format into the intermediate buffer layer, calculates the element number of each table, and finally establishes the table in batches in the Lua virtual machine through the data of the intermediate buffer layer, and realizes memory pre-allocation by using the element num in the process of establishing the table in batches, so that the table rehash operation can be greatly reduced, the performance is improved, and the unpacking performance is finally improved by about 60%.

It should be noted that the present solution belongs to a pure background solution, is used for a reverse-serialization scenario of a network byte stream, and can be applied to a messaging packet scenario of a bottom Client/Server (C2S) and a Server/Server (S2S). The optimization algorithm performs table element pre-statistics on the C++ layer through the intermediate buffer layer, so that the rehash times of the Lua virtual machine are reduced.

In summary, the core scheme of the optimization algorithm is to reduce the rehash triggering frequency when unpacking and constructing the table. Pre-allocating memory for the table, and performing rehash only when the pre-allocated memory is not enough, and allocating 2 times of the original memory. Therefore, at the time of C++ layer unpacking (decode), the creation of the table is delayed, and the number of elements of the current table is synchronously calculated (the array part and the hash table part are separately calculated). Therefore, when the table is created in the later period, enough memory can be allocated in advance, and the subsequent insertion of the lua_setfield (used for inserting the data on the lua stack into the table) into the table element is a simple assignment operation, and the rehash is not triggered any more, so that the performance is improved.

The following will specifically describe the unpacking optimization algorithm for reducing Lua table rehash provided by the embodiment of the present application with reference to fig. 8:

when the C++ layer receives network data (network byte stream) and unpacks (decodes), distinguishing the types of data types (table), integer, string, boolean (pool) and the like of pre-parsed data in the network data, and temporarily storing the data in an intermediate buffer layer in a TLV format. In the preprocessing process, the actual element number of the table is synchronously calculated, wherein the calculation process is as follows: the TLV data of each table type has table_head type data (representing the beginning of a table) and table_tail type data (representing the end of the table) in the analysis process, when the table_head is analyzed in the counting process, counting is carried out through a variable num, the num count is increased by 1 when a pair of key value data (key data and value data) is continuously analyzed until the table_tail is analyzed, and the counting is ended, so that the num represents the actual element number ElementNum of the table. The actual element number of the table is written into the TLV format, and only when the actual element number of the table is calculated in advance, memory can be pre-allocated directly through the element number when the table is created through the intermediate buffer layer, so that table rehash operation is avoided. Finally, the intermediate buffer layer data is pushed to the lua virtual machine layer, wherein the data pushing is realized only by a pointer of one piece of data, and no data copy is consumed. The table is created in batches and data is inserted in the lua virtual machine layer, and memory pre-allocation can be realized by knowing the number of elements of the table, so that rehash operation is not triggered any more.

In fig. 8, the lua_cloneable is an application program interface (API, application Programming Interface) added at the bottom layer of the Lua virtual machine, which can parse the TLV format of the intermediate buffer layer, create the tables in batches, and pre-allocate the memory by using the number of table elements in the TLV format when creating the tables.

The following description is made for intermediate buffer layer data encoding:

the intermediate buffer layer is a continuous memory, and different types of data are stored continuously. For different types of data, TLV mode encoding is adopted, and meanwhile, some optimization is carried out on the Lua table so as to facilitate the identification of the Lua virtual machine.

As shown in fig. 9, each type of TLV format has a value field of a body type occupying 1 byte, a value field of an integer type occupying 8 bytes, a value field of a number type occupying 8 bytes, a length field of a string type occupying 4 bytes, a value field occupying 4 bytes, a length field of a table type occupying 2 bytes (i.e., an elem_num field of tag information representing the number of elements, array, or hash), the value field containing elem_num key value data (key data and value data).

It should be noted that, for the table type data, the key and the value respectively correspond to the data entry of one TLV (the key corresponds to the data entry of one TLV, and the value also corresponds to the data entry of one TLV). For the elem_num field of the table, since the table of Lua includes an array and a hash, the elem_num should contain tag information of a tag array or a hash type.

Therefore, elem_num= (is_array &0x 01) < <15|num &0x7fff in the embodiment of the present application, where (is_array &0x 01) < <15 is used to mark array or hash type, and num &0x7fff is used to record the actual element number ElementNum of the table. It should be noted that, although the table may include two parts of array and hash at the same time, because the table transmitted by C2S or S2S generally includes only array or hash, there are few cases where array and hash coexist, and thus only 1bit (is_array &0x 01) < <15 tag types are needed. Meanwhile, the number of the C2S or S2S network transmission table elements cannot be too large, and 15 bits may indicate 32767 elements are enough. And the frequent triggering stage of the rehash is the initial element insertion stage of the empty table, if the memory of 32767 elements is pre-allocated, the frequency of triggering the rehash again is very low.

In the following, description will be given by taking s2s. Report_data (data_list) as an example, where report_data represents an RPC interface called between servers and is used to report service layer data to a platform, and report represents an internal system platform identifier, and data_list represents a data element of an internal system platform. As shown in fig. 10, taking the flat=10 integer, the data_list= {11, 12, 13} array as an example, the intermediate buffer layer is stored in the data structure shown in fig. 10.

The following description is made for Lua virtual machine lua_closed batch creation:

lua_cloneable is a function interface added at the bottom layer of the Lua virtual machine, and is used for analyzing TLV data format of an intermediate buffer layer of a C++ layer and creating tables in a batch recursion mode. Of course, the TLV data format of the table type already contains elem_num information, which can be used for memory pre-allocation.

The following describes the procedure for lua_closed batch creation with reference to fig. 11:

step 1, the lua_cloneable receives a c++ layer TLV data buffer (buffer) pointer (an address pointing to TLV structure data of an intermediate buffer layer), and obtains TLV format data based on the buffer pointer.

And 2, analyzing TLV data format data through a get_val function.

And 3, when the TLV entry is analyzed to be of a table type, creating a table through a lua_createtable (lua_State. L, int narry, int nrec) and pre-allocating a memory.

Where L represents the virtual machine address, narray represents the length of the array portion, nrec represents the length of the hash portion. And according to elem_num= (is_array &0x 01) < <15|num &0x7fff in TLV entry, obtain the element number of array or hash in the table, and then can be transferred into the element number when the table is created, finally realize the pre-allocation memory.

And 4, when the analyzed TLV entry is a non-table, the analyzed TLV entry is the data entry of the current table, and the analyzed TLV entry exists in a key value pair mode. And as the key and the value are connected together, respectively analyzing, and finally calling the luaH_set function to insert the key and the value into the table.

And analyzing the key data in the table through the decode_key function, and analyzing the value data in the table through the decode_value function.

And 5, repeating the steps 2, 3 and 4, and continuously recursively creating a table (a condition that a nested table exists) and inserting elements until the TLV data of the intermediate buffer layer is analyzed.

As shown in fig. 12, the table data structure in the Lua layer, the constructed table includes an array portion or a hash table portion.

And 6, calling an api_incr_top function to change a lua stack top pointer and returning the pointer to an application layer (namely a C++ layer) because the constructed table is on a stack of the lua virtual machine.

It should be noted that, the whole buffer data is managed by referring to the structure TLV of PB, which is simple and clear; compared with PB, all length fields and integer are not subjected to varint coding (a method for serializing integers by using one or more bytes, the integers are coded into variable-length bytes), so that the calculation amount is saved; for the table type, the element numbers of the ordered array and the hash table part need to be distinguished, and because two memories at the bottom layer of the table are independently managed, the highest bit in elem_num=is_list < <15|real_num is used for marking the ordered array or the hash table; for STRING types, the length of STRING is 4 bytes, but for most character STRINGs, the key field of the table or the character name in the game will not exceed 256 bytes, so the length of SRTING is fixed to 1 byte, the length of string_BIG is 4 bytes, and memory copy is reduced.

The following is an effect comparison between the optimization algorithm and the unpacking process in the related technology:

for primary database (DB, data Base) Data, the effect pair is shown in table 1:

TABLE 1

As can be seen from table 1, the optimization algorithm of the embodiment of the present application can improve the unpacking performance by about 60% for the main DB data.

For the main system data of the game, the effect pairs are shown in table 2:

TABLE 2

As shown in table 2, the optimization algorithm of the embodiment of the application can improve the unpacking performance by about 60% for the main system data.

As shown in fig. 13, for the unpacking optimization algorithm of the embodiment of the present application, the intermediate buffer layer data preprocessing consumes 50% more than the unpacking (decode). However, as the rehash is reduced, the whole table construction step saves 50% of calculated amount relative to the calculation amount before optimization, so that the overall performance is saved by 50% -60% compared with the calculation amount before optimization.

Having thus described the network data processing method provided by the embodiment of the present application in connection with the exemplary application and implementation of the terminal or server provided by the embodiment of the present application, the following continues to describe the implementation of the network data processing scheme by cooperation of each module in the network data processing apparatus 465 provided by the embodiment of the present application.

An acquisition module 4651 for acquiring a network byte stream for deserialization; a statistics module 4652, configured to perform table element statistics processing on the network byte stream when the data type of the network byte stream is a table type, to obtain the element number of the table to be deserialized; the parsing module 4653 is configured to perform parsing processing of a triplet message format based on the number of elements of the table to be de-serialized, so as to obtain triplet message information of the table to be de-serialized; an allocation module 4654, configured to perform a pre-allocation process of a storage space based on the number of elements included in the triplet message information; and the deserializing module 4655 is configured to perform deserializing processing on the table to be deserialized based on the storage space obtained by the pre-allocation, so as to obtain a deserialized table.

In some embodiments, the obtaining module 4651 is further configured to receive, through an application layer, a stream of network bytes for deserialization; before the pre-allocation processing of the storage space based on the number of elements included in the triplet message information, the parsing module 4653 is further configured to create an intermediate buffer layer in the application layer; caching the triple message information of the table to be reverse-sequenced in the middle buffer layer; the allocation module 4654 is further configured to combine the element number included in the triplet message information of the table to be deserialized, and perform, by using a virtual machine layer, pre-allocation processing of a storage space for the table to be deserialized, to obtain the storage space allocated by the table to be deserialized.

In some embodiments, the statistics module 4652 is further configured to begin performing table element statistics on data entries following any data entry in the network byte stream when the type of the any data entry is a start identification of a table type; and when the type of the data entry after any data entry is the end identification of the table type, ending the table element statistical processing, and taking the corresponding element number when the table element statistical processing is ended as the element number of the table to be deserialized.

In some embodiments, the statistics module 4652 is further configured to, for data entries subsequent to the arbitrary data entry, perform the following: when continuously analyzing to a key field entry and a value field entry, adding 1 to the element number corresponding to the table element statistical processing.

In some embodiments, the allocation module 4654 is further configured to perform the following processing by the virtual machine layer: receiving a buffer pointer sent by the application layer, wherein the buffer pointer represents the address of the triple message information in the middle buffer layer; determining the element number in the triple message information through the buffer pointer; and performing pre-allocation processing on the storage space based on the element number included in the triple message information to obtain the storage space allocated by the table to be de-sequenced.

In some embodiments, the parsing module 4653 is further configured to perform a conversion process on the network byte stream in a triplet message format, to obtain the triplet message format of the network byte stream; wherein, when the data structure of the data entry of the table type is an array or a hash table, the triplet message format includes a type field, a length field, and a value field, the length field includes a first byte for characterizing the data structure, and a second byte for the number of elements of the table type; setting the value of the type field to be a value representing the type of the table, setting the value of the value field to be a data entry of the table to be deserialized, setting the value of the first byte to be a value representing the data type of the data entry of the table to be deserialized, and setting the value of the second byte to be the number of elements of the table to be deserialized, wherein the data type of the data entry of the table to be deserialized is the array or the hash table.

In some embodiments, when the data types of the data entries of the table to be de-serialized include an array and a hash table, the de-serializing module 4655 is further configured to, based on the pre-allocated storage space, perform de-serializing processing on the data entries of the first data type included in the table to be de-serialized to obtain a partial table; the first data type is a first data type in the table to be deserialized, and the first data type is any one of the array and the hash table; and performing anti-serialization processing based on re-hash on the data items of a second data type included in the table to be anti-serialized based on the part table to obtain the table after anti-serialization, wherein the second data type is the other one of the array and the hash table.

In some embodiments, when the data structure of the data entry of the table type includes an array and a hash table, the length field includes a first sub-length field and a second sub-length field; wherein the first sub-length field includes a third byte for characterizing the array and a fourth byte for the number of elements of the array, and the second sub-length field includes a fifth byte for characterizing the hash table and a sixth byte for the number of elements of the hash table; the parsing module 4653 is further configured to set the value of the fourth byte to the number of elements of the array included in the table to be deserialized, and set the value of the sixth byte to the number of elements of the hash table included in the table to be deserialized.

In some embodiments, the element number included in the triplet message information includes an element number of an array included in the table to be deserialized and an element number of a hash table included in the table to be deserialized; the allocation module 4654 is further configured to perform a pre-allocation process of a storage space based on the number of elements of the array included in the table to be de-serialized, to obtain a first storage space; performing pre-allocation processing on the storage space based on the element number of the hash table included in the table to be de-serialized to obtain a second storage space;

The anti-serialization module 4655 is further configured to perform anti-serialization processing on an array included in the table to be anti-serialized based on the first storage space, to obtain a first partial table; performing deserialization processing on the hash table included in the table to be deserialized based on the second storage space to obtain a second partial table; and combining the first part table and the second part table to obtain the anti-sequenced table.

In some embodiments, when the data type of the network byte stream is a non-table type, the anti-serialization module 4655 is further configured to perform analysis processing on the network byte stream in a triplet message format, to obtain triplet message information of the table to be anti-serialized; performing deserialization processing on the triplet message information to obtain the deserialized table; wherein the non-table type comprises Boolean type, integer type, character string type and numerical value type.

In some embodiments, the anti-serialization module 4655 is further configured to create the table to be anti-serialized based on the storage space obtained by the pre-allocation, to obtain an empty table; and carrying out assignment processing on the empty table based on key data items and value data items included in the triple message information to obtain the table after reverse serialization.

Embodiments of the present application provide a computer program product or computer program comprising computer instructions stored in a computer readable storage medium. The processor of the electronic device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the electronic device executes the network data processing method according to the embodiment of the present application.

Embodiments of the present application provide a computer readable storage medium having stored therein executable instructions which, when executed by a processor, cause the processor to perform a network data processing method provided by embodiments of the present application, for example, as shown in fig. 3-5.

In some embodiments, the computer readable storage medium may be FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; but may be a variety of devices including one or any combination of the above memories.

In some embodiments, the executable instructions may be in the form of programs, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and they may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

As an example, the executable instructions may, but need not, correspond to files in a file system, may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a hypertext markup language (HTML, hyper Text Markup Language) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub-programs, or portions of code).

As an example, executable instructions may be deployed to be executed on one computing device or on multiple computing devices located at one site or, alternatively, distributed across multiple sites and interconnected by a communication network.

It will be appreciated that in the embodiments of the present application, related data such as user information is involved, and when the embodiments of the present application are applied to specific products or technologies, user permissions or agreements need to be obtained, and the collection, use and processing of related data need to comply with relevant laws and regulations and standards of relevant countries and regions.

The foregoing is merely exemplary embodiments of the present application and is not intended to limit the scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A method of network data processing, the method comprising:

acquiring a network byte stream for deserialization;

2. The method of claim 1, wherein the step of determining the position of the substrate comprises,

the obtaining a network byte stream for deserialization includes:

receiving, by an application layer, a network byte stream for deserialization;

before the pre-allocation processing of the storage space is performed based on the element number included in the triplet message information, the method further includes:

creating an intermediate buffer layer in the application layer;

The pre-allocation processing of the storage space based on the element number included in the triplet message information comprises the following steps:

and combining the element number included in the triple message information of the table to be deserialized, and performing pre-allocation processing on the storage space of the table to be deserialized through a virtual machine layer to obtain the storage space allocated by the table to be deserialized.

3. The method of claim 2, wherein said performing table element statistics on said network byte stream to obtain the number of elements of the table to be de-serialized comprises:

when the type of any data entry in the network byte stream is a start identification of a table type, starting to perform table element statistical processing on the data entry after the any data entry;

4. A method according to claim 3, wherein said starting table element statistical processing of data entries subsequent to said arbitrary data entry comprises:

The following processing is performed for a data entry following the arbitrary data entry:

5. The method according to claim 2, wherein the pre-allocation of the storage space for the table to be de-serialized by the virtual machine layer in combination with the element number in the triplet message information of the table to be de-serialized, to obtain the storage space allocated by the table to be de-serialized, includes:

the following processing is executed through the virtual machine layer:

6. The method according to claim 1, wherein the parsing of the triplet message format based on the number of elements of the table to be de-serialized, to obtain triplet message information of the table to be de-serialized, includes:

Performing conversion processing on the network byte stream to obtain a triplet message format of the network byte stream;

7. The method of claim 6, wherein the step of providing the first layer comprises,

when the data type of the data entry of the table to be deserialized includes an array and a hash table, the deserializing processing is performed on the table to be deserialized based on the storage space obtained by pre-allocation to obtain a deserialized table, which includes:

Performing deserialization processing on data items of a first data type included in the table to be deserialized based on the storage space obtained by the pre-allocation to obtain a partial table;

the method further comprises the steps of:

8. The method of claim 6, wherein the step of providing the first layer comprises,

when the data structure of the data entry of the table type comprises an array and a hash table, the length field comprises a first sub-length field and a second sub-length field;

The method further comprises the steps of:

setting the value of the fourth byte to the number of elements of an array included in the table to be deserialized, and setting the value of the sixth byte to the number of elements of a hash table included in the table to be deserialized.

9. The method of claim 8, wherein the step of determining the position of the first electrode is performed,

the element number included in the triple message information comprises the element number of an array included in the table to be deserialized and the element number of a hash table included in the table to be deserialized;

performing pre-allocation processing of the storage space based on the element number of the array included in the table to be de-sequenced to obtain a first storage space;

the anti-serialization processing is performed on the table to be anti-serialized based on the storage space obtained by pre-allocation, so as to obtain an anti-serialized table, which comprises the following steps:

performing deserialization processing on an array included in the table to be deserialized based on the first storage space to obtain a first partial table;

10. The method of claim 1, wherein when the data type of the network byte stream is a non-table type, the method further comprises:

analyzing the network byte stream in a triplet message format to obtain triplet message information of the table to be de-serialized;

11. The method according to claim 1, wherein the performing the deserialization on the table to be deserialized based on the memory space obtained by the pre-allocation to obtain a deserialized table comprises:

creating the table to be de-sequenced based on the storage space obtained by the pre-allocation to obtain an empty table;

12. A network data processing apparatus, the apparatus comprising:

13. An electronic device, the electronic device comprising:

a memory for storing executable instructions;

a processor for implementing the network data processing method of any one of claims 1 to 11 when executing executable instructions stored in said memory.

14. A computer readable storage medium storing executable instructions for implementing the network data processing method of any one of claims 1 to 11 when executed by a processor.

15. A computer program product comprising a computer program or instructions which, when executed by a processor, implements the network data processing method of any one of claims 1 to 11.