CN117596144A

CN117596144A - Method and device for acquiring background data

Info

Publication number: CN117596144A
Application number: CN202311545799.8A
Authority: CN
Inventors: 蓝飘; 张承业; 田丰
Original assignee: GRG Banking IT Co Ltd
Current assignee: GRG Banking IT Co Ltd
Priority date: 2023-11-17
Filing date: 2023-11-17
Publication date: 2024-02-23

Abstract

The application discloses a method and a device for acquiring background data, and belongs to the technical field of computer networks. The method for acquiring the background data comprises the following steps: receiving an access request of a client; returning first data to the client; the first data is a binary data stream obtained by serializing second data corresponding to the access request through a serialization method; the serialization method is used to remove at least a portion of the non-valid values in the second data. According to the method and the device for acquiring the background data, the background data which is subjected to serialization processing by the serialization method is transmitted in the network, so that the data transmission volume is smaller, the requirement of network bandwidth can be reduced to the greatest extent, the network transmission of large data volume can be realized, and the scene of acquiring a large amount of background data can be met. And the transmitted background data is the data processed by the specific serialization method, so that the security of the transmission in the network is higher, and the risk of leakage of the transmission data in the network can be greatly reduced.

Description

Method and device for acquiring background data

Technical Field

The application belongs to the technical field of computer networks, and particularly relates to a method and a device for acquiring background data.

Background

Under Browser/Server (B/S) architecture, a client can obtain data from a Server through a Browser by installing the Browser. In general, a client is used as a front end, a server is used as a background, and the acquisition of background data is to acquire data from the server.

Under the current B/S architecture, for scenes needing to acquire a small amount of background data and having little concurrency, the background data acquisition with good performance can be realized. However, for a scene where a large amount of background data needs to be acquired, the performance is poor, and problems such as high delay and high bandwidth requirement are easy to occur.

Disclosure of Invention

The present application aims to solve at least one of the technical problems existing in the prior art. Therefore, the application provides a method and a device for acquiring background data, which can reduce the requirement of network bandwidth and realize high-performance large-data-volume network transmission.

In a first aspect, the present application provides a method for acquiring background data, the method comprising:

receiving an access request of a client;

returning first data to the client; the first data is a binary data stream obtained by serializing the second data corresponding to the access request through a serialization method; the serialization method is used for removing at least part of non-valid values in the second data.

According to the method for acquiring the background data, the first data is obtained by carrying out serialization processing on the second data by the serialization method, so that the background data which is subjected to serialization processing by the serialization method is transmitted in the network, the data transmission volume is smaller, the requirement of network bandwidth can be reduced to the greatest extent, the network transmission of a large amount of data can be realized, and the scene of acquiring a large amount of background data can be met. And the transmitted background data is the data processed by the specific serialization method, so that the security of the transmission in the network is higher, and the risk of leakage of the transmission data in the network can be greatly reduced.

According to one embodiment of the present application, before the first data is returned to the client and after the receiving the access request of the client, the method further includes:

obtaining the second data from a database server;

and carrying out serialization processing on the second data through the serialization method to obtain the first data.

According to one embodiment of the present application, the serializing the second data by the serializing method includes:

acquiring a data sequence spliced by a plurality of data barrels based on the second data; the data buckets are in one-to-one correspondence with class object instance data in the second data; each data bucket is spliced by at least one data block; the data block corresponds to the field meaning in the class object instance data corresponding to the data bucket; each data block is spliced by the length and the value of the corresponding field;

And converting the value of each field into binary, acquiring a first binary number, and carrying out variable length coding on the value of the valid bit in the first binary number according to the data type of the field.

According to one embodiment of the present application, the variable length encoding the value of the significant bit in the first binary number according to the data type of the field includes:

under the condition that the data type of the field is an unsigned number, sequentially intercepting values of 7 significant digits in the first binary number as the last 7 bits of one byte according to the sequence from low bits to high bits, and determining the value of the most significant digit of the byte based on whether the significant digit exists in the high bit direction of the 7 significant digits or not until all significant digits are taken; splicing the bytes according to the intercepted sequence;

converting the first binary number into a second binary number of an unsigned number type when the data type of the field is a signed number; sequentially intercepting the values of 7 significant digits in the second binary number as the last 7 digits of one byte according to the sequence from the low order to the high order, and determining the value of the most significant digit of the byte based on whether the significant digit exists in the high order direction of the 7 significant digits until all the significant digits are taken; splicing the bytes according to the intercepted sequence;

And in the case that the data type of the field is a character string, encoding the value of the significant bit in the first binary number based on the unicode.

attempting to obtain the first data from a local cache;

and returning the first data to the client under the condition that the first data is successfully acquired from the local cache.

According to the method for acquiring the background data, based on the two-stage cache, first, the first data is tried to be acquired from the constructed local cache, and under the condition that the first data is successfully acquired from the local cache, the first data is returned to the client, so that the number of times of IO of the database and the bandwidth required by the request cache database can be reduced to the greatest extent, the data acquisition efficiency can be accelerated, and the data acquisition efficiency is faster.

According to one embodiment of the present application, before the attempting to obtain the first data from the local cache, the method further includes:

acquiring the second data from a database server based on the target page displayed by the client according to the time period;

Carrying out serialization processing on the second data by the serialization method to obtain first data;

and storing the first data in the local cache, and synchronously storing the first data in a cache database.

According to one embodiment of the present application, the obtaining the second data from a database server includes:

attempting to obtain the first data from a local cache;

attempting to acquire the first data from a cache database if the first data is not successfully acquired from the local cache;

and under the condition that the first data is not successfully acquired from the cache database, acquiring the second data from the database server.

According to one embodiment of the present application, after attempting to obtain the first data from the cache database in a case where the first data is not successfully obtained from the local cache, the method further includes:

and returning the first data to the client under the condition that the first data is successfully acquired from the cache database.

And under the condition that the first data is successfully acquired from the cache database, synchronously storing the first data in the local cache.

In a second aspect, the present application provides an apparatus for acquiring background data, the apparatus comprising:

the receiving module is used for receiving the access request of the client;

the return module is used for returning the first data to the client; the first data is a binary data stream obtained by serializing the second data corresponding to the access request through a serialization method; the serialization method is used for removing at least part of non-valid values in the second data.

According to the device for acquiring the background data, the first data is obtained by carrying out serialization processing on the second data by the serialization method, so that the background data which is subjected to serialization processing by the serialization method is transmitted in the network, the data transmission volume is smaller, the requirement of network bandwidth can be reduced to the greatest extent, the network transmission of a large amount of data can be realized, and the scene of acquiring a large amount of background data can be met. And the transmitted background data is the data processed by the specific serialization method, so that the security of the transmission in the network is higher, and the risk of leakage of the transmission data in the network can be greatly reduced.

In a third aspect, the present application provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, where the processor implements the method for obtaining background data according to the first aspect described above when the processor executes the computer program.

In a fourth aspect, the present application provides a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements a method of acquiring background data as described in the first aspect above.

In a fifth aspect, the present application provides a chip, the chip including a processor and a communication interface, the communication interface and the processor being coupled, the processor being configured to execute a program or instructions to implement the method for acquiring background data according to the first aspect.

In a sixth aspect, the present application provides a computer program product comprising a computer program which, when executed by a processor, implements a method of acquiring background data as described in the first aspect above.

Additional aspects and advantages of the application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the application.

Drawings

The foregoing and/or additional aspects and advantages of the present application will become apparent and readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings, wherein:

fig. 1 is one of schematic diagrams of an application scenario of a method for acquiring background data according to an embodiment of the present application;

fig. 2 is a second schematic diagram of an application scenario of a method for obtaining background data according to an embodiment of the present application;

FIG. 3 is a flowchart of a method for obtaining background data according to an embodiment of the present disclosure;

fig. 4 is a schematic structural diagram of a data sequence in a method for acquiring background data according to an embodiment of the present application;

fig. 5 is a flowchart illustrating a pre-step in a method for acquiring background data according to an embodiment of the present application;

FIG. 6 is a second flowchart of a method for obtaining background data according to an embodiment of the present disclosure;

fig. 7 is a schematic structural diagram of an apparatus for acquiring background data according to an embodiment of the present application;

fig. 8 is a schematic structural diagram of an electronic device according to an embodiment of the present application.

Detailed Description

Technical solutions in the embodiments of the present application will be clearly described below with reference to the drawings in the embodiments of the present application, and it is apparent that the described embodiments are some embodiments of the present application, but not all embodiments. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments in the present application are within the scope of the protection of the present application.

The terms first, second and the like in the description and in the claims, are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged, as appropriate, such that embodiments of the present application may be implemented in sequences other than those illustrated or described herein, and that the objects identified by "first," "second," etc. are generally of a type and not limited to the number of objects, e.g., the first object may be one or more. Furthermore, in the description and claims, "and/or" means at least one of the connected objects, and the character "/", generally means that the associated object is an "or" relationship.

In the related art, the B/S architecture, also referred to as browser/server architecture or B/S architecture, is emerging with the continued maturity of Web technology. B/S architecture can be understood as a change and promotion to C/S (client/server) architecture.

With the continuous rapid development of network technology, the B/S architecture is becoming more and more powerful. First, based on the B/S architecture, information distributed processing can be performed, so that the resource cost can be effectively reduced, and the performance of the system can be improved. Secondly, the B/S architecture greatly simplifies the client in the processing mode, a user only needs to install a browser on the client, and the application logic is concentrated on the server and the middleware, so that the B/S architecture can improve the data processing performance and has wider application range. Third, in terms of software universality, the client of the B/S architecture has better universality and smaller dependence on application environments. In addition, the client uses the browser, so that development and maintenance are more convenient, and the cost of system development and maintenance can be reduced.

Fig. 1 is a schematic diagram of an application scenario of a method for obtaining background data according to an embodiment of the present application. In addition to clients accessing Web servers directly through a browser, proxy servers may also be employed in the B/S architecture as a transfer station between the client and Web server, as shown in fig. 1. The client accesses the proxy server through the browser, the access request is transferred to the Web server through the proxy server, the Web server receives the access request and performs corresponding service processing, and then the database server is requested to acquire corresponding data, and the corresponding data is returned to the client and displayed through the browser. Fig. 1 shows a B/S architecture of an application service distributed multi-node, i.e. comprising a plurality of Web servers (Web server 1, web server 2 and Web server 3 are exemplarily shown in fig. 1, it being understood that the number of Web servers may not be limited to 3).

The proxy server can be used for proxy static resources and reverse proxy back-end application servers and plays a role in load balancing. Static resources may include HTML (hypertext markup language ), CSS (cascading style sheets, cascading Style Sheets), JS (JavaScript), pictures, and the like of a web page.

In most business scenes, such as a network disk file tree, an audited unit tree, an administrative area tree or a commodity classification tree, all data needs to be acquired at one time and displayed on a browser, and the data cannot be acquired in batches and paged and displayed on a page. Similar business scenes are often all present at the first page position of the system, the first page is the most frequently accessed page, and the background service is frequently requested under the conditions of large data volume and simultaneous access of multiple people, and the background service frequently requests a database server to acquire data, so that the page is slow to open and even blocked. The solution adopted in the related art is to cache the data in the database server through a cache database, wherein the Web server firstly acquires the data in the cache database, and then acquires the data from the database server under the condition that the data cannot be acquired. Because the cache database is an internal memory database, the access efficiency is higher than that of a direct access database server, and the frequent operation of the Web server on the database IO can be avoided. The browser and the proxy server can transmit the message in a compressed mode, and the browser realizes the caching of returned data based on a caching mechanism, so that the background is prevented from being frequently requested.

Fig. 2 is a second schematic diagram of an application scenario of the method for obtaining background data according to the embodiment of the present application. The application scenario of the above solution may be as shown in fig. 2. Fig. 2 shows a B/S architecture of an application service distributed multi-node with added cache, and as can be seen from comparing fig. 1 and fig. 2, a third party cache database is added between the Web server and the database server.

The method for acquiring the background data, the device for acquiring the background data, the electronic equipment and the readable storage medium provided by the embodiment of the application are described in detail below through specific embodiments and application scenes thereof with reference to the accompanying drawings.

The method for acquiring the background data can be applied to the terminal, and can be specifically executed by hardware or software in the terminal.

The terminal includes, but is not limited to, a portable communication device such as a mobile phone or tablet having a touch sensitive surface (e.g., a touch screen display and/or a touch pad). It should also be appreciated that in some embodiments, the terminal may not be a portable communication device, but rather a desktop computer having a touch-sensitive surface (e.g., a touch screen display and/or a touch pad).

In the following various embodiments, a terminal including a display and a touch sensitive surface is described. However, it should be understood that the terminal may include one or more other physical user interface devices such as a physical keyboard, mouse, and joystick.

The execution main body of the method for acquiring the background data provided in the embodiment of the present application may be an electronic device or a functional module or a functional entity capable of implementing the method for acquiring the background data in the electronic device, where the electronic device in the embodiment of the present application includes, but is not limited to, a mobile phone, a tablet computer, a camera, a wearable device, and the like, and the method for acquiring the background data provided in the embodiment of the present application is described below by taking the electronic device as an execution main body as an example.

In actual implementation, the electronic device may be any of the Web servers in fig. 1 or fig. 2.

As shown in fig. 3, the method for acquiring background data includes: step 310 and step 320.

Step 310, an access request of the client is received.

In actual execution, the client may issue an access request through the browser. And the access request is used for requesting to access a certain page. The page may be referred to as a destination page, and accordingly, the access request may carry identification information of the destination page.

In some embodiments, the identification information may include at least one of a uniform resource locator, an IP address, and the like.

The electronic device may receive the access request over a computer network. The process of transmitting the access request from the client to the electronic device may refer to fig. 1 or fig. 2, and will not be described herein.

Step 320, returning the first data to the client; the first data is a binary data stream obtained by serializing second data corresponding to the access request through a serialization method; the serialization method is used to remove at least a portion of the non-valid values in the second data.

In actual execution, the target page may be determined based on the access request. After the target page is determined, the data for meeting the display requirement of the target page can be used as second data corresponding to the access request. The second data is data which meets the display requirement of the target page and needs to be displayed on the target page at one time.

In some embodiments, after determining the second data, the first data may be obtained directly from a data source, or the second data may be obtained from the database server in fig. 1 or fig. 2, and then the second data is serialized by a serialization method, so as to obtain the first data.

It should be noted that, the first data is obtained by serializing the second data by a serializing method. The serialization processing is performed on the second data by the serialization method, so as to remove part or all of the non-valid values in the second data.

A significant value refers to the value of each digit used to determine the accuracy of one datum. The values in one data other than the valid value are all non-valid values. Data without actual physical meaning is also a non-valid value.

For example, in a fixed-length field, if the length of the valid bit of the field is smaller than the length of the field (may be "preset length"), zero is filled in the high-order direction of the valid bit until a byte of the preset length is obtained. And the zero added in the high-order direction is a non-effective value.

For another example, in some data structures, to facilitate distinguishing between two fields, the two fields may be separated by a particular character (e.g., space, semicolon, comma, isoparaffin, colon, or vertical line, etc.). The specific character is a separator. The delimiter has no data of actual physical meaning and is thus also a non-valid value.

It will be appreciated that since some or all of the non-valid values in the second data are removed by the serialization process, the storage of the fields in the first data is very compact, the storage space utilization is very high, and the data transfer volume is smaller than the second data.

It should be noted that, in the embodiment of the present application, any method that can remove some or all of the non-valid values in the second data may be used as the serialization method. The embodiments of the present application are not limited to specific serialization methods.

It should be noted that, since the first data is obtained by serializing the second data by the serialization method, the first data is not plaintext data, and the security of transmitting the first data is higher than that of transmitting the plaintext second data. Accordingly, after receiving the first data, the client needs to perform deserialization processing on the first data based on an deserialization method corresponding to the serialization method to obtain second data, and then presents the second data through a browser, so that the target page is displayed.

In some embodiments, after the file compression process is performed on the file storing the first data, the file may be returned to the client, so as to further reduce the data transmission volume.

The file compression process may employ a conventional file compression algorithm, such as Zlib, gzip, bzip2, deflter, lz4, lzo, or Snappy. The embodiments of the present application are not limited to a specific file compression algorithm.

For example, taking the data volume of the database table data as 1M as an example, the volume of the transmission file can be reduced by 2 to 10 times compared with JSON, XML and other modes by performing serialization processing and Gzip (GNU zip) compression on the data. That is, for the same database table data of about 1M, the volume of the file transmitted by JSON, XML, or the like is 2 to 10 times the volume of the file transmitted by the serialization processing and Gzip compression method.

According to the method for acquiring the background data, the background data which is subjected to serialization processing by the serialization method is transmitted in the network, so that the data transmission volume is smaller, the requirement of network bandwidth can be reduced to the greatest extent, the network transmission of large data volume can be realized, and the scene of acquiring a large amount of background data can be met. And the transmitted background data is the data processed by the specific serialization method, so that the security of the transmission in the network is higher, and the risk of leakage of the transmission data in the network can be greatly reduced.

In some embodiments, before returning the first data to the client and after receiving the access request of the client, the method further comprises: second data is obtained from a database server.

In an actual implementation, the second data may be obtained from the database server after the second data is determined.

And carrying out serialization processing on the second data by a serialization method to obtain the first data.

In practical implementation, the second data may be serialized to obtain the first data by any method that may implement removing some or all of the non-valid values in the second data.

According to the method for acquiring the background data, the first data is obtained by carrying out serialization processing on the second data by the serialization method, the background data which is subjected to serialization processing by the serialization method is transmitted in the network, the data transmission volume is smaller, the requirement of network bandwidth can be reduced to the greatest extent, the network transmission with large data volume can be realized, and the scene of acquiring a large amount of background data can be met. And the transmitted background data is the data processed by the specific serialization method, so that the security of the transmission in the network is higher, and the risk of leakage of the transmission data in the network can be greatly reduced.

In some embodiments, the serializing the second data by the serializing method includes: acquiring a data sequence spliced by a plurality of data barrels based on the second data; the data barrels are in one-to-one correspondence with class object instance data in the second data; each data bucket is spliced by at least one data block; the data blocks are in one-to-one correspondence with fields in class object instance data corresponding to the data buckets; each data block is spliced by the length and value of the corresponding field.

In practical implementation, the serialization method may be a method adopting a pipeline sequence elastic length barrel coding technology, so as to realize that the second data is packaged in a class object mode.

In some embodiments, a file for defining a description specification of a one-to-one mapping relationship between class object instance data and a data bucket may be pre-established. Each class object instance data corresponds to a data bucket. Different class object instance data corresponds to different buckets of data.

In some embodiments, the syntax employed by the file may be "field type field name=sequence number". The field type and the field name are respectively consistent with the data type and the name of the field in the class object instance data; the sequence number is used to indicate the integer number of the order of the fields in the class object instance data.

It will be appreciated that in a description specification file, the sequence number of a field is unique for each class object instance data. The sequence numbers of fields in the class object instance data may be incremented.

In some embodiments, the delimiter may be removed from the second data first, and the fields in the second data may be rearranged according to the description specification of the data bucket defined in the foregoing document, to obtain the data sequence.

In some embodiments, rearranging the respective fields in the second data may include: respectively aggregating fields in class object instance data to obtain a plurality of data barrels, and splicing the data barrels; for each data bucket, serializing each field according to the sequence of increasing sequence numbers to obtain at least one data block and splicing a plurality of data blocks.

In some embodiments, the fields may be serialized in the format of LV (Length-Value), resulting in a data sequence as shown in FIG. 4. Wherein L represents the length of the field, L is an integer; v represents the Value of this field.

It will be appreciated that each field may be determined by L and V, so that two adjacent fields may be separated without a separator.

Fig. 4 is a schematic structural diagram of a data sequence in the method for acquiring background data according to the embodiment of the present application. As shown in fig. 4, the second data is serialized into N data buckets, illustrating that there are N class object instance data in the second data; one data block in each data bucket stores L and V for one field, respectively. Wherein N is a positive integer; m is a positive integer representing the number of fields in the data sequence.

The data sequence in the embodiment of the present application may be referred to as a pipeline sequence, and the data storage manner of the data sequence in the embodiment of the present application may be referred to as a pipeline sequence LV data bucket storage manner. As can be seen from fig. 4, the binary stream obtained in the storage mode can separate the fields without using the separator, so that the use of the separator can be reduced, the storage of each field is more compact, and the utilization rate of the storage space is very high.

Note that, in the case where the data type of a field is int32, int64, uint32, uint64, sint32, sint64, bol, or enum, L of the field may be a negative integer.

For each field, converting the value of the field into binary, acquiring a first binary number, and carrying out variable length coding on the value of the valid bit in the first binary number according to the data type of the field.

In actual implementation, for each field, the value of the field may be first converted into a first binary number in binary representation, then the first binary number is encoded with a variable length, and some or all of the non-valid values in the value of the field are removed.

In the embodiment of the present application, the variable length coding may be performed by any coding method that may implement removal of some or all of the non-valid values in the field. The embodiments of the present application are not limited to specific encoding methods.

In some embodiments, for fields of different data types, the coding method corresponding to the data type may be used, i.e. for different fields, different coding methods may be used due to their different data types.

In some embodiments, the same encoding method may be used for each field, with some or all of the non-valid values in the values of that field removed.

According to the method for acquiring the background data, based on the second data, the data sequence spliced by the data barrels is acquired, each data barrel is formed by splicing at least one data block, separation symbols are removed, the value of each field is converted into binary system for each field, the first binary system is acquired, the value of the effective bit in the first binary system is subjected to variable length coding according to the data type of the field, partial or all non-effective values are removed, the data transmission volume is smaller, the network bandwidth requirement can be reduced to the greatest extent, network transmission with a large amount of data can be achieved, and the scene of acquiring a large amount of background data can be met. And the transmitted background data is the data processed by the specific serialization method, so that the security of the transmission in the network is higher, and the risk of leakage of the transmission data in the network can be greatly reduced.

In some embodiments, variable length encoding the value of the significant bit in the first binary number according to the data type of the field comprises: under the condition that the data type of the field is an unsigned number, sequentially intercepting values of 7 significant digits in a first binary number as last 7 digits of one byte according to the sequence from low digits to high digits, and determining the value of the most significant digit of the byte based on whether the significant digit exists in the high digit direction of the 7 significant digits or not until all significant digits are taken; and splicing the bytes according to the intercepted sequence.

In practical implementations, integer data flexible coding methods may be used for both unsigned number type and signed number type fields where L is a negative integer. For fields of data type where L is a positive integer, the encoding may be performed using Unicode (Unicode). Illustratively, the above encoding method may be as shown in table 1.

Table 1 coding scheme list method

For any field of the unsigned number type, the step of integer data resilient encoding may comprise: after converting the value of the field into a first binary number, obtaining valid data in the first binary number to construct a new byte; the new bytes are spliced into 1 string in order as the result of encoding the value of this field.

Acquiring the valid data in the first binary number to construct a new byte can comprise sequentially selecting valid values of 7 digits from the tail of the first binary number as values of 7 digits after the new byte; the highest bit of the new byte is a flag bit for indicating whether a valid value exists in the high bit direction of the 7 digits.

In some embodiments, in the case where there is also a significant value in the upper bit direction of the 7 digits, the value of the most significant bit of the new byte is 1; in the case where the significant value does not exist in the upper bit direction of the 7 digits, the value of the upper bit of the new byte is 0.

It is understood that a digit may be a significant digit or a significant bit in the case where the value of the digit is a significant value.

The specific procedure for encoding the unsigned number is described below taking the field of data type int32 with value 186 as an example.

For int32 num=186, converting it to binary, resulting in a first binary number of "00000000 0000000000000000 10111010"; sequentially intercepting 7 digits from the end of the first binary number and supplementing 1 at the most significant bit (because the significant value exists in the high-order direction of the 7 digits), so as to obtain a new byte '10111010'; continuing to intercept 7 digits and complement 0 at the highest order (because the 7 digits have no valid value in the high order direction), obtaining a new byte "00000001"; the two new bytes are spliced into 1 character string '10111010 00000001' in sequence.

As can be seen from this example, by encoding, the value of the field originally represented by 4 bytes is reduced to be represented by only 2 bytes, greatly reducing the amount of data.

Converting the first binary number into a second binary number of an unsigned number type under the condition that the data type of the field is a signed number; sequentially intercepting the values of 7 significant digits in the second binary number as the last 7 digits of one byte according to the sequence from the low order to the high order, and determining the value of the highest order of the byte based on whether the significant digits exist in the high order direction of the 7 significant digits until all the significant digits are acquired; and splicing the bytes according to the intercepted sequence.

In practical implementation, for any field of the signed number type, the method of converting the signed number into the unsigned number (including but not limited to methods such as zigbee coding, etc., the specific method of converting the signed number into the unsigned number is not limited in the embodiment of the present application) may be used to convert the first binary number of the signed type into the second binary number of the unsigned number type; and then, for the second binary number, performing the step of elastically encoding the integer data of any field of the unsigned number type, and obtaining the encoding result of the value of the field.

For the second binary number, the procedure of performing the foregoing step of elastically encoding integer data for any field of the unsigned number type may be referred to the foregoing embodiment, and will not be repeated here.

The specific procedure for encoding the signed number described above will be described below taking the data type int32, a field with a value of-3 as an example.

For int32 num = -3, with the zigbee coding, the step of converting it into an unsigned binary number may comprise:

the highest bit in the symbol number 10000011 is the symbol bit, and the last 7 bits are the original codes of 3; the sign bit is unchanged, and the original code part of 3 is inverted to obtain '11111100'; the sign bit is unchanged, 1 is added to the original code part of 3 as a whole, and '11111101' is obtained as a binary representation of the final-3;

Converting "11111101" to an unsigned number may include: the whole body is shifted left by 1 bit, the right side is supplemented with 0 to obtain '11111010', the whole body is shifted right by 31 bits, the right side is supplemented with 1 to obtain '11111111'; bitwise exclusive or is performed on "11111010" and "11111111" to obtain "00000101".

Through the above steps, the first binary number "10000011" is converted into the second binary number "00000101". It can be seen that signed-3 is unsigned 5 after being encoded by the zigbee.

For the second binary number "00000101", 7 digits are sequentially truncated from the end of the second binary number and 0 is complemented at the most significant bit (because the significant value does not exist in the high order direction of the 7 digits), resulting in a new byte "00000101".

In the case where the data type of the field is a character string, the value of the significant bit in the first binary number is encoded based on unicode.

In practical implementation, for the value of the field whose data type is a character string, any unicode may be adopted, and the value of the field is encoded to obtain the encoding result of the value of the field.

It will be appreciated that unicode is a variable length encoding method that essentially encodes the value of the significant bit in the first binary number as well.

For example, each character in the value of the field whose data type is a character string may be encoded according to the encoding method of UTF-8.

According to the method for acquiring the background data, provided by the embodiment of the application, the integer data elastic coding method is adopted for the fields of the unsigned number type and the signed number type with L being negative integers, and the unified code is adopted for coding the fields of each data type with L being positive integers, so that part or all of non-effective values can be effectively removed, the byte number of the fields is reduced, and the data quantity can be greatly reduced.

In some embodiments, before returning the first data to the client and after receiving the access request of the client, the method further comprises: an attempt is made to obtain the first data from the local cache.

In actual implementation, on the basis of fig. 2, a local cache may be set in each Web server, so as to form a two-level cache in which the local cache and the third-party cache database cooperate.

In the two-level cache mode, the Web server may acquire data from the database server and store the data in the local cache based on a certain mechanism (e.g., timing acquisition or conditional acquisition).

After receiving the access request and determining the second data, an attempt may be made to retrieve the first data from the local cache.

In some embodiments, whether the first data obtained after the second data is serialized by the serialization method is stored in the local cache may be determined by a query or statistical method; after determining that the first data is stored in the local cache, the first data is acquired from the local cache.

In some embodiments, the operation of retrieving the first data from the local cache may also be performed.

In actual execution, where the first data is already stored in the local cache, the first data may be successfully retrieved from the local cache and then returned to the client over the computer network.

According to the method for acquiring the background data, based on the two-level cache, first, the first data is tried to be acquired from the constructed local cache, and under the condition that the first data is successfully acquired from the local cache, the first data is returned to the client, so that the number of times of IO of the database and the bandwidth required by the request cache database can be reduced to the greatest extent, the data acquisition efficiency can be accelerated, and the data acquisition efficiency is higher.

For example, taking the data volume of the database table data as 1M as an example, compared with the related art, the method for acquiring the background data provided in the embodiment of the present application can improve the efficiency of acquiring the data by 20-100 times.

In some embodiments, prior to attempting to obtain the first data from the local cache, further comprising: and acquiring second data from the database server based on the target page displayed by the client according to the time period.

In actual implementation, the second data for meeting the display requirement of the target page may be acquired from the database server periodically based on the target page displayed by the client according to the time period.

It will be appreciated that the time period may be set according to actual requirements. For example, it may take 1 minute or 3 minutes as a time period. The embodiments of the present application are not limited to specific values of the duration of the time period.

In actual execution, after the second data is acquired, the second data is subjected to serialization processing based on a serialization method, so as to obtain the first data.

For a specific step of serializing the second data by the serialization method, reference can be made to the previous embodiments, where no water tree is used.

The first data is stored in a local cache and synchronously stored in a cache database.

In actual execution, after the first data is obtained, the first data may be stored in a local cache and synchronously stored in a cache database, so as to make full use of the two-level cache.

The above steps performed per time period may be referred to as pre-steps. Fig. 5 is a flowchart illustrating a pre-step in a method for acquiring background data according to an embodiment of the present application. As shown in fig. 5, the pre-steps performed per cycle may include:

step 510, obtaining data from a database server.

Data for satisfying the target page display requirement can be acquired from the database server based on the target page displayed by the client.

It will be appreciated that the data may or may not be second data. In the case that the data is the second data, the first data may be successfully obtained from the local cache or the cache database in the following steps; in case the data is not the second data, the first data cannot be successfully retrieved from the local cache in a later step.

Step 520, the data is serialized by a serialization method.

The data obtained by step 510 may be serialized by the serialization method described previously.

It will be appreciated that in the case where the data obtained by step 510 is the second data, the first data is obtained by performing the serialization process by step 520.

Step 530, updating the cache database.

The data obtained by the serialization processing in step 520 is stored in the local cache and synchronously updated to the cache database.

In some embodiments, according to the unique constraint of keys (primary keys) in the cache database, the same keys cover the Value mechanism, and this operation is just to set the Value directly in the cache database. It should be noted that, since binary data is serialized, binary data to be stored needs to be converted into a Hex string of 16 system, and the Hex string is stored in the cache database as a field value.

In some embodiments, the time period may also be an expiration time of the local cache, i.e. after a time period, the local cache is emptied and the pre-step is re-performed. It will be appreciated that, due to the network environment or the task being executed by the client, the first data stored in the local cache may not be successfully returned to the client in the present period, so that the situation may occur that the first data is still stored in the local cache in the next period, and then the first data is obtained from the local cache and returned to the client, or that other data is stored in the local cache in the next period, but because the first data is already synchronously stored in the cache database, the first data may be obtained from the cache database and returned to the client.

The expiration time of the local cache is set to be used for realizing updating the data in the local cache. The time period can be flexibly set according to the timeliness of the data.

The data stored in the local cache is stored on the Web server, and the frequency of updates may be different for different Web servers (i.e., the time period may be different for different Web servers). Although this approach may lead to data inconsistencies, it is less sensitive to the user for data that is updated less frequently, and even if there is some variance in the data seen by the user in a short time, it will not have a significant impact on the user. In addition, the method is very suitable for the scenes such as personal network disks and the like which are all personal data of users, the data can be cached in the method, and newly added modified data of the individuals can be presented on a page in real time and synchronously and asynchronously updated to a database server.

According to the method for acquiring the background data, which is provided by the embodiment of the application, the pre-step is periodically executed, and the first data is stored in the local cache and/or the cache database, so that the times of IO of the database and the bandwidth required by the request cache database can be reduced to the greatest extent, the data acquisition efficiency can be accelerated, and the data acquisition efficiency is higher.

In some embodiments, obtaining the second data from the database server includes: attempting to obtain first data from the local cache; in the event that the first data is not successfully retrieved from the local cache, an attempt is made to retrieve the first data from the cache database.

In practical implementation, the step of attempting to obtain the first data from the local cache may be referred to the foregoing embodiments, and will not be described herein.

In the case where the first data is not stored in the local cache, the first data cannot be successfully acquired from the local cache. In the event that the first data is not successfully retrieved from the local cache, an attempt may be made to retrieve the first data from the cache database.

In some embodiments, whether the first data obtained by serializing the second data through a serialization method is stored in the cache database may be determined through a query or statistical method; after determining that the first data is stored in the cache database, the first data is acquired from the cache database.

In some embodiments, the operation of retrieving the first data from the cache database may also be performed.

In the event that the first data is not successfully retrieved from the cache database, the second data is retrieved from the database server.

In actual execution, the first data cannot be successfully acquired from the cache database without storing the first data in the cache database. In the event that the first data is not successfully retrieved from the cache database, an attempt may be made to retrieve the first data from the database server.

According to the method for acquiring the background data, based on the two-level cache, first, the first data are attempted to be acquired from the constructed local cache, under the condition that the first data are not successfully acquired from the local cache, then the first data are attempted to be acquired from the cache database, and under the condition that the first data are not successfully acquired from the cache database, the second data are acquired from the database server, so that the number of times of IO of the database and the bandwidth required by the request cache database can be reduced to the greatest extent, the data acquisition efficiency can be accelerated, and the data acquisition efficiency is faster.

In some embodiments, in the event that the first data is not successfully retrieved from the local cache, after attempting to retrieve the first data from the cache database, the method further comprises: and returning the first data to the client under the condition that the first data is successfully acquired from the cache database.

In actual execution, where the first data is already stored in the cache database, the first data may be successfully retrieved from the cache database and then returned to the client over the computer network.

According to the method for acquiring the background data, based on the two-level cache, first, the first data is tried to be acquired from the constructed local cache, under the condition that the first data is not successfully acquired from the local cache, then the first data is tried to be acquired from the cache database, under the condition that the first data is successfully acquired from the local cache, the first data is returned to the client, the number of times of IO of the database and the bandwidth required by the request cache database can be reduced to the greatest extent, the data acquisition efficiency can be accelerated, and the data acquisition efficiency is faster.

In some embodiments, in the event that the first data is not successfully retrieved from the local cache, after attempting to retrieve the first data from the cache database, the method further comprises: and under the condition that the first data is successfully acquired from the cache database, synchronously storing the first data in a local cache.

In actual execution, under the two-level cache mode, under the condition that the first data is not successfully obtained from the local cache and the first data is successfully obtained from the cache database, the first data can be synchronously stored in the local cache after the first data is obtained from the cache database, so that the two-level cache is fully utilized, and the success rate of obtaining the first data from the local cache can be improved in the process of subsequently obtaining the background data.

According to the method for acquiring the background data, under the condition that the first data is successfully acquired from the cache database, the first data is synchronously stored in the local cache, so that the success rate of acquiring the first data from the local cache can be improved in the process of subsequently acquiring the background data, the times of IO of the database and the bandwidth required by the request cache database can be reduced to the greatest extent, the data acquisition efficiency can be accelerated, and the data acquisition efficiency is higher.

In order to facilitate understanding of the above embodiments of the present application, a process of obtaining background data by a client under an architecture in which a local cache is provided in each Web server is described below based on the architecture shown in fig. 2.

Fig. 6 is a second flowchart of a method for obtaining background data according to an embodiment of the present application. As shown in fig. 6, taking the Web server as an execution body of the method for acquiring the background data as an example, the implementation process of the method for acquiring the background data may include the following steps:

step 610, accessing a Web server.

The client can access the Web server through the browser based on the http/https protocol, and receive the access request sent by the client.

Step 620, obtain data from the local cache.

The Web server attempts to obtain the first data from the local cache.

In some embodiments, local caches may be placed in the direct internal wetting of the Web server to avoid frequent GC of data (garbage collection ) from burdening the server and being unaffected by virtual machine memory.

Taking a Java environment as an example, the data in the local cache can be read through a ByteBuffer class.

Step 621, judging whether the process is successful.

The Web server determines whether the first data was successfully obtained from the local cache. In case of success, step 670 is performed; in the event of an unsuccessful, step 630 is performed.

Step 630, obtain data from the cache server.

The Web server attempts to obtain the first data from the cache server.

Step 631, judging whether the operation is successful.

The Web server determines whether the first data was successfully obtained from the local cache. In case of success, step 661 is performed; in the event of unsuccessful, step 640 is performed.

Step 640, obtain data from the database server.

The Web server obtains the second data from the database server. And acquiring data which needs to be displayed on the target page at one time, namely second data, from a corresponding table in the database server according to the display requirement of the target page.

Step 650, serialization processing.

And the Web server performs serialization processing on the second data, namely, performs serialization processing on the data obtained from the database by adopting a pipeline sequence elastic length barrel coding technology to obtain the first data.

Step 660, updating the cache database.

The Web server may store the first data in a cache database, updating the cache database.

Step 661, update local cache.

The Web server may update the local cache after storing the first data in the cache database, and also store the first data in the local cache.

Step 670, returning the result.

The Web server may return the first data to the proxy server.

Step 680, gzip compression is performed.

The proxy server can compress the message carrying the first data through the Web container and then transmit the compressed message to the browser through the http/https protocol. Illustratively, the transmitted message may be Gzip compressed by nginnx.

Step 690, decompression.

The client can decompress the received message by the Gzip through the browser.

Step 691, reverse serialization processing.

The client can adopt a pipeline sequence elastic length barrel coding technology to perform deserialization processing on the decompressed data. The decompressed data can be restored into a real message, namely the second data, based on the file of the description specification for defining the one-to-one mapping relation between the class object instance data and the data bucket.

The specific steps of deserialization may include:

for each field, whether the length of the field is a positive integer or a negative integer is determined according to whether the most significant bit of the L value of the field is 1 or 0. In the case that the length of the field is a negative integer, it can be determined that the L value of the field is elastically encoded by integer data; otherwise, the coding method of UTF-8 is adopted.

And for the field adopting the integer data elastic coding method, restoring the value of the field according to the integer data elastic coding method.

Taking the aforementioned data type int32, field with value 186 as an example, byte 1 is "10111010", and byte 2 is "00000001". The most significant bit of byte 1 is 1, meaning that the last 1 byte (i.e., byte 2) is part of the value of the field; the most significant bit of byte 2 is 0, indicating that the byte is the last 1 byte.

The order of bytes is adjusted. The string "10111010 00000001" is adjusted to "00000001 10111010".

The last 7 bits in "00000001" and "10111010" are extracted respectively to obtain "0000001 0111010"; "0000001 0111010" is converted to decimal and reduced to 186.

After the deserialization processing, the deserialized data is restored into a message close to reality according to the file of the description specification for defining the one-to-one mapping relationship between the class object instance data and the data bucket.

In some embodiments, the detection of a field whose data type is sint32 or sint64 requires the re-decoding of its value using the formula "x= (x > > 1)/(x & 1)", resulting in real data.

According to the method for acquiring the background data, the execution main body can be a device for acquiring the background data. In the embodiment of the present application, a method for executing a method for acquiring background data by using a device for acquiring background data is taken as an example, and the device for acquiring background data provided in the embodiment of the present application is described.

The embodiment of the application also provides a device for acquiring the background data.

As shown in fig. 7, the apparatus for acquiring background data includes: a receiving module 710 and a returning module 720.

A receiving module 710, configured to receive an access request of a client;

a return module 720, configured to return the first data to the client; the first data is a binary data stream obtained by serializing second data corresponding to the access request through a serialization method; the serialization method is used to remove at least a portion of the non-valid values in the second data.

According to the device for acquiring the background data, which is provided by the embodiment of the application, the background data which is subjected to serialization processing by the serialization method is transmitted in the network, so that the data transmission volume is smaller, the requirement of network bandwidth can be reduced to the greatest extent, the network transmission with large data volume can be realized, and the scene of acquiring a large amount of background data can be met. And the transmitted background data is the data processed by the specific serialization method, so that the security of the transmission in the network is higher, and the risk of leakage of the transmission data in the network can be greatly reduced.

In some embodiments, the apparatus may further comprise:

the second acquisition module is used for acquiring second data from the database server;

and the serialization processing module is used for serializing the second data through a serialization method to obtain the first data.

In some embodiments, the serialization processing module may include:

an acquisition unit for acquiring a data sequence spliced by a plurality of data buckets based on the second data; the data barrels are in one-to-one correspondence with class object instance data in the second data; each data bucket is spliced by at least one data block; the data blocks are in one-to-one correspondence with fields in class object instance data corresponding to the data buckets; each data block is spliced by the length and the value of the corresponding field;

the encoding unit is used for converting the value of the field into binary system for each field, acquiring a first binary system, and encoding the value of the valid bit in the first binary system with variable length according to the data type of the field.

In some embodiments, the coding unit may be specifically configured to:

under the condition that the data type of the field is an unsigned number, sequentially intercepting values of 7 significant digits in a first binary number as last 7 digits of one byte according to the sequence from low digits to high digits, and determining the value of the most significant digit of the byte based on whether the significant digit exists in the high digit direction of the 7 significant digits or not until all significant digits are taken; splicing all bytes according to the intercepted sequence;

Converting the first binary number into a second binary number of an unsigned number type under the condition that the data type of the field is a signed number; sequentially intercepting the values of 7 significant digits in the second binary number as the last 7 digits of one byte according to the sequence from the low order to the high order, and determining the value of the highest order of the byte based on whether the significant digits exist in the high order direction of the 7 significant digits until all the significant digits are acquired; splicing all bytes according to the intercepted sequence;

In some embodiments, the apparatus may further comprise:

the first acquisition module is used for attempting to acquire first data from the local cache;

the return module 720 may be specifically configured to return the first data to the client in case the first data is successfully obtained from the local cache.

In some embodiments, the apparatus may further comprise: a storage module and a synchronization module;

the second obtaining module may be specifically configured to obtain, according to a time period, second data from the database server based on the target page displayed by the client;

the storage module is used for storing the first data in the local cache;

And the synchronization module is used for synchronously storing the first data in the cache database.

In some embodiments, the first obtaining module may be further configured to attempt to obtain the first data from the cache database if the first data is not successfully obtained from the local cache;

the second obtaining module may be specifically configured to obtain the second data from the database server in a case where the first data is not successfully obtained from the cache database.

In some embodiments, the return module 720 may be specifically configured to return the first data to the client in case the first data is successfully obtained from the cache database.

In some embodiments, the storage module is further configured to store the first data synchronously in the local cache if the first data is successfully obtained from the cache database.

The device for acquiring the background data in the embodiment of the present application may be an electronic device, or may be a component in the electronic device, for example, an integrated circuit or a chip. The electronic device may be a terminal, or may be other devices than a terminal. By way of example, the electronic device may be a mobile phone, tablet computer, notebook computer, palm computer, vehicle-mounted electronic device, mobile internet appliance (Mobile Internet Device, MID), augmented reality (augmented reality, AR)/Virtual Reality (VR) device, robot, wearable device, ultra-mobile personal computer, UMPC, netbook or personal digital assistant (personal digital assistant, PDA), etc., but may also be a server, network attached storage (Network Attached Storage, NAS), personal computer (personal computer, PC), television (TV), teller machine or self-service machine, etc., and the embodiments of the present application are not limited in particular.

The device for acquiring the background data in the embodiment of the present application may be a device with an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiments of the present application.

The device for acquiring background data provided in the embodiment of the present application can implement each process implemented by the embodiments of the methods of fig. 1 to 6, and in order to avoid repetition, a description is omitted here.

In some embodiments, as shown in fig. 8, an electronic device 800 is further provided in the embodiments of the present application, which includes a processor 810, a memory 820, and a computer program stored in the memory 820 and capable of running on the processor 810, where the program, when executed by the processor 810, implements the processes of the above-described method embodiment of obtaining background data, and the same technical effects are achieved, and for avoiding repetition, a detailed description is omitted herein.

The electronic device in the embodiment of the application includes the mobile electronic device and the non-mobile electronic device described above.

The embodiment of the present application further provides a non-transitory computer readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements each process of the above-described method embodiment for obtaining background data, and can achieve the same technical effect, so that repetition is avoided, and no further description is given here.

Wherein the processor is a processor in the electronic device described in the above embodiment. The readable storage medium includes computer readable storage medium such as computer readable memory ROM, random access memory RAM, magnetic or optical disk, etc.

The embodiment of the application also provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to realize the method for acquiring the background data.

The embodiment of the application further provides a chip, where the chip includes a processor and a communication interface, where the communication interface is coupled to the processor, and the processor is configured to run a program or an instruction, implement each process of the method embodiment for obtaining background data, and achieve the same technical effect, so that repetition is avoided, and no further description is given here.

It should be understood that the chips referred to in the embodiments of the present application may also be referred to as system-on-chip chips, chip systems, or system-on-chip chips, etc.

It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element. Furthermore, it should be noted that the scope of the methods and apparatus in the embodiments of the present application is not limited to performing the functions in the order shown or discussed, but may also include performing the functions in a substantially simultaneous manner or in an opposite order depending on the functions involved, e.g., the described methods may be performed in an order different from that described, and various steps may also be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

From the above description of the embodiments, it will be clear to those skilled in the art that the above-described embodiment method may be implemented by means of software plus a necessary general hardware platform, but of course may also be implemented by means of hardware, but in many cases the former is a preferred embodiment. Based on such understanding, the technical solutions of the present application may be embodied essentially or in a part contributing to the prior art in the form of a computer software product stored in a storage medium (such as ROM/RAM, magnetic disk, optical disk), comprising several instructions for causing a terminal (which may be a mobile phone, a computer, a server, or a network device, etc.) to perform the methods described in the embodiments of the present application.

The embodiments of the present application have been described above with reference to the accompanying drawings, but the present application is not limited to the above-described embodiments, which are merely illustrative and not restrictive, and many forms may be made by those of ordinary skill in the art without departing from the spirit of the present application and the scope of the claims, which are also within the protection of the present application.

In the description of the present specification, reference to the terms "one embodiment," "some embodiments," "illustrative embodiments," "examples," "specific examples," or "some examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the present application. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiments or examples. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

While embodiments of the present application have been shown and described, it will be understood by those of ordinary skill in the art that: many changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the application, the scope of which is defined by the claims and their equivalents.

Claims

1. A method of obtaining background data, comprising:

receiving an access request of a client;

2. The method of claim 1, wherein the method further comprises, prior to returning the first data to the client and after receiving the access request of the client:

obtaining the second data from a database server;

3. The method for obtaining background data according to claim 2, wherein the serializing the second data by the serializing method includes:

4. A method of obtaining background data according to claim 3, wherein said variable length encoding the value of the significant bit in the first binary number according to the data type of the field comprises:

5. The method of claim 1, wherein the method further comprises, prior to returning the first data to the client and after receiving the access request of the client:

attempting to obtain the first data from a local cache;

6. The method of claim 5, wherein prior to attempting to obtain the first data from a local cache, the method further comprises:

7. The method of acquiring background data according to any one of claims 2 to 4, wherein the acquiring the second data from a database server comprises:

Attempting to obtain the first data from a local cache;

8. The method of claim 7, wherein, in the event that the first data is not successfully obtained from the local cache, attempting to obtain the first data from a cache database, the method further comprises:

9. The method of claim 8, wherein, in the event that the first data is not successfully obtained from the local cache, attempting to obtain the first data from a cache database, the method further comprises:

10. An apparatus for obtaining background data, comprising:

the receiving module is used for receiving the access request of the client;