CN113760986A - Data query method, device, equipment and storage medium - Google Patents
Data query method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN113760986A CN113760986A CN202110129861.XA CN202110129861A CN113760986A CN 113760986 A CN113760986 A CN 113760986A CN 202110129861 A CN202110129861 A CN 202110129861A CN 113760986 A CN113760986 A CN 113760986A
- Authority
- CN
- China
- Prior art keywords
- data
- pipeline
- batch
- query request
- packets
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24553—Query execution of query operations
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/252—Integrating or interfacing systems involving database management systems between a Database Management System and a front-end application
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the invention discloses a data query method, which is applied to a server, wherein the server comprises a service interface layer and a data processing layer, and the method comprises the following steps: the service interface layer responds to the detected data query request, creates a data pipeline corresponding to the data query request, calls a data generation interface of the data processing layer, and transmits the data pipeline and the data query request to the data processing layer; the data processing layer acquires batch data packets corresponding to the data query request and writes the batch data packets into the data pipeline; the service interface layer circularly reads the batch data packets in the data pipeline and generates response information based on the batch data packets. According to the method provided by the embodiment of the invention, the target data corresponding to the data query request is transmitted in batches by creating the data pipeline, so that the occupation of the memory when the query data volume is large or the concurrent number is large is reduced, the use of the system memory is more reasonable, and the stability of the system is improved.
Description
Technical Field
The embodiment of the invention relates to the technical field of computers, in particular to a data query method, a data query device, data query equipment and a storage medium.
Background
With the rapid development of information technology and the popularization of the internet, the data volume has a leap growth, the data storage scale of application services and the data access volume are increased, the traditional relational database cannot meet the requirements, and the non-relational database is produced at the same time. Key value storage is a typical storage mode of a non-relational database, data in a key value storage system is organized and stored in a key value pair mode, and due to a performance expansion mode of transverse expansion, the key value storage system can better meet the requirements of the current big data era, and becomes one of mainstream modes of the current storage industry.
In the process of implementing the invention, the inventor finds that at least the following technical problems exist in the prior art: the existing key value storage system has unreasonable memory use control, so that the system stability is poor, and the system is crashed when the query data volume is overlarge or the request concurrency is more.
Disclosure of Invention
The embodiment of the invention provides a data query method, a data query device, data query equipment and a data query storage medium, so that the memory use of a key value storage system is reasonably controlled, and the stability of the system is improved.
In a first aspect, an embodiment of the present invention provides a data query method applied to a server, where the server includes a service interface layer and a data processing layer, and the method includes:
the service interface layer responds to the detected data query request, creates a data pipeline corresponding to the data query request, calls a data generation interface of the data processing layer, and transmits the data pipeline and the data query request to the data processing layer;
the data processing layer acquires batch data packets corresponding to the data query request and writes the batch data packets into the data pipeline in sequence;
the service interface layer cyclically reads the batch data packets in the data pipeline and generates response information based on the batch data packets.
In a second aspect, an embodiment of the present invention further provides a data query apparatus, including:
the data pipeline creating module is configured on the service interface layer and used for responding to the detected data query request, creating a data pipeline corresponding to the data query request, calling a data generating interface of the data processing layer and transmitting the data pipeline and the data query request to the data processing layer;
the target data writing module is configured on the data processing layer and used for acquiring batch data packets corresponding to the data query request and writing the batch data packets into the data pipeline;
and the response information generating module is configured at the service interface layer and used for circularly reading the batch data packets in the data pipeline and generating response information based on the batch data packets.
In a third aspect, an embodiment of the present invention further provides a computer device, where the computer device includes:
one or more processors;
storage means for storing one or more programs;
when the one or more programs are executed by the one or more processors, the one or more processors are caused to implement the data query method as provided by any embodiment of the invention.
In a fourth aspect, the embodiments of the present invention further provide a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the data query method provided in any embodiment of the present invention.
The embodiment of the invention responds to the detected data query request through the service interface layer, creates a data pipeline corresponding to the data query request, calls a data generation interface of the data processing layer, and transmits the data pipeline and the data query request to the data processing layer; the data processing layer acquires batch data packets corresponding to the data query request and writes the batch data packets into the data pipeline in sequence; the service interface layer circularly reads the batch data packets in the data pipeline, generates response information based on the batch data packets, and transmits target data corresponding to the data query request in batches by establishing the data pipeline, so that the occupation of a memory when the query data volume is large or the concurrent number is large is reduced, the use of the system memory is more reasonable, and the stability of the system is improved.
Drawings
FIG. 1 is a flowchart of a data query method according to an embodiment of the present invention;
fig. 2a is a schematic flowchart of a data query method of a server of a key-value storage system according to a second embodiment of the present invention;
fig. 2b is a flowchart illustrating a data query method for a key value storage system client according to a second embodiment of the present invention
Fig. 3 is a schematic structural diagram of a data query device according to a third embodiment of the present invention;
fig. 4 is a schematic structural diagram of a computer device according to a fourth embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting of the invention. It should be further noted that, for the convenience of description, only some of the structures related to the present invention are shown in the drawings, not all of the structures.
Example one
Fig. 1 is a flowchart of a data query method according to an embodiment of the present invention. The embodiment is applicable to the case when the key-value storage system responds to the received data query request. The method can be executed by a data query device, which can be implemented in a software and/or hardware manner, for example, the data query device can be configured at a server of a key-value storage system, and the server of the key-value storage system can be presented in the form of a computer device. As shown in fig. 1, the method includes:
s110, the service interface layer responds to the detected data query request, creates a data pipeline corresponding to the data query request, calls a data generation interface of the data processing layer, and transmits the data pipeline and the data query request to the data processing layer.
In this embodiment, the server includes a service interface layer and a data processing layer. The service interface layer realizes information interaction with the application end, and the data processing layer realizes data query and other operations. For example, after detecting a data query operation triggered by a user, an application terminal generates a data query request and sends the data query request to a server terminal, a service interface layer of the server terminal forwards the data query request to a data processing layer, so that the data processing layer queries target data corresponding to the data query request, the data processing layer transmits the target data to the service interface layer, and the service interface layer generates response information based on the target data to respond.
When the data processing layer queries the target data corresponding to the data query request, the queried target data needs to be cached, and after the query is completed, all the target data are packaged and transmitted to the service interface layer. However, when the amount of data is too large or the amount of concurrency is too large, the system is easy to crash when the memory usage is too large. In this embodiment, in order to avoid system crash caused by excessive memory occupation, when the data processing layer performs query, data transmission is not performed until the query is completed, but after a batch of target data is accumulated, the target data is sent to the service interface layer, and the service interface layer sends the target data corresponding to the data query request to the application terminal in batches and in batches.
In the embodiment, the target data is transmitted in batches and in times by establishing the data pipeline. Specifically, after receiving the data query request, the service interface layer establishes a data pipeline corresponding to the data query request, and is used for data transmission of the data query request. After the pipeline is established, the data query request and the pipeline are transmitted to the data generation interface of the data processing layer, and then the data transmission of the service interface layer and the data processing layer can be realized.
In one embodiment, creating a data pipe corresponding to the data query request includes: and calling a pipeline creating function to create a data pipeline, wherein execution parameters of the pipeline creating function comprise a pipeline identifier and a data type, and the data type represents a data transmission type in the data pipeline. Taking the Go language as an example, the pipe creation function may be ch1 equal to make (chan datatype), where ch1 is a pipe identifier of the data pipe, and the datatype represents a datatype that can be transmitted within the data pipe. For example, the data type may be set to be integer, and an integer type data pipe is created for transmitting integer type data; the data type can also be set as an empty interface type, and a data pipeline of the empty interface type is created for transmitting data of any format type. It will be appreciated that the data type may be set according to actual requirements.
S120, the data processing layer obtains batch data packets corresponding to the data query request and writes the batch data packets into the data pipeline in sequence.
And after receiving the data pipeline and the data query request, the data processing layer queries target data corresponding to the data query request, generates a plurality of batch data packets, and sequentially writes the batch data packets into the data pipeline. Optionally, a data amount threshold may be preset, and the data amount of the batch data packets is set to be not greater than the preset data amount threshold, and the data amount threshold may be set so that the data processing layer does not occupy too large memory when obtaining the target data cache, thereby ensuring the availability of the memory when the total amount of data queries is too large or too many concurrences are performed.
Optionally, the obtaining, by the data processing layer, the batch data packets corresponding to the data query request includes: calling a key value database to inquire target data corresponding to the data query request, storing the data into a cache array, packing the target data in the cache array into batch data packets when the data in the cache array reaches a preset data volume, and writing the batch data packets into a data pipeline; and continuing to query the data corresponding to the data query request until the data corresponding to the data processing query request is queried. After the data processing layer inquires the target data, caching the target data into a cache array, when the data volume in the cache array reaches a preset data volume threshold value, packaging the target data in the cache data into batch data packets, putting the batch data packets into a data pipeline, continuously acquiring the target data, generating the batch data packets, and writing the batch data packets into the data pipeline until all the target data are inquired. Taking a key value database as BoltDB as an example, a data processing layer calls a kvIndex index tree according to a query request condition of a data query request, obtains version review lists of all data to be queried through keywords, calls the database according to each version to query a corresponding value KeyValue, stores the queried data into a cache array, and writes the data into a data pipeline passed by a service interface layer as a batch data packet to circulate when a batch of data is accumulated until all the data is written into the data pipeline. When the target data is inquired in the data processing layer, the cached versions are not large, and the memory occupied by each version is small, so that the memory occupied by all versions obtained at one time cannot be too large when the concurrency is high, and the stability of the system is improved. The batch data packets may be written into the data pipe by a data write function. The data write function may be determined according to the development language, and in the Go language for example, the batch data packet XX may be written into the data pipe by the data write function DataChan ← XX.
The data processing layer may add an identifier indicating whether the data query is completed when the batch data packet is generated, so that the service interface layer determines whether to continue reading the data in the data pipeline according to the identifier. Other means may also be used to indicate completion of service interface layer data transfer. In an embodiment of the present invention, after all the batch data packets are written into the data pipe, an empty data packet is generated and written into the data pipe, so that the service interface layer confirms that the transmission of the target data corresponding to the data query request is completed according to the empty data packet. Optionally, after all target data are transmitted, generating an empty data packet to write into the data pipeline, and after the service interface layer reads the empty data packet, confirming that the target data corresponding to the data query request are transmitted.
S130, the service interface layer circularly reads the batch data packets in the data pipeline and generates response information based on the batch data packets.
And after the data pipeline is sent to the data processing layer, circularly reading the data in the data pipeline, after batch data packets in the data pipeline are read, generating response information based on the batch data packets, and sending the response information to the application end. It can be understood that, in this embodiment, there are a plurality of batch data packets, and each time a service interface layer reads one batch data packet, a response message is formed, and the use of the system memory is more reasonable by allocating and returning batch data packets in several times. Optionally, the cyclically reading, by the service interface layer, the batch data packets in the data pipe includes: and the service interface layer circularly calls a pipeline data reading function to read the batch data packets in the data pipeline. The pipeline data reading function can be determined according to a development language, and taking Go language as an example, batch data packets in the data pipeline can be read through the pipeline data reading function data ═ to ← dataChan.
Optionally, generating response information based on the batch data packet includes: and after filling the packet header information in the batch data packets, sequencing the batch data packets to generate response information. After reading the batch data packets, filling packet header information response headers, including cluster identifiers, processing node membersids, expiration numbers Raft Term, data version information Revision and the like, then performing serialization of network IO on the batch data packets filled with header information to generate response data, returning the response data to the client, and circulating the process. The batch data packets can be serialized into network IO through a Grpc Response tool. In the above process, the loop stop determination mode may be various and may be determined according to the setting of the data processing layer. For example, whether the target data transmission is completed or not may be confirmed according to the identifier in the batch data packet, or the target data transmission is completed after the batch data packet is not read within a set time period, or the target data transmission is completed when an empty data packet in the data pipeline is read.
Optionally, after reading the empty data packet in the data pipeline, the service interface layer generates an empty data packet and sends the empty data packet to the application end, so that the application end confirms that the transmission of the target data corresponding to the data query request is completed according to the empty data packet. Specifically, the null data packet indicates that the target data transmission is completed, the service interface layer generates a null data packet and sends the null data packet to the application terminal after reading the null data packet in the data pipeline, and the application terminal confirms that the target data transmission corresponding to the data query request is completed after receiving the null data.
It should be noted that the application end may adjust the logic of the received data according to actual requirements. In order to ensure that the application program of the service does not change codes as much as possible, the data receiving logic driven by the client of the system can be adjusted to ensure that the target data corresponding to the data query request is returned to the application program at one time, and the application program only needs to adjust the interface. Preferably, a streaming query method can be used at the bottom layer of client-driven network IO communication, and each batch of data is received and added to the memory array list until all the queried data are received, and the data are returned to the application program once again.
The embodiment of the invention responds to the detected data query request through the service interface layer, creates a data pipeline corresponding to the data query request, calls a data generation interface of the data processing layer, and transmits the data pipeline and the data query request to the data processing layer; the data processing layer acquires batch data packets corresponding to the data query request and writes the batch data packets into the data pipeline in sequence; the service interface layer circularly reads the batch data packets in the data pipeline, generates response information based on the batch data packets, and transmits target data corresponding to the data query request in batches by establishing the data pipeline, so that the occupation of a memory when the query data volume is large or the concurrent number is large is reduced, the use of the system memory is more reasonable, and the stability of the system is improved.
Example two
The present embodiment provides a preferred embodiment based on the above-described scheme. In this embodiment, a key value storage system is embodied as an ETCD, and a data query method is described.
Fig. 2a is a flowchart illustrating a data query method of a server of a key-value storage system according to a second embodiment of the present invention. As shown in fig. 2a, the key-value storage system is embodied as an ETCD, after a service interface layer of an ETCD server receives a data query request (ranging request) sent by a client, a data pipeline corresponding to the request is created, a lower-layer data generation interface is asynchronously called, and the data pipeline is transmitted to the interface, so that data generated at the lower layer can be returned to the local layer through the data pipeline for processing. And circulating the data pipeline, filling a header information Response header after receiving a batch of data, wherein the header information Response header comprises a cluster ID, a processing node memberID, a task number Raft Term and data version information Revision, and then serializing the batch of data through a Grpc Response to carry out network IO serialization and returning the data to the client. The above process is looped until NULL data is received.
After receiving the data pipeline transmitted by the upper layer, the data processing layer calls the memory index tree kvIndex according to the request condition to be queried by the service, obtains a replay list of all data to be queried through a keyword, calls the key value database according to each replay version to query the corresponding value KeyValue, stores the value KeyValue into the buffer of the cache array, writes the accumulated data into the data pipeline transmitted by the upper layer to return to the upper layer every time a batch of data is accumulated, and repeats until all data are returned.
Fig. 2b is a flowchart illustrating a data query method for a key-value storage system client according to a second embodiment of the present invention. As shown in fig. 2b, the method of streaming queries is used within the client driver. Specifically, the application program only needs to add a calling interface of the streaming query, for example, changing the range () interface to the range stream () interface as an interface call of the streaming query. And (3) using a stream query method at the bottom layer of the client-side driven network IO communication, adding the data into the memory array list after receiving a batch of data until all the queried data are returned, and returning the data to the application program once again.
According to the embodiment of the invention, when the key value storage system is used for data query, a chain that the memory use is in direct proportion to the data scale is broken through, no matter how much data quantity is stored in the database, data can be returned through stream query, the returned data quantity of each batch is very small, and the memory of the key value storage system can be increased in a geometric mode under the condition that high concurrency is not needed to be worried about, so that a powerful guarantee is provided for the stability of the key value storage system on a cluster line in the data query environment, the use of the memory is controlled more reasonably when a service conducts high concurrency on the key value storage system and queries mass data, and the problem of system stability such as memory overflow caused by instant full occupation of the memory is avoided.
EXAMPLE III
Fig. 3 is a schematic structural diagram of a data query apparatus according to a third embodiment of the present invention. The data query device can be implemented in software and/or hardware, for example, the data query device can be configured in a computer device. As shown in fig. 3, the apparatus includes a data pipe creation module 310, a target data writing module 320, and a response information generation module 330, wherein:
a data pipeline creating module 310 configured at the service interface layer, and configured to respond to the detected data query request, create a data pipeline corresponding to the data query request, call a data generating interface of the data processing layer, and transmit the data pipeline and the data query request to the data processing layer;
the target data writing module 320 is configured on the data processing layer and is configured to obtain batch data packets corresponding to the data query request and write the batch data packets into the data pipeline;
and the response information generating module 330 is configured at the service interface layer and configured to cyclically read the batch data packets in the data pipeline and generate response information based on the batch data packets.
The embodiment of the invention responds to the detected data query request through the service interface layer, creates a data pipeline corresponding to the data query request, calls a data generation interface of the data processing layer, and transmits the data pipeline and the data query request to the data processing layer; the data processing layer acquires batch data packets corresponding to the data query request and writes the batch data packets into the data pipeline in sequence; the service interface layer circularly reads the batch data packets in the data pipeline, generates response information based on the batch data packets, and transmits target data corresponding to the data query request in batches by establishing the data pipeline, so that the occupation of a memory when the query data volume is large or the concurrent number is large is reduced, the use of the system memory is more reasonable, and the stability of the system is improved.
Optionally, on the basis of the foregoing scheme, the data pipeline creating module 310 is specifically configured to:
and calling a pipeline creating function to create a data pipeline, wherein execution parameters of the pipeline creating function comprise a pipeline identifier and a data type, and the data type represents a data transmission type in the data pipeline.
Optionally, on the basis of the above scheme, the response information generating module 330 is specifically configured to:
and circularly calling a pipeline data reading function to read the batch data packets in the data pipeline.
Optionally, on the basis of the above scheme, the target data writing module 320 is specifically configured to:
calling a key value database to inquire target data corresponding to the data query request, storing the data into a cache array, packing the target data in the cache array into batch data packets when the data in the cache array reaches a preset data volume, and writing the batch data packets into a data pipeline;
and continuing to query the data corresponding to the data query request until the data corresponding to the data processing query request is queried.
Optionally, on the basis of the foregoing scheme, the target data writing module 320 is further configured to:
and when all the batch data packets are written into the data pipeline, generating empty data packets to be written into the data pipeline so that the service interface layer confirms that the target data corresponding to the data query request is transmitted according to the empty data packets.
Optionally, on the basis of the foregoing scheme, the response information generating module 330 is further configured to:
and after filling the packet header information in the batch data packets, sequencing the batch data packets to generate response information.
The data query device provided by the embodiment of the invention can execute the data query method provided by any embodiment of the invention, and has corresponding functional modules and beneficial effects of the execution method.
Example four
Fig. 4 is a schematic structural diagram of a computer device according to a fourth embodiment of the present invention. FIG. 4 illustrates a block diagram of an exemplary computer device 412 suitable for use in implementing embodiments of the present invention. The computer device 412 shown in FIG. 4 is only one example and should not impose any limitations on the functionality or scope of use of embodiments of the present invention.
As shown in FIG. 4, computer device 412 is in the form of a general purpose computing device. Components of computer device 412 may include, but are not limited to: one or more processors 416, a system memory 428, and a bus 418 that couples the various system components (including the system memory 428 and the processors 416).
The system memory 428 may include computer system readable media in the form of volatile memory, such as Random Access Memory (RAM)430 and/or cache memory 432. The computer device 412 may further include other removable/non-removable, volatile/nonvolatile computer system storage media. By way of example only, storage 434 may be used to read from and write to non-removable, nonvolatile magnetic media (not shown in FIG. 4, and commonly referred to as a "hard drive"). Although not shown in FIG. 4, a magnetic disk drive for reading from and writing to a removable, nonvolatile magnetic disk (e.g., a "floppy disk") and an optical disk drive for reading from or writing to a removable, nonvolatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 418 by one or more data media interfaces. Memory 428 can include at least one program product having a set (e.g., at least one) of program modules that are configured to carry out the functions of embodiments of the invention.
A program/utility 440 having a set (at least one) of program modules 442 may be stored, for instance, in memory 428, such program modules 442 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which examples or some combination thereof may comprise an implementation of a network environment. The program modules 442 generally perform the functions and/or methodologies of the described embodiments of the invention.
The computer device 412 may also communicate with one or more external devices 414 (e.g., keyboard, pointing device, display 424, etc.), with one or more devices that enable a user to interact with the computer device 412, and/or with any devices (e.g., network card, modem, etc.) that enable the computer device 412 to communicate with one or more other computing devices. Such communication may occur via input/output (I/O) interfaces 422. Also, computer device 412 may communicate with one or more networks (e.g., a Local Area Network (LAN), a Wide Area Network (WAN) and/or a public network, such as the Internet) through network adapter 420. As shown, network adapter 420 communicates with the other modules of computer device 412 over bus 418. It should be appreciated that although not shown in the figures, other hardware and/or software modules may be used in conjunction with the computer device 412, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, among others.
The processor 416 executes programs stored in the system memory 428 to perform various functional applications and data processing, such as implementing a data query method provided by an embodiment of the present invention, the method including:
the service interface layer responds to the detected data query request, creates a data pipeline corresponding to the data query request, calls a data generation interface of the data processing layer, and transmits the data pipeline and the data query request to the data processing layer;
the data processing layer acquires batch data packets corresponding to the data query request and writes the batch data packets into the data pipeline;
the service interface layer cyclically reads the batch data packets in the data pipeline and generates response information based on the batch data packets.
Of course, those skilled in the art can understand that the processor can also implement the technical solution of the data query method provided by any embodiment of the present invention.
EXAMPLE five
The fifth embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the data query method provided in the fifth embodiment of the present invention, and the method includes:
the service interface layer responds to the detected data query request, creates a data pipeline corresponding to the data query request, calls a data generation interface of the data processing layer, and transmits the data pipeline and the data query request to the data processing layer;
the data processing layer acquires batch data packets corresponding to the data query request and writes the batch data packets into the data pipeline;
the service interface layer cyclically reads the batch data packets in the data pipeline and generates response information based on the batch data packets.
Of course, the computer program stored on the computer-readable storage medium provided by the embodiments of the present invention is not limited to the above method operations, and may also perform related operations of the data query method provided by any embodiments of the present invention.
Computer storage media for embodiments of the invention may employ any combination of one or more computer-readable media. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Computer program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + +, or the like, as well as conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present invention and the technical principles employed. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments illustrated herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the present invention has been described in greater detail by the above embodiments, the present invention is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present invention, and the scope of the present invention is determined by the scope of the appended claims.
Claims (10)
1. A data query method is applied to a server, wherein the server comprises a service interface layer and a data processing layer, and the method comprises the following steps:
the service interface layer responds to the detected data query request, creates a data pipeline corresponding to the data query request, calls a data generation interface of the data processing layer, and transmits the data pipeline and the data query request to the data processing layer;
the data processing layer acquires batch data packets corresponding to the data query request and writes the batch data packets into the data pipeline;
the service interface layer circularly reads the batch data packets in the data pipeline and generates response information based on the batch data packets.
2. The method of claim 1, wherein the creating of the data pipe corresponding to the data query request comprises:
and calling a pipeline creating function to create the data pipeline, wherein the execution parameters of the pipeline creating function comprise a pipeline identifier and a data type, and the data type represents the data transmission type in the data pipeline.
3. The method of claim 2, wherein the service interface layer cyclically reads the batch of packets in the data pipe, comprising:
and the service interface layer circularly calls a pipeline data reading function to read the batch data packets in the data pipeline.
4. The method of claim 1, wherein the data processing layer obtains batch data packets corresponding to the data query request and writes the batch data packets into the data pipeline, and the method comprises:
calling a key value database to inquire target data corresponding to the data inquiry request, storing the data into a cache array, packing the target data in the cache array into batch data packets when the data in the cache array reaches a preset data volume, and writing the batch data packets into the data pipeline;
and continuing to query the data corresponding to the data query request until the data corresponding to the data processing query request is queried.
5. The method of claim 4, further comprising:
and when all the batch data packets are written into the data pipeline, generating empty data packets and writing the empty data packets into the data pipeline so that the service interface layer confirms that the target data corresponding to the data query request is transmitted completely according to the empty data packets.
6. The method according to claim 5, wherein after the service interface layer reads the null data packet in the data pipeline, a null data packet is generated and sent to an application end, so that the application end confirms that the transmission of the target data corresponding to the data query request is completed according to the null data packet.
7. The method of claim 1, wherein generating response information based on the batch data packets comprises:
and after filling packet header information in the batch data packets, carrying out serialization on the batch data packets to generate the response information.
8. A data query apparatus, comprising:
the data pipeline creating module is configured on the service interface layer and used for responding to the detected data query request, creating a data pipeline corresponding to the data query request, calling a data generating interface of the data processing layer and transmitting the data pipeline and the data query request to the data processing layer;
the target data writing module is configured on the data processing layer and used for acquiring batch data packets corresponding to the data query request and writing the batch data packets into the data pipeline;
and the response information generation module is configured at the service interface layer and used for circularly reading the batch data packets in the data pipeline and generating response information based on the batch data packets.
9. A computer device, the device comprising:
one or more processors;
storage means for storing one or more programs;
when executed by the one or more processors, cause the one or more processors to implement the data query method of any one of claims 1-7.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the data query method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110129861.XA CN113760986A (en) | 2021-01-29 | 2021-01-29 | Data query method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110129861.XA CN113760986A (en) | 2021-01-29 | 2021-01-29 | Data query method, device, equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113760986A true CN113760986A (en) | 2021-12-07 |
Family
ID=78786506
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110129861.XA Pending CN113760986A (en) | 2021-01-29 | 2021-01-29 | Data query method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113760986A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114201319A (en) * | 2022-02-17 | 2022-03-18 | 广东东华发思特软件有限公司 | Data scheduling method, device, terminal and storage medium |
-
2021
- 2021-01-29 CN CN202110129861.XA patent/CN113760986A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114201319A (en) * | 2022-02-17 | 2022-03-18 | 广东东华发思特软件有限公司 | Data scheduling method, device, terminal and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111078147B (en) | Processing method, device and equipment for cache data and storage medium | |
| CN107273213B (en) | Calculation control method, network card and electronic equipment | |
| CN113094430B (en) | Data processing method, device, equipment and storage medium | |
| CN112130748A (en) | Data access method, network card and server | |
| US20120224482A1 (en) | Credit feedback system for parallel data flow control | |
| CN109857545B (en) | Data transmission method and device | |
| CN112433920A (en) | Buried point data reporting method, device, equipment and medium | |
| CN111339057A (en) | Method, apparatus and computer readable storage medium for reducing back-to-source requests | |
| CN108337301A (en) | Network request processing method, device, server and the storage medium of application program | |
| US6986147B2 (en) | Method for transparent, location-independent, remote procedure calls in a heterogeneous network environment | |
| CN112835885B (en) | Processing method, device and system for distributed form storage | |
| CN111966653A (en) | Data processing method, device, server and storage medium for micro-service call link | |
| US11190620B2 (en) | Methods and electronic devices for data transmission and reception | |
| CN111338834A (en) | Data storage method and device | |
| US20150074316A1 (en) | Reflective memory bridge for external computing nodes | |
| US8938522B2 (en) | Tiered XML services in a content management system | |
| CN113760986A (en) | Data query method, device, equipment and storage medium | |
| US20090157896A1 (en) | Tcp offload engine apparatus and method for system call processing for static file transmission | |
| CN114584618A (en) | Information interaction method, device, equipment, storage medium and system | |
| CN105491082A (en) | Remote resource access method and switch equipment | |
| CN116541461A (en) | Data processing method, device, equipment and storage medium applied to database | |
| US8688867B2 (en) | System and methods for communicating between serial communications protocol enabled devices | |
| CN115426375A (en) | Data processing method and data processing system | |
| CN111949648A (en) | Memory cache data system and data indexing method | |
| CN113051244B (en) | Data access method and device, and data acquisition method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |