WO2021179170A1 - 数据推送方法、装置、服务器及存储介质 - Google Patents

数据推送方法、装置、服务器及存储介质 Download PDF

Info

Publication number
WO2021179170A1
WO2021179170A1 PCT/CN2020/078634 CN2020078634W WO2021179170A1 WO 2021179170 A1 WO2021179170 A1 WO 2021179170A1 CN 2020078634 W CN2020078634 W CN 2020078634W WO 2021179170 A1 WO2021179170 A1 WO 2021179170A1
Authority
WO
WIPO (PCT)
Prior art keywords
data
query
handle
thread
handles
Prior art date
Application number
PCT/CN2020/078634
Other languages
English (en)
French (fr)
Inventor
程际
Original Assignee
深圳市欢太科技有限公司
Oppo广东移动通信有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 深圳市欢太科技有限公司, Oppo广东移动通信有限公司 filed Critical 深圳市欢太科技有限公司
Priority to CN202080094645.XA priority Critical patent/CN115004168A/zh
Priority to PCT/CN2020/078634 priority patent/WO2021179170A1/zh
Publication of WO2021179170A1 publication Critical patent/WO2021179170A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24—Querying

Definitions

  • This application relates to the field of data processing technology, and more specifically, to a data push method, device, server, and storage medium.
  • Information recommendation technology mainly recommends related information based on user tag data.
  • information recommendation there are usually business parties that query tag data, and it takes more time to pull and send data from the database.
  • this application proposes a data push method, device, server, and storage medium.
  • an embodiment of the present application provides a data push method, the method includes: obtaining a data query request sent by a business end; obtaining multiple query handles corresponding to a data query result according to the data query request; Each of the multiple query handles is assigned to a thread, and the thread corresponding to each query handle is different; each thread pulls the data query result according to the corresponding query handle and sends the data query result to all The business side.
  • an embodiment of the present application provides a data push device, the device includes: a request acquisition module, a handle acquisition module, a handle allocation module, and a data sending module, wherein the request acquisition module is used to acquire The data query request; the handle acquisition module is used to obtain multiple query handles corresponding to the data query result according to the data query request; the handle allocation module is used to assign each query handle of the multiple query handles Assigned to a thread, each query handle corresponds to a different thread; the data sending module is used for each thread to pull the data query result according to the corresponding query handle and send the data query result to the service end.
  • an embodiment of the present application provides a server, including: one or more processors; a memory; one or more application programs, wherein the one or more application programs are stored in the memory and are It is configured to be executed by the one or more processors, and the one or more programs are configured to execute the data pushing method provided in the above-mentioned first aspect.
  • an embodiment of the present application provides a computer-readable storage medium.
  • the computer-readable storage medium stores program code, and the program code can be invoked by a processor to execute the data provided in the first aspect. Push method.
  • the solution provided by this application obtains a data query request sent by a business end, according to the query request, obtains multiple query handles corresponding to the data query result, and assigns each query handle of the multiple query handles to a thread, where, Each query handle corresponds to a different thread. Then, according to the corresponding query handle, each thread pulls the data query result and sends the data query result to the business end, so that it can pull and send data in parallel, and improve the data push speed.
  • Fig. 1 shows a flowchart of a data pushing method according to an embodiment of the present application.
  • Fig. 2 shows a flowchart of a data pushing method according to another embodiment of the present application.
  • Fig. 3 shows a flowchart of a data pushing method according to another embodiment of the present application.
  • Fig. 4 shows a flowchart of a data pushing method according to still another embodiment of the present application.
  • Fig. 5 shows a block diagram of a data pushing device according to an embodiment of the present application.
  • Fig. 6 is a block diagram of a server for executing the data push method according to an embodiment of the present application.
  • FIG. 7 is a storage unit for storing or carrying program code for implementing the data pushing method according to the embodiment of the present application according to an embodiment of the present application.
  • the business side queries the tag data, and the data side pulls data from the database and sends it to the business side according to the data requirements of the business side. Therefore, the business party can determine the push content, determine the push target, etc. according to the obtained data.
  • the inventors proposed the data push method, device, server, and storage medium provided by the embodiments of the present application.
  • each thread is based on the sharding Query the handle, pull the data in parallel and send the data to the business end.
  • the specific data pushing method will be described in detail in the subsequent embodiments.
  • FIG. 1 shows a schematic flowchart of a data push method provided by an embodiment of the present application.
  • the data pushing method is applied to the data pushing device 400 shown in FIG. 5 and the server 100 configured with the data pushing device 400 (FIG. 6).
  • the following will take a server as an example to describe the specific process of this embodiment.
  • the server applied in this embodiment may be a traditional server, a cloud server, etc., which is not limited here.
  • the process shown in Fig. 1 will be described in detail below, and the data push method may specifically include the following steps:
  • Step S110 Obtain the data query request sent by the service end.
  • the business end may send a data query request to the server when it needs to query data.
  • the server can receive the data query request sent by the business end, respond to the data query request, and return the corresponding data to the business end.
  • the data query request may carry the data requirements of the business party corresponding to the business end, for example, the query condition may be carried.
  • the query conditions can include one or more conditions.
  • the business side wants to advertise for users who have installed both Application 1 and Application 2, but not installed Application 3, and set the query conditions as users who have installed both Application 1 and Application 2, but have not installed Application 3; another example ,
  • the game company wants to increase the retention rate of recent active game players, and wants to give game rewards to recent active game players.
  • the query conditions set are users who have played more than 24 hours or logged in more than 10 times in the last week. .
  • the data query request sent by the business end may be a request for obtaining tag data, or a request for obtaining other types of data, for example, obtaining document data.
  • the type of data used by the data query request may not be used as a limitation.
  • Step S120 Acquire multiple query handles corresponding to the data query result according to the data query request.
  • the server may determine the data query result according to the data query request.
  • the data query result may include data corresponding to the data query request.
  • the server can determine which data meets the demand information according to the demand information carried in the data query request. For example, if the data query request is a request for querying label data, and the data query request carries users who have installed application A and application B, the server can determine which users have installed application A and application B. For another example, if a data query request is a request for querying document data, and the data query request carries keywords A and B, the server can determine which documents include keywords A and B.
  • the server may determine the query handle corresponding to the data query result according to the data query result.
  • the query handle may be information used to read the data query result.
  • the query handle may include storage location, data identification, fields, and so on. According to the query handle, the corresponding data query result can be read.
  • the server may pre-store the query handle corresponding to each data, and the server may determine the query handle corresponding to the data query result according to the data included in the data query result.
  • the server can divide the data query result into multiple parts of the data query result according to the data query result, and determine the query handle corresponding to the data query result of each part, so that multiple data query results corresponding to the data query result can be obtained.
  • Query handle It is understandable that the obtained multiple query handles can be used to read data respectively, and the data read by the multiple query handles can constitute a complete data query result.
  • the server may divide the data query result into multiple parts according to the location of the data distribution in the data query result, and determine the query handle corresponding to each part.
  • the data corresponding to the data query result is stored in the database cluster, so multiple databases storing the data query result can be determined, and the query handle can be determined according to the data query results included in the multiple databases.
  • the data corresponding to the data query result is stored in different shards in the database, so multiple shards where the data query result is located can be determined, and the query handle can be determined based on the data query results existing in the multiple shards.
  • the server may also divide the data query result according to different types of data in the data query result, and determine the query handle corresponding to each part of the data query result according to the divided result.
  • Step S130 Allocate each query handle of the multiple query handles to a thread, and the thread corresponding to each query handle is different.
  • the server after the server determines multiple query handles corresponding to the data query result, it can allocate a different thread for each query handle, so as to read data from the database and send it to the service end.
  • a thread pool can be created in the server, and the server can allocate a different thread for each query handle from the thread pool, and each thread is used to read the corresponding query handle from the database separately.
  • the data of the data query result can be created in the server, and the server can allocate a different thread for each query handle from the thread pool, and each thread is used to read the corresponding query handle from the database separately. The data of the data query result.
  • Step S140 Each thread pulls the data query result according to the corresponding query handle and sends the data query result to the service end.
  • each thread can pull the data query result from the database according to the allocated query handle, and while pulling the data, the data Send to the business end to avoid sending the data after pulling the data at one time, resulting in excessive memory usage and memory overflow. For example, when the business end is Kafka, after each thread pulls the data, it sends the data to Kafka.
  • each query handle of the multiple query handles is assigned to one Thread, where each query handle corresponds to a different thread, and then each thread pulls the data query result according to the corresponding query handle and sends the data query result to the business end, so that parallel data can be pulled and sent.
  • the method of pulling and sending is adopted, which may not occupy too much memory and ensure the stable operation of the program.
  • FIG. 2 shows a schematic flowchart of a data pushing method provided by another embodiment of the present application.
  • This data push method can be applied to the above-mentioned server. The following will elaborate on the process shown in Figure 2.
  • the data push method may specifically include the following steps:
  • Step S210 Obtain the data query request sent by the service end.
  • step S210 can refer to the content of the foregoing embodiment, which will not be repeated here.
  • Step S220 Determine multiple segments corresponding to the data query result according to the data query request.
  • the data when storing data in the database, the data may be fragmented.
  • the fragmentation function of ElasticSearch (ES) can be used to fragment data.
  • ElasticSearch (ES) is a distributed search engine.
  • An ES cluster is composed of one or more nodes, and an ES instance is a node.
  • An index in ES is a collection of documents, each index has one or more shards, and the indexed data is allocated and stored on each shard.
  • Sharding can be understood as the smallest unit block of data, and the data written into the database is allocated and stored on the shards.
  • an index can be created in ES, and each index can correspond to multiple shards.
  • Multiple shards may be on the same node or on different nodes. For example, suppose that the index IndexA has 2 shards. Insert 10 pieces of data (10 documents), then the 10 pieces of data can be divided into 5 pieces and stored in the first shard, and the remaining 5 pieces will be stored in another shard. The data corresponding to the index can be stored in its corresponding shard, and when data needs to be queried, the data can be queried from the corresponding shard of the index.
  • the server when the server determines the multiple query handles corresponding to the data query result, the server can determine the segment corresponding to the data in the data query result.
  • the data query request sent by the service end may carry query conditions.
  • the server can extract the features in the query conditions according to the query conditions, and determine the data matching the features as the data query result.
  • the server determines the shard where the data in the data query result is located. It is understandable that the server can store information about each data and the shard where each data is located. Based on this, the server can determine the data query result that meets the query conditions according to the query conditions, and determine the shard where the data is located.
  • the server may determine the index corresponding to the data query result according to the data query request, and obtain multiple shards corresponding to the index. It is understandable that the server can find the index corresponding to the data query result according to the determined data query result, and determine the multiple fragments corresponding to the index according to the information of the index. For example, index 1 and index 2 are included in the data query result, and the IDs of the shards of index 1 and index 2 are determined.
  • Step S230 Determine the query handle of the partial data query result corresponding to each segment, and obtain multiple query handles.
  • the server can determine the query handle of the partial data query result corresponding to each shard, and obtain multiple query handles.
  • the query handle may be information used to read the data query result.
  • the query handle may include storage location, data identification, fields, and so on.
  • the identification information of the partial data query result corresponding to each shard is obtained, and the identification information is used as the query handle to obtain multiple query handles. Understandably, by identifying the message, data can be read from the corresponding shard. The data pulled out from the corresponding shards by each query handle can form a complete data query result.
  • Step S240 Allocate each query handle of the plurality of query handles to a thread, and the thread corresponding to each query handle is different.
  • step S240 can refer to the content of the foregoing embodiment, which will not be repeated here.
  • Step S250 After each thread pulls a preset amount of data according to the corresponding query handle, it sends the pulled data to the service end.
  • each thread can pull the data query result from the database according to the allocated query handle, and while pulling the data, the data Sent to the business end. Specifically, each thread can pull data from the database multiple times. Among them, each thread can pull data from the corresponding shard according to the query handle, that is, each query handle corresponds to a partial data query result in a shard, so each thread is based on the query handle When pulling data, the data is pulled from the allocation corresponding to the query handle. Specifically, the shards to be taken out can be determined according to the preference parameter. If the parameter starts with _shards, it means that only the specified shards are queried. Therefore, each thread can limit the parameters starting with _shards to follow the corresponding The distribution pulls data.
  • each thread can use the scroll cursor function of ElasticSearch according to the corresponding query handle to pull the data query result and send the data query result to the business end at the same time. Understandably, if we want to take out a large amount of data from the ES cluster, taking it out at one time will cause excessive IO pressure and performance problems, and paging queries are likely to cause deep paging problems, so you can Use the scroll query of ES to check batch by batch until all the data is queried.
  • the data to be queried is a user ID
  • each thread uses a query handle allocated by itself to query a certain amount of user IDs, compose a piece of message and send it to Kafka, so as to realize batch sending within each shard query handle.
  • each thread may, according to the corresponding query handle, after pulling a preset amount of data, send the pulled data to the business end. In other words, after each thread pulls a batch of data, it stores the pulled data into the memory, and then sends the pulled data from the memory.
  • the server may also determine the amount of data that needs to be pulled each time before pulling the data, that is, the above-mentioned preset data amount.
  • the server may obtain the current remaining memory value, and determine the preset data amount according to the remaining memory value. It is understandable that the method of pulling and sending data is adopted to pull and send data in batches. The main consideration is that the memory of the server may be occupied by the data pulled, which may cause the problem of operation crash. Therefore, the preset amount of data to be pulled each time can be determined according to the current remaining memory value of the server.
  • the server determines the preset data amount according to the current remaining memory value, which may include: judging whether the remaining memory value is greater than the specified memory value; if the remaining memory value is greater than the specified memory value, then The first data amount is used as the preset data amount; if the remaining memory value is less than the specified memory value, then the second data amount is used as the preset data amount, and the second data amount is less than the first data amount.
  • the amount of data can be used as a criterion for judging whether the current memory usage of the server takes up a lot. If the remaining memory value is greater than the specified memory value, it means that the server currently has more remaining memory.
  • the first amount of data is used as the preset amount of data; if the remaining memory value is less than or equal to the specified memory value, it means that the current remaining memory of the server is small, and its memory is relatively tight, so you can use the second smaller amount of data than the first The amount of data is used as the preset amount of data.
  • the server can determine the memory that can be allocated when the remaining memory value is greater than the specified memory value according to the remaining memory value, and calculate the allocated memory and the threads corresponding to multiple query handles The ratio of the quantity, the ratio is regarded as the first data quantity.
  • the server can also determine the memory that can be allocated when the remaining memory value is less than or equal to the specified memory value according to the remaining memory value, and calculate the ratio of the allocated memory to the number of threads corresponding to multiple query handles, and use this ratio as the first The amount of data.
  • the memory that can be allocated should be greater than the memory that can be allocated if the remaining memory value is less than or equal to the specified memory value. Therefore, the determined first data amount should be greater than the second Big amount of data.
  • the first data volume and the second data volume may also be preset in the server.
  • the first amount of data should satisfy: when the threads corresponding to multiple query handles pull data, the total amount of data stored in the memory is less than the specified memory value; the second amount of data should satisfy: the threads corresponding to multiple query handles are in When pulling data, the total amount of data stored in the memory is less than the remaining memory value.
  • the server determines the preset data amount according to the current remaining memory value, which may include: obtaining the preset data amount corresponding to the remaining memory value according to the correspondence relationship between the preset memory value and the preset data amount .
  • different memory values and the amount of cached data that can be supported can be determined in advance according to experiments, so as to establish the correspondence between the memory value and the preset data amount. It should be noted that the preset data amount determined according to the corresponding relationship can all satisfy the normal operation of the program in the server.
  • the preset data amount is determined by the memory value of the server. On the basis of pulling the data amount in batches to avoid the problem of excessive memory occupation, it further ensures that the server memory will not occupy too much Problem, to ensure the normal operation of the server.
  • the server may also obtain the ratio of the total data amount to the specified pull times according to the total data amount of the data query result and the designated pull times, and use the ratio as the preset data amount. It is understandable that a designated number of pull times may be set in the server, and the designated number of pull times is the total number of times that each thread pulls data from the shards.
  • the setting of the specified pull times can meet the needs of different scenarios, and it can also ensure that the number of data pulls will not be too many, which will lead to an excessively long time for the overall data export.
  • the number of pull times can be set by the user, or can be determined according to the current read data and sending speed.
  • Step S260 Repeat each thread according to the corresponding query handle, after pulling a preset amount of data, send the pulled data to the business end until all the data corresponding to the data query result is sent To the business end.
  • step S250 is repeated until all the data query results are read and sent to the service end, thereby completing the pull and sending of data in batches, and the data can be sent while pulling the data. The effect of this avoids the problem that the pulled data occupies a lot of memory.
  • the data push method by obtaining the data query request sent by the business end, according to the data query request, determine the multiple shards corresponding to the data query result, and then determine the partial data query corresponding to each shard
  • the result of the query handle obtain multiple query handles, and then assign each of the multiple query handles to a thread, where each query handle corresponds to a different thread, and then each thread is divided according to the corresponding query handle
  • Pull data in batches and send it to the business end until all the data in the data query result is sent to the business end, so that it can pull and send data in parallel, improve the speed of data push, and each thread is pulling and sending
  • the method of pulling and sending is adopted, which does not occupy too much memory and ensures the stable operation of the program, thereby greatly improving the data export speed.
  • the export time of 100 million-level data can be achieved. From half an hour to 2 minutes, all data is sent to the business end.
  • FIG. 3 shows a schematic flowchart of a data push method provided by another embodiment of the present application.
  • the data pushing method can be applied to the above-mentioned server. The following will elaborate on the process shown in FIG. 3 in detail.
  • the data pushing method may specifically include the following steps:
  • Step S310 Obtain the data query request sent by the service end.
  • Step S320 Determine whether the total data amount of the data query result is greater than the specified data amount.
  • the server can determine the data query result according to the data query request, and then determine the total data volume of the data query result, that is, the data size of all data in the data query result. The server then compares the total data volume of the data query result with the specified data volume, and according to the comparison result, determines whether the total data volume is greater than the specified data volume.
  • the specified data amount is used as a criterion for determining whether the data amount in the data query result is large. If the total data amount is greater than the specified data amount, it means that the amount of data that needs to be pushed this time is large, so step S340 to step can be executed.
  • step S360 After multiple query handles are determined, multiple threads are used to read data from the partition in batches according to the allocated query handles, and then sent to the business end, so as to pull data in parallel, improve the export speed of data, and send it at the same time Data is sent at the same time to avoid the problem of excessive memory usage. If the total amount of data is less than or equal to the specified amount of data, it means that the amount of data that needs to be pushed this time is not much, so step S330 can be executed, that is, after reading all the data query results from the database, the data is stored in the memory , And then send the data from the memory to the business end.
  • the specific value of the specified data volume is not a limitation. It should be understood that if the total data volume is less than or equal to the specified data volume, all the data of the data query result will be stored in the memory, and there will be no excessive memory usage, which will cause the operation to crash. Case.
  • Step S330 After reading all the data query results from the database, send the data query results to the service end.
  • step S330 is executed.
  • Step S340 Acquire multiple query handles corresponding to the data query result according to the data query request.
  • step S340 to step S360 are executed.
  • Step S350 Allocate each query handle of the multiple query handles to a thread, and the thread corresponding to each query handle is different.
  • Step S360 Each thread pulls the data query result according to the corresponding query handle and sends the data query result to the service end.
  • steps S330 to S360 can refer to the content of the foregoing embodiment, which will not be repeated here.
  • the data push method by obtaining the data query request sent by the business end, according to the query request, it is determined whether the total data volume of the data query result is greater than the specified data volume, and if it is greater than the specified data volume, the corresponding data query result is obtained
  • Each of the multiple query handles is assigned to a thread, where each query handle corresponds to a different thread, and then each thread pulls the data query result according to the corresponding query handle Send the data query results to the business end, so that when the total data volume of the data query results is greater than the specified data volume, it can pull and send data in parallel, improve the speed of data push, and each thread is pulling and sending data
  • the method of pulling and sending is adopted, which can not occupy too much memory and ensure the stable operation of the program.
  • FIG. 4 shows a schematic flowchart of a data push method provided by another embodiment of the present application.
  • This data pushing method can be applied to the above-mentioned server. The following will elaborate on the process shown in FIG. 4.
  • the data pushing method may specifically include the following steps:
  • Step S410 Obtain the data query request sent by the service end.
  • Step S420 Acquire multiple query handles corresponding to the data query result according to the data query request.
  • step S410 and step S420 can refer to the content of the foregoing embodiment, and will not be repeated here.
  • Step S430 Obtain available threads in the thread pool.
  • the server may use the thread pool to allocate threads to the query handle.
  • the thread pool is a form of multi-threaded processing, which can assign tasks to threads, and threads can automatically execute tasks.
  • a thread pool can be created in the server in advance, or a thread pool can be created when data push is required.
  • the server can create a thread pool by using the ThreadPoolExecutor class, and can also create a thread pool by using the Executors class.
  • the specific method of creating the thread pool is not limited.
  • the server may obtain available threads in the thread pool, where the available threads may refer to threads that are in an idle state, that is, threads that have no tasks to be executed.
  • the available threads can be used to allocate query handles to perform the task of pulling data from the partition.
  • Step S440 Allocate each of the plurality of query handles to one thread among the available threads.
  • the server after the server determines the available threads from the thread pool, it can allocate a query handle to the threads in the available threads, where each query handle should be guaranteed to be allocated to an available thread to ensure data query The result can be all pulled out for sending.
  • the server may also determine the number of available threads, compare the number with the number corresponding to multiple query handles, and determine whether the number of available threads is less than the multiple query handles based on the comparison result The corresponding quantity. If the number of available threads is greater than or equal to the number corresponding to multiple query handles, it means that the current available threads can guarantee that each query handle is allocated to a different thread, so each query handle of the multiple query handles can be allocated Give one of the available threads. If the number of available threads is less than the number corresponding to multiple query handles, it means that the current available threads cannot guarantee that each query handle is allocated to a different thread. At this time, it is necessary to ensure that all query handles can be allocated to threads to complete the pulling and sending of all data query results.
  • the number of available threads can be used to create a target number of threads in the thread pool, where the sum of the target number and the number of available threads is greater than or equal to The number corresponding to multiple query handles.
  • the server may request the allocation of available resources of the processor to create a target number of new threads in the thread pool, and the sum of the target number and the number of available threads is greater than or equal to the number corresponding to multiple query handles, thereby creating After threading, the number of available threads can ensure that each query handle is allocated to a different thread.
  • the server may also obtain the load rate of the processor before creating the thread in the thread pool, and then compare the load rate with the specified load rate, and determine whether the load rate is less than the specified load rate according to the comparison result.
  • the specified load rate is used to determine whether the load of the processor is high, if the load rate is less than the specified load rate, it means that the load of the processor is not high, so only the target number of threads can be created in the thread pool. If the load rate is greater than or equal to the specified load rate, it means that the load on the processor is high and it is not suitable for creating new threads in the thread pool.
  • the server may assign at least two query handles to the same thread respectively to ensure that all query handles are allocated to threads, so that the data corresponding to the data query result can be All are pulled and sent.
  • the server may determine the data volume of the partial data query result corresponding to each query handle among the multiple query handles, and then divide the at least two query handles respectively according to the data volume To the same thread, and make the size of the amount of data pulled and sent by each thread is basically the same.
  • query handle 1 there are 8 query handles, namely query handle 1, query handle 2, query handle 3, query handle 4, query handle 5, query handle 6, query handle 7, query handle 8, and the amount of data corresponding to query handle 1 is 100 megabytes (Mb), the data volume corresponding to query handle 2 is 200Mb, the data volume corresponding to query handle 3 is 400Mb, the data volume corresponding to query handle 4 is 300Mb, the data volume corresponding to query handle 5 is 200Mb, and the data volume corresponding to query handle 6 is 200Mb.
  • the data volume of query handle 7 is 150Mb
  • the data volume corresponding to query handle 7 is 150Mb
  • the data volume corresponding to query handle 8 is 110Mb.
  • Query handle 1 and query handle 4 can be allocated to the same thread, query handle 2 and query handle 5 can be allocated to the same thread, query handle 4 can be allocated to the same thread, query handle 6, query handle 7 and query Handle 8 is allocated to the same thread, so that the amount of data that each thread needs to pull is about 400Mb.
  • the server can also determine the priority of the task executed by the currently non-idle thread, and can assign the thread corresponding to the task with a low priority to the query handle to For data pulling and sending, after completing this data push, these threads then perform the previous tasks.
  • the priority can be determined according to the real-time performance of the task, for example, the real-time performance can be determined according to the data type.
  • Step S450 Each thread pulls the data query result according to the corresponding query handle and sends the data query result to the service end.
  • step S450 can refer to the content of the foregoing embodiment, and will not be repeated here.
  • the data push method by obtaining the data query request sent by the business end, according to the query request, multiple query handles corresponding to the data query result are obtained, and then the available threads in the thread pool are obtained, and then multiple queries are obtained.
  • Each query handle in the handle is assigned to one of the available threads, and it is ensured that the thread corresponding to each query handle is different, and then each thread pulls the data query result according to the corresponding query handle and sends the data query result to On the business side, data can be pulled and sent in parallel, and the speed of data push can be improved.
  • the method of pulling and sending is adopted, which may not occupy too much memory and ensure the stable operation of the program.
  • FIG. 5 shows a structural block diagram of a data pushing device 400 provided by an embodiment of the present application.
  • the data pushing device 400 uses the aforementioned server, and the data pushing device 400 includes: a request acquiring module 410, a handle acquiring module 420, a handle allocating module 430, and a data sending module 440.
  • the request obtaining module 410 is used to obtain the data query request sent by the service end; the handle obtaining module 420 is used to obtain multiple query handles corresponding to the data query result according to the data query request; the handle allocating module 430 is used For assigning each query handle of the plurality of query handles to a thread, the thread corresponding to each query handle is different; the data sending module 440 is used for each thread to pull data query results according to the corresponding query handle At the same time, the data query result is sent to the service end.
  • the handle obtaining module 420 may include: an allocation determining unit and a handle determining unit.
  • the shard determining unit is used to determine multiple shards corresponding to the data query result according to the data query request; the handle determining unit is used to determine the query handle of the partial data query result corresponding to each shard to obtain multiple queries Handle.
  • the shard determining unit may be specifically configured to: according to the data query request, obtain the shard where the data query result that satisfies the query condition is located.
  • the shard determining unit may be specifically configured to: determine an index corresponding to the data query result according to the data query request; and obtain multiple shards corresponding to the index.
  • the data pushing device 400 may further include: a data slicing module.
  • the data fragmentation module can be used to fragment the data stored in the database.
  • the handle determining unit may be specifically configured to: obtain identification information of a partial data query result corresponding to each fragment, and use the identification information as a query handle to obtain multiple query handles.
  • the data sending module 440 may be specifically configured to: after each thread pulls a preset amount of data according to the corresponding query handle, send the pulled data to the service end; repeat all According to the corresponding query handle, each thread pulls a preset amount of data, and then sends the pulled data to the business end until all the data corresponding to the data query result is sent to the business end .
  • the data sending module 440 may also be used to: obtain the current remaining memory value; and determine the preset data amount according to the remaining memory value.
  • the data sending module 440 determining the preset data amount according to the remaining memory value may include: determining whether the remaining memory value is greater than a specified memory value; if the remaining memory value is greater than the specified memory value, Value, the first data amount is used as the preset data amount; if the remaining memory value is less than the designated memory value, the second data amount is used as the preset data amount, and the second data amount is less than The first amount of data.
  • the data sending module 440 may also be used to obtain the ratio of the total data volume to the specified pull times according to the total data volume of the data query result and the designated pull times, and compare the total data volume to the designated pull times. The ratio is used as the preset data amount.
  • the data sending module 440 may be specifically configured to: each thread uses the scroll cursor function of ElasticSearch according to the corresponding query handle to pull the data query result while sending the data query result to the service end. .
  • the data pushing device 400 may further include a data amount judgment module.
  • the data amount judging module is used for judging whether the total data amount of the data query result is greater than the specified data amount before obtaining the multiple query handles corresponding to the data query result according to the data query request. If the total data amount is greater than the specified data amount, the handle obtaining module 420 obtains multiple query handles corresponding to the data query result according to the data query request.
  • the data sending module 440 may also be configured to send the data query results to the data query result after reading all the data query results from the database if the total data volume is less than or equal to the specified data volume. Business side.
  • the handle allocation module may include: a thread acquisition unit and an allocation unit.
  • the thread obtaining unit is used to obtain available threads in the thread pool; the allocation unit is used to allocate each of the plurality of query handles to one thread of the available threads.
  • the handle allocation module may further include: a thread judgment unit.
  • the thread judging unit is used to judge whether the number of available threads is less than the number corresponding to the multiple query handles. If the number of available threads is greater than or equal to the number corresponding to the plurality of query handles, the allocation unit allocates each query handle of the plurality of query handles to one thread of the available threads.
  • the handle allocation module may also include a thread creation unit.
  • the thread creation unit is configured to create a target number of threads in the thread pool if the number of available threads is less than the number corresponding to the multiple query handles, wherein the sum of the target number and the number of available threads The value is greater than or equal to the number corresponding to multiple query handles.
  • the thread creation unit may be specifically used to: obtain the load rate of the processor; if the load rate is less than a specified load rate, create a target number of threads in the thread pool.
  • the coupling between the modules may be electrical, mechanical or other forms of coupling.
  • the functional modules in the various embodiments of the present application may be integrated into one processing module, or each module may exist alone physically, or two or more modules may be integrated into one module.
  • the above-mentioned integrated modules can be implemented in the form of hardware or software functional modules.
  • the solution provided by this application obtains the data query request sent by the business end, according to the query request, obtains multiple query handles corresponding to the data query result, and assigns each query handle of the multiple query handles to A thread, where each query handle corresponds to a different thread, and then each thread pulls the data query result according to the corresponding query handle and sends the data query result to the business end, so that parallel data can be pulled and sent , Improve the speed of data push.
  • the server 100 may be a server capable of running application programs, such as a traditional server or a cloud server.
  • the server 100 in this application may include one or more of the following components: a processor 110, a memory 120, a touch screen 130, and one or more application programs, where one or more application programs may be stored in the memory 120 and configured as Executed by one or more processors 110, one or more programs are configured to execute the methods described in the foregoing method embodiments.
  • the processor 110 may include one or more processing cores.
  • the processor 110 uses various interfaces and lines to connect various parts of the entire server 100, and executes the server by running or executing instructions, programs, code sets, or instruction sets stored in the memory 120, and calling data stored in the memory 120. 100's various functions and processing data.
  • the processor 110 may adopt at least one of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA).
  • DSP Digital Signal Processing
  • FPGA Field-Programmable Gate Array
  • PDA Programmable Logic Array
  • the processor 110 may be integrated with one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), a modem, and the like.
  • the CPU mainly processes the operating system, user interface, and application programs; the GPU is used for rendering and drawing of display content; the modem is used for processing wireless communication. It can be understood that the above-mentioned modem may not be integrated into the processor 110, but may be implemented by a communication chip alone.
  • the memory 120 may include random access memory (RAM) or read-only memory (Read-Only Memory).
  • the memory 120 may be used to store instructions, programs, codes, code sets or instruction sets.
  • the memory 120 may include a program storage area and a data storage area, where the program storage area may store instructions for implementing the operating system and instructions for implementing at least one function (such as touch function, sound playback function, image playback function, etc.) , Instructions for implementing the following method embodiments, etc.
  • the storage data area can also store data (such as phone book, audio and video data, chat record data) created by the server 100 during use.
  • FIG. 7 shows a structural block diagram of a computer-readable storage medium provided by an embodiment of the present application.
  • the computer-readable medium 800 stores program code, and the program code can be invoked by a processor to execute the method described in the foregoing method embodiment.
  • the computer-readable storage medium 800 may be an electronic memory such as flash memory, EEPROM (Electrically Erasable Programmable Read Only Memory), EPROM, hard disk, or ROM.
  • the computer-readable storage medium 800 includes a non-transitory computer-readable storage medium.
  • the computer-readable storage medium 800 has storage space for the program code 810 for executing any method steps in the above-mentioned methods. These program codes can be read from or written into one or more computer program products.
  • the program code 810 may be compressed in a suitable form, for example.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

本申请公开了一种数据推送方法、装置、服务器及存储介质，该数据推送方法包括：获取业务端发送的数据查询请求；根据所述数据查询请求，获取数据查询结果对应的多个查询句柄；将所述多个查询句柄中的每个查询句柄分配给一个线程，每个查询句柄对应的线程不同；每个线程根据对应的查询句柄，拉取数据查询结果的同时将所述数据查询结果发送至所述业务端。本方法可以实现快速的推送查询结果至业务端。

Description

数据推送方法、装置、服务器及存储介质 技术领域
本申请涉及数据处理技术领域，更具体地，涉及一种数据推送方法、装置、服务器及存储介质。
背景技术
随着网络信息技术的快速发展，基于大数据技术的信息推荐技术也应运而生。信息推荐技术中主要依据用户的标签数据而进行相关信息的推荐，在信息推荐中通常会存在业务方对标签数据进行查询，而从数据库中拉取和发送数据时需要花费较多时间。
发明内容
鉴于上述问题，本申请提出了一种数据推送方法、装置、服务器及存储介质。
第一方面，本申请实施例提供了一种数据推送方法，所述方法包括：获取业务端发送的数据查询请求；根据所述数据查询请求，获取数据查询结果对应的多个查询句柄；将所述多个查询句柄中的每个查询句柄分配给一个线程，每个查询句柄对应的线程不同；每个线程根据对应的查询句柄，拉取数据查询结果的同时将所述数据查询结果发送至所述业务端。
第二方面，本申请实施例提供了一种数据推送装置，所述装置包括：请求获取模块、句柄获取模块、句柄分配模块以及数据发送模块，其中，所述请求获取模块用于获取业务端发送的数据查询请求；所述句柄获取模块用于根据所述数据查询请求，获取数据查询结果对应的多个查询句柄；所述句柄分配模块用于将所述多个查询句柄中的每个查询句柄分配给一个线程，每个查询句柄对应的线程不同；所述数据发送模块用于每个线程根据对应的查询句柄，拉取数据查询结果的同时将所述数据查询结果发送至所述业务端。
第三方面，本申请实施例提供了一种服务器，包括：一个或多个处理器；存储器；一个或多个应用程序，其中所述一个或多个应用程序被存储在所述存储器中并被配置为由所述一个或多个处理器执行，所述一个或多个程序配置用于执行上述第一方面提供的数据推送方法。
第四方面，本申请实施例提供了一种计算机可读取存储介质，所述计算机可读取存储介质中存储有程序代码，所述程序代码可被处理器调用执行上述第一方面提供的数据推送方法。
本申请提供的方案，通过获取业务端发送的数据查询请求，根据该查询请求，获取数据查询结果对应的多个查询句柄，将多个查询句柄中的每 个查询句柄分配给一个线程，其中，每个查询句柄对应的线程不同，然后每个线程根据对应的查询句柄，拉取数据查询结果的同时将数据查询结果发送至业务端，从而可以实现并行的拉取和发送数据，提升数据推送的速度。
附图说明
为了更清楚地说明本申请实施例中的技术方案，下面将对实施例描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本申请的一些实施例，对于本领域技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。
图1示出了根据本申请一个实施例的数据推送方法流程图。
图2示出了根据本申请另一个实施例的数据推送方法流程图。
图3示出了根据本申请又一个实施例的数据推送方法流程图。
图4示出了根据本申请再一个实施例的数据推送方法流程图。
图5示出了根据本申请一个实施例的数据推送装置的一种框图。
图6是本申请实施例的用于执行根据本申请实施例的数据推送方法的服务器的框图。
图7是本申请实施例的用于保存或者携带实现根据本申请实施例的数据推送方法的程序代码的存储单元。
具体实施方式
为了使本技术领域的人员更好地理解本申请方案，下面将结合本申请实施例中的附图，对本申请实施例中的技术方案进行清楚、完整地描述。
在信息推送中，通常存在业务方对标签数据进行查询的情况，数据端根据业务方的数据需求，从数据库中拉取数据并发送至业务方。从而业务方可以根据获得的数据，进行推送内容的确定、推送对象的确定等。
在传统的数据拉取和发送的方案中，通常会将所有数据从数据库中读取到内存，再从内存发送到业务方。这样的方案简单明了，实现了数据的导出。
发明人经过长期研究发现，在传统的数据导出的方案中，随着业务方数量和需求的持续增长，业务方需求的数据越来越多，数据端将海量的查询结果读取到内存，不仅非常耗时，而且容易触发内存溢出。例如：某业务方需要导出同时安装了应用A和应用B的用户，这样的条件是非常容易圈取到2亿多的用户的，这样的亿级数量级的用户信息的导出，目前的技术方案发送速度很慢且有宕机风险。
针对上述问题，发明人提出了本申请实施例提供的数据推送方法、装置、服务器以及存储介质，通过将数据查询结果对应的多个查询句柄，分片至不同线程，每个线程根据分片的查询句柄，并行的拉取数据的同时将数据发送至业务端。其中，具体的数据推送方法在后续的实施例中进行详细的说明。
请参阅图1，图1示出了本申请一个实施例提供的数据推送方法的流程示意图。在具体的实施例中，所述数据推送方法应用于如图5所示的数据推送装置400以及配置有所述数据推送装置400的服务器100(图6)。下面将以服务器为例，说明本实施例的具体流程，当然，可以理解的，本实施例所应用的服务器可以为传统服务器、云服务器等，在此不做限定。下面将针对图1所示的流程进行详细的阐述，所述数据推送方法具体可以包括以下步骤：
步骤S110：获取业务端发送的数据查询请求。
在本申请实施例中，业务端在需要进行数据的查询时，可以向服务器发送数据查询请求。对应的，服务器可以接收业务端发送数据查询请求，并且对数据查询请求进行响应，将相应的数据返回至业务端。
在一些实施方式中，数据查询请求中可以携带有业务端对应的业务方的数据需求，例如可以携带有查询条件。查询条件中可以包括一个或多个条件。例如，业务方想对同时安装了应用1和应用2，但未安装应用3的用户进行广告投放，设置的查询条件为同时安装了应用1和应用2，但未安装应用3的用户；又例如，游戏公司向提升近期的比较活跃的游戏玩家的留存率，想要对近期比较活跃的游戏玩家投放游戏奖励，设置的查询条件为最近一周内游戏时长超过24小时或者登陆次数超过10次的用户。
在一些实施方式中，业务端发送的数据查询请求可以是用于获取标签数据的请求，也可以是用于获取其他类型的数据的请求，例如，获取文档数据等。数据查询请求用于获取的数据的类型可以不作为限定。
步骤S120：根据所述数据查询请求，获取数据查询结果对应的多个查询句柄。
在本申请实施例中，服务器在获取到数据查询请求之后，可以根据数据查询请求，确定数据查询结果。其中，数据查询结果可以包括数据查询请求对应的数据。也就是说，服务器可以根据数据查询请求中携带的需求信息，确定哪些数据满足该需求信息。例如，数据查询请求为用于查询标签数据的请求，且数据查询请求中携带有安装有应用A和应用B的用户，则服务器可以确定出安装有应用A和应用B的用户有哪些。又例如，数据查询请求为用于查询文档数据的请求，且数据查询请求中携带有包括关键词A和B，则服务器可以确定出哪些文档包括关键词A和B。
在本申请实施例中，服务器在根据数据查询请求确定出数据查询结果之后，可以根据数据查询结果，确定数据查询结果所对应的查询句柄。其中，查询句柄可以为用于读取数据查询结果的信息。查询句柄可以包括存储位置、数据的标识、字段等。根据查询句柄，可以读取到相应的数据查询结果。服务器中可以预先存储有各个数据对应的查询句柄，服务器可以根据数据查询结果中包括的数据，确定数据查询结果对应的查询句柄。
在一些实施方式中，服务器可以根据数据查询结果，将数据查询结果分为多部分的数据查询结果，并确定各个部分的数据查询结果所对应的查 询句柄，从而可以获得数据查询结果对应的多个查询句柄。可以理解的，获得的多个查询句柄，可以分别用于读取数据，根据多个查询句柄读取的数据，可以构成完整的数据查询结果。
作为一种实施方式，服务器可以根据数据查询结果中数据分布的位置，而将数据查询结果分为多个部分，并确定各个部分所对应的查询句柄。例如，数据查询结果对应的数据存储于数据库集群，因此可以确定出存储有数据查询结果的多个数据库，并根据多个数据库中包括的数据查询结果，确定查询句柄。又例如，数据查询结果对应的数据存储于数据库中的不同分片中，因此可以确定出数据查询结果所在的多个分片，并根据多个分片中存在的数据查询结果，确定查询句柄。
作为另一种实施方式，服务器也可以根据数据查询结果中不同的类型的数据，对数据查询结果进行划分，并根据划分的结果，确定各个部分的数据查询结果所对应的查询句柄。
当然，具体获取数据查询结果对应的多个查询句柄的方式可以不作为限定。
步骤S130：将所述多个查询句柄中的每个查询句柄分配给一个线程，每个查询句柄对应的线程不同。
在本申请实施例中，服务器在确定出数据查询结果对应的多个查询句柄之后，可以针对每个查询句柄，分配一个不同的线程，以便从数据库中读取数据后发送至业务端。
在一些实施方式中，服务器中可以创建有线程池，服务器可以从线程池中为每个查询句柄分配一个不同的线程，每个线程用于单独的从数据库读取其分配的查询句柄所对应的数据查询结果的数据。
步骤S140：每个线程根据对应的查询句柄，拉取数据查询结果的同时将所述数据查询结果发送至所述业务端。
在本申请实施例中，服务器在对每个查询句柄分配一个线程之后，每个线程可以根据其分配的查询句柄，从数据库中拉取数据查询结果，并且在边拉取数据的同时，将数据发送至业务端，避免一次性拉取完数据之后再进行发送，导致占用的内存过多，而引起内存溢出。例如，业务端为kafka时，每个线程拉取数据之后，将数据发送至kafka。
本申请实施例提供的数据推送方法，通过获取业务端发送的数据查询请求，根据该查询请求，获取数据查询结果对应的多个查询句柄，将多个查询句柄中的每个查询句柄分配给一个线程，其中，每个查询句柄对应的线程不同，然后每个线程根据对应的查询句柄，拉取数据查询结果的同时将数据查询结果发送至业务端，从而可以实现并行的拉取和发送数据，提升数据推送的速度。并且，各个线程在拉取和发送数据时，采用边拉取边发送的方式，可以不占用过多的内存，保证程序的稳定运行。
请参阅图2，图2示出了本申请另一个实施例提供的数据推送方法的流程示意图。该数据推送方法可应用于上述服务器，下面将针对图2所示 的流程进行详细的阐述，所述数据推送方法具体可以包括以下步骤：
步骤S210：获取业务端发送的数据查询请求。
在本申请实施例中，步骤S210可以参阅前述实施例的内容，在此不再赘述。
步骤S220：根据所述数据查询请求，确定数据查询结果所对应的多个分片。
在本申请实施例中，在存储数据到数据库时，可以对数据进行分片。具体地，可以利用ElasticSearch(ES)的分片功能进行数据的分片。其中，ElasticSearch(ES)是一种分布式的搜索引擎，一个ES集群由一个或多个节点组成，一个ES实例为一个节点。ES中的一个索引为文档的集合，每个索引有一个或多个分片，索引的数据被分配存放到各个分片上。分片可以理解为数据的最小单元块，写入数据库中的数据被分配存在分片上。具体的，ES中可以创建索引，每个索引可以对应有多个分片，多个分片可能在相同节点，也可能在不同节点，例如，假设索引IndexA有2个分片，在向IndexA中插入10条数据(10个文档)，那么这10条数据中可以分为5条存储在第一个分片，剩下的5条会存储在另一个分片中。索引对应的数据可以存储于其对应的分片中，当需要查询数据时，可以从索引对应的分片中查询数据。
在本申请实施例中，服务器在确定数据查询结果对应的多个查询句柄时，可以确定数据查询结果中数据所对应的分片。在一些实施方式中，业务端发送的数据查询请求中可以携带有查询条件。服务器可以根据查询条件，提取查询条件中的特征，并且根据确定与特征匹配的数据，作为数据查询结果。在确定出数据查询结果后，服务器确定数据查询结果中数据所在的分片。可以理解的，服务器中可以存储有各个数据的信息，以及各个数据所在的分片，基于此，服务器可以根据查询条件确定出满足查询条件的数据查询结果，并确定出数据所在的分片。
在一些实施方式中，服务器可以根据数据查询请求，确定数据查询结果对应的索引，并获取索引对应的多个分片。可以理解的，服务器可以根据确定出的数据查询结果，查找数据查询结果所对应的索引，并根据索引的信息，确定索引对应的多个分片。例如，数据查询结果中包括索引1以及索引2，并确定出索引1以及索引2的分片的ID。
步骤S230：确定每个分片所对应的部分数据查询结果的查询句柄，获得多个查询句柄。
在本申请实施例中，服务器在确定出数据查询结果所对应的多个分片之后，可以确定每个分片所对应的部分数据查询结果的查询句柄，获得多个查询句柄。其中，查询句柄可以为用于读取数据查询结果的信息。查询句柄可以包括存储位置、数据的标识、字段等。
在一些实施方式中，获取每个分片所对应的部分数据查询结果的标识信息，将标识信息作为查询句柄，获得多个查询句柄。可以理解的，通过 标识消息，可以从相应的分片中读取数据。利用各个查询句柄从相应的分片拉取出的数据，可以构成完整的数据查询结果。
步骤S240：将所述多个查询句柄中的每个查询句柄分配给一个线程，每个查询句柄对应的线程不同。
在本申请实施例中，步骤S240可以参阅前述实施例的内容，在此不再赘述。
步骤S250：每个线程根据对应的查询句柄，在拉取预设数据量的数据后，将拉取到的数据发送至所述业务端。
在本申请实施例中，服务器在对每个查询句柄分配一个线程之后，每个线程可以根据其分配的查询句柄，从数据库中拉取数据查询结果，并且在边拉取数据的同时，将数据发送至业务端。具体地，每个线程可以分多次从数据库中拉取数据。其中，每个线程可以根据查询句柄，从相应的分片中拉取数据，也就是说，每个查询句柄都与一个分片中的部分数据查询结果相对应，因此每个线程在根据查询句柄拉取数据时，是从查询句柄对应的分配中拉取数据。具体地，可以根据preference参数来决定取出的分片的，如果参数以_shards开头，则表示只查询指定的分片，因此每个线程可以通过对以_shards开头的参数进行限定，以从相应的分配拉取数据。
在一些实施方式中，每个线程根据对应的查询句柄，可以通过ElasticSearch的scroll游标功能，来实现拉取数据查询结果的同时将数据查询结果发送至业务端。可以理解的，如果我们要把大批量的数据从ES集群中取出，一次性取出会导致IO压力过大，性能容易出问题，而分页查询又容易造成深度分页(deep paging)的问题，因此可以使用ES的scroll查询，一批一批的查，直到所有数据都查询完。例如，查询的数据为用户身份标识，每个线程用自己分配到的查询句柄查询到一定量的用户身份标识，组成一片消息发送到Kafka，从而实现每个分片查询句柄内的分批发送。
在一些实施方式中，每个线程可以根据对应的查询句柄，在拉取预设数据量的数据后，将拉取到的数据发送至业务端。也就是说，每个线程在拉取到一批数据之后，将拉取到的数据存入内存，然后从再从内存中将拉取到的数据进行发送。
在该实施方式下，服务器还可以在进行拉取数据之前，确定每次需要拉取的数据量，也就是上述预设数据量。
作为一种实施方式，服务器可以获取当前的剩余内存值，根据剩余内存值，确定预设数据量。可以理解的，采用边拉取边发送的方式，分批次拉取和发送数据，主要考虑到服务器的内存可能被拉取的数据占用较多，而导致运行崩溃的问题。因此，可以根据服务器当前剩余的内存值，确定每次拉取的预设数据量。
在该实施方式中，服务器根据当前的剩余内存值，确定预设数据量，可以包括：判断所述剩余内存值是否大于指定内存值；如果所述剩余内存值大于所述指定内存值，则将第一数据量作为所述预设数据量；如果所述 剩余内存值小于所述指定内存值，则将第二数据量作为所述预设数据量，所述第二数据量小于所述第一数据量。其中，指定内存值可以作为判断服务器当前内存使用情况是否占用较多的标准，如果剩余内存值大于指定内存值，则表示服务器当前的剩余内存较多，因此，可以选择相对第二数据量较大的第一数据量作为预设数据量；如果剩余内存值小于或等于指定内存值，则表示服务器当前的剩余内存较少，其内存比较紧张，因此可以使用相对第一数据量较小的第二数据量来作为预设数据量。
进一步地，以上第一数据量以及第二数据量均小于指定内存值。在确定第一数据量以及第二数据量时，服务器可以根据剩余内存值，确定剩余内存值大于指定内存值的情况下可分配的内存，并计算可分配的内存与多个查询句柄对应的线程数量的比值，将该比值作为第一数据量。服务器还可以根据剩余内存值，确定剩余内存值小于或等于指定内存值的情况下可分配的内存，并计算可分配的内存与多个查询句柄对应的线程数量的比值，将该比值作为第一数据量。可以理解的，剩余内存值在大于指定内存值的情况下可分配的内存，应当大于剩余内存值小于或等于指定内存值的情况下可分配的内存，因此确定的第一数据量应当比第二数据量大。
当然，第一数据量以及第二数据量也可以为预先设置于服务器中。另外，第一数据量应当满足：多个查询句柄对应的线程在拉取数据时，存入内存的总的数据量小于指定内存值；第二数据量应当满足：多个查询句柄对应的线程在拉取数据时，存入内存的总的数据量小于剩余内存值。
在该实施方式中，服务器根据当前的剩余内存值，确定预设数据量，可以包括：根据预先设定的内存值与预设数据量的对应关系，获取剩余内存值所对应的预设数据量。其中，可以预先根据实验来确定不同内存值，所能支持的缓存数据量，以建立内存值与预设数据量的对应关系。需要说明的是，根据该对应关系确定出的预设数据量，均能满足服务器中程序的正常运行。
在该实施方式中，通过服务器的内存值来确定预设数据量，在分批次拉取数据量，避免内存占用过多的问题的基础上，进一步的保证了服务器内存不会占用过多的问题，保证了服务器的正常运行。
在一些实施方式中，服务器也可以根据数据查询结果的总数据量，以及指定拉取次数，获取总数据量与指定拉取次数的比值，将比值作为预设数据量。可以理解的，服务器中可以设置有指定拉取次数，该指定拉取次数为每个线程从分片中拉取数据的总次数。指定拉取次数的设置可以满足不同场景的需求，也能保证数据拉取次数不会过多，而导致整体导出数据的时间过长。拉取次数可以为用户设置，也可以根据当前读取数据和发送的速度来确定。
步骤S260：重复所述每个线程根据对应的查询句柄，在拉取预设数据量的数据后，将拉取到的数据发送至所述业务端，直至所述数据查询结果对应的数据全部发送至所述业务端。
在本申请实施例中，重复步骤S250，直至数据查询结果中的全部读取完毕，并发送至业务端，从而完成了分批次的拉取和发送数据，能够实现边拉取数据边发送数据的效果，避免了拉取的数据对内存占用较多的问题。
本申请实施例提供的数据推送方法，通过获取业务端发送的数据查询请求，根据该数据查询请求，确定数据查询结果所对应的多个分片，然后确定每个分片所对应的部分数据查询结果的查询句柄，获得多个查询句柄，再将多个查询句柄中的每个查询句柄分配给一个线程，其中，每个查询句柄对应的线程不同，然后每个线程根据对应的查询句柄，分批次的拉取数据并发送至业务端，直至数据查询结果中所有数据发送至业务端，从而可以实现并行的拉取和发送数据，提升数据推送的速度，并且，各个线程在拉取和发送数据时，采用边拉取边发送的方式，可以不占用过多的内存，保证程序的稳定运行，进而总体大幅的提升了数据的导出速度，实际应用中，亿级数据的导出时长可以做到从半小时缩短到2分钟，将数据全部发送到业务端。
请参阅图3，图3示出了本申请又一个实施例提供的数据推送方法的流程示意图。该数据推送方法可应用于上述服务器，下面将针对图3所示的流程进行详细的阐述，所述数据推送方法具体可以包括以下步骤：
步骤S310：获取业务端发送的数据查询请求。
步骤S320：判断数据查询结果的总数据量是否大于指定数据量。
在本申请实施例中，服务器在获取到数据查询请求之后，可以根据数据查询请求，确定数据查询结果，然后确定数据查询结果的总数据量，也就是数据查询结果中所有数据的数据大小。服务器再将数据查询结果的总数据量与指定数据量进行比较，并根据比较结果，确定总数据量是否大于指定数据量。其中，指定数据量用于作为确定数据查询结果中数据量是否较大的标准，如果总数据量大于指定数据量，则表示此次需要进行推送的数据量较大，因此可以执行步骤S340至步骤S360，即确定多个查询句柄之后，利用多个线程根据分配的查询句柄从分区中分批读取数据，然后发送至业务端，实现并行拉取数据，提升数据的导出速度，并在边发送数据的同时边发送数据，避免内存占用过多的问题。如果总数据量小于或等于指定数据量，则表示此次需要进行推送的数据量不多，因此可以执行步骤S330，也就是将从数据库中读取全部的数据查询结果之后，将数据存入内存，再从内存中将数据发送至业务端。其中，指定数据量的具体数值不作为限定，应当理解的是，总数据量小于或等于指定数据量，将数据查询结果的全部数据存入内存，不会出现内存占用过多，而导致运行崩溃的情况。
步骤S330：从数据库中读取全部的数据查询结果之后，将数据查询结果发送至所述业务端。
如果总数据量小于或等于指定数据量，执行步骤S330。
步骤S340：根据所述数据查询请求，获取数据查询结果对应的多个查询句柄。
如果总数据量小于或等于指定数据量，执行步骤S340至步骤S360。
步骤S350：将所述多个查询句柄中的每个查询句柄分配给一个线程，每个查询句柄对应的线程不同。
步骤S360：每个线程根据对应的查询句柄，拉取数据查询结果的同时将所述数据查询结果发送至所述业务端。
在本申请实施例中，步骤S330至步骤S360可以参阅前述实施例的内容，在此不再赘述。
本申请实施例提供的数据推送方法，通过获取业务端发送的数据查询请求，根据该查询请求，判读数据查询结果的总数据量是否大于指定数据量，如果大于指定数据量，获取数据查询结果对应的多个查询句柄，将多个查询句柄中的每个查询句柄分配给一个线程，其中，每个查询句柄对应的线程不同，然后每个线程根据对应的查询句柄，拉取数据查询结果的同时将数据查询结果发送至业务端，从而在数据查询结果的总数据量大于指定数据量时，可以实现并行的拉取和发送数据，提升数据推送的速度，并且，各个线程在拉取和发送数据时，采用边拉取边发送的方式，可以不占用过多的内存，保证程序的稳定运行。
请参阅图4，图4示出了本申请再一个实施例提供的数据推送方法的流程示意图。该数据推送方法可应用于上述服务器，下面将针对图4所示的流程进行详细的阐述，所述数据推送方法具体可以包括以下步骤：
步骤S410：获取业务端发送的数据查询请求。
步骤S420：根据所述数据查询请求，获取数据查询结果对应的多个查询句柄。
在本申请实施例中，步骤S410以及步骤S420可以参阅前述实施例的内容，在此不再赘述。
步骤S430：获取线程池中的可用线程。
在本申请实施例中，服务器可以利用线程池，来对查询句柄分配线程。其中，线程池是一种多线程处理形式，可以将任务分配给线程，线程可以自动对任务进行执行。服务器中可以预先创建有线程池，或者在需要进行数据推送时创建线程池。例如，服务器可以通过使用ThreadPoolExecutor类创建线程池，也可以通过使用Executors类创建线程池，具体创建线程池的方式可以不作为限定。
在一些实施方式中，服务器可以获取线程池中的可用线程，其中，可用线程可以指处于空闲状态的线程，也就是未有待执行任务的线程。可用线程可以用于分配查询句柄，以执行从分区拉取数据的任务。
步骤S440：将所述多个查询句柄中的每个查询句柄分配给所述可用线程中的一个线程。
在本申请实施例中，服务器在从线程池中确定出可用线程之后，则可以对可用线程中的线程分配一个查询句柄，其中，每个查询句柄应当保证分配到一个可用线程，以保证数据查询结果能够全部被拉取出来进行发送。
在一些实施方式中，服务器在确定出可用线程之后，还可以确定可用线程的数量，将该数量与多个查询句柄对应的数量进行比较，根据比较结果确定可用线程的数量是否小于多个查询句柄对应的数量。如果可用线程的数量大于或等于多个查询句柄对应的数量，则表示当前的可用线程可以保证每个查询句柄被分配到一个不同的线程，因此可以将多个查询句柄中的每个查询句柄分配给可用线程中的一个线程。如果可用线程的数量小于多个查询句柄对应的数量，则表示当前的可用线程不能保证每个查询句柄被分配到一个不同的线程。此时，需要保证所有的查询句柄能够被分配给线程，以完成全部数据查询结果的拉取和发送。
作为一种实施方式，如果可用线程的数量小于多个查询句柄对应的数量时，可用于线程池中创建目标数量的线程，其中，该目标数量与所述可用线程的数量的和值大于或等于多个查询句柄对应的数量。其中，服务器可以请求分配处理器的可用资源，以创建目标数量的新线程到线程池中，并且目标数量与所述可用线程的数量的和值大于或等于多个查询句柄对应的数量，从而创建线程后，可用线程的数量能够保证每个查询句柄被分配到一个不同的线程。
在该实施方式中，服务器在创建线程到线程池中之前，还可以获取处理器的负载率，然后将负载率与指定负载率进行比较，根据比较结果确定负载率是否小于指定负载率。其中，指定负载率用于确定处理器的负载是否较高，如果如果负载率小于指定负载率，表示处理器的负载不高，因此可以只需于线程池中创建目标数量的线程。如果如果负载率大于或等于指定负载率，则表示处理器的负载较高，不适于创建新的线程到线程池中。
在一些实施方式中，如果负载率大于或等于指定负载率，服务器可以采用将至少两个查询句柄分别给同一线程的方式，保证所有查询句柄均被分配到线程，使数据查询结果对应的数据能被全部拉取和发送。
作为一种实施方式，如果负载率大于或等于指定负载率，服务器可以确定多个查询句柄中每个查询句柄对应的部分数据查询结果的数据量，然后根据数据量来将至少两个查询句柄分别给同一线程，并且使得每个线程拉取和发送的数据量的大小基本保持一致。
例如，有8个查询句柄，分别为查询句柄1、查询句柄2、查询句柄3、查询句柄4、查询句柄5、查询句柄6、查询句柄7、查询句柄8，查询句柄1对应的数据量为100兆(Mb)，查询句柄2对应的数据量为200Mb，查询句柄3对应的数据量为400Mb，查询句柄4对应的数据量为300Mb，查询句柄5对应的数据量为200Mb，查询句柄6对应的数据量为150Mb，查询句柄7对应的数据量为150Mb，查询句柄8对应的数据量为110Mb。则可以将查询句柄1以及查询句柄4分配至同一个线程，将查询句柄2和查询句柄5分配至同一个线程，将查询句柄4分配至同一个线程，将查询句柄6、查询句柄7和查询句柄8分配至同一个线程，从而使得每个线程需要拉取的数据量为400Mb左右。
在一些实施方式中，如果负载率大于或等于指定负载率时，服务器也可以确定当前非空闲的线程所执行的任务的优先级，可将优先级不高的任务对应的线程分配查询句柄，以进行数据拉取和发送，在完成本次的数据推送之后，这些线程再执行此前的任务。其中，优先级可以根据任务的实时性确定，例如，根据数据类型来确定实时性。
步骤S450：每个线程根据对应的查询句柄，拉取数据查询结果的同时将所述数据查询结果发送至所述业务端。
在本申请实施例中，步骤S450可以参阅前述实施例的内容，在此不再赘述。
本申请实施例提供的数据推送方法，通过获取业务端发送的数据查询请求，根据该查询请求，获取数据查询结果对应的多个查询句柄，然后获取线程池中的可用线程，再将多个查询句柄中的每个查询句柄分配给可用线程中的一个线程，并且保证每个查询句柄对应的线程不同，然后每个线程根据对应的查询句柄，拉取数据查询结果的同时将数据查询结果发送至业务端，从而可以实现并行的拉取和发送数据，提升数据推送的速度。并且，各个线程在拉取和发送数据时，采用边拉取边发送的方式，可以不占用过多的内存，保证程序的稳定运行。
请参阅图5，其示出了本申请实施例提供的一种数据推送装置400的结构框图。该数据推送装置400应用上述的服务器，该数据推送装置400包括：请求获取模块410、句柄获取模块420、句柄分配模块430以及数据发送模块440。其中，请求获取模块410用于获取业务端发送的数据查询请求；所述句柄获取模块420用于根据所述数据查询请求，获取数据查询结果对应的多个查询句柄；所述句柄分配模块430用于将所述多个查询句柄中的每个查询句柄分配给一个线程，每个查询句柄对应的线程不同；所述数据发送模块440用于每个线程根据对应的查询句柄，拉取数据查询结果的同时将所述数据查询结果发送至所述业务端。
在一些实施方式中，句柄获取模块420可以包括：分配确定单元以及句柄确定单元。分片确定单元用于根据所述数据查询请求，确定数据查询结果所对应的多个分片；句柄确定单元用于确定每个分片所对应的部分数据查询结果的查询句柄，获得多个查询句柄。
在该实施方式中，分片确定单元可以具体用于：根据所述数据查询请求，获取满足所述查询条件的数据查询结果所在的分片。
在该实施方式中，分片确定单元可以具体用于：根据所述数据查询请求，确定数据查询结果对应的索引；获取所述索引对应的多个分片。
在该实施方式中，数据推送装置400还可以包括：数据分片模块。数据分片模块可以用于对存储至数据库的数据进行分片。
在该实施方式中，句柄确定单元可以具体用于：获取每个分片所对应的部分数据查询结果的标识信息，将所述标识信息作为查询句柄，获得多个查询句柄。
在一些实施方式中，数据发送模块440可以具体用于：每个线程根据对应的查询句柄，在拉取预设数据量的数据后，将拉取到的数据发送至所述业务端；重复所述每个线程根据对应的查询句柄，在拉取预设数据量的数据后，将拉取到的数据发送至所述业务端，直至所述数据查询结果对应的数据全部发送至所述业务端。
在一些实施方式中，数据发送模块440还可以用于：获取当前的剩余内存值；根据所述剩余内存值，确定所述预设数据量。
在该实施方式中，数据发送模块440根据所述剩余内存值，确定所述预设数据量可以包括：判断所述剩余内存值是否大于指定内存值；如果所述剩余内存值大于所述指定内存值，则将第一数据量作为所述预设数据量；如果所述剩余内存值小于所述指定内存值，则将第二数据量作为所述预设数据量，所述第二数据量小于所述第一数据量。
在一些实施方式中，数据发送模块440还可以用于：根据所述数据查询结果的总数据量，以及指定拉取次数，获取所述总数据量与所述指定拉取次数的比值，将所述比值作为所述预设数据量。
在一些实施方式中，数据发送模块440可以具体用于：每个线程根据对应的查询句柄，通过ElasticSearch的scroll游标功能实现拉取数据查询结果的同时将所述数据查询结果发送至所述业务端。
在一些实施方式中，数据推送装置400还可以包括数据量判断模块。数据量判断模块用于在所述根据所述数据查询请求，获取数据查询结果对应的多个查询句柄之前，判断所述数据查询结果的总数据量是否大于指定数据量。如果所述总数据量大于所述指定数据量，则句柄获取模块420根据所述数据查询请求，获取数据查询结果对应的多个查询句柄。
在一些实施方式中，数据发送模块440还可以用于如果所述总数据量小于或等于所述指定数据量，则从数据库中读取全部的数据查询结果之后，将数据查询结果发送至所述业务端。
在一些实施方式中，句柄分配模块可以包括：线程获取单元以及分配单元。线程获取单元用于获取线程池中的可用线程；分配单元用于将所述多个查询句柄中的每个查询句柄分配给所述可用线程中的一个线程。
在该实施方式中，句柄分配模块还可以包括：线程判断单元。线程判断单元用于判断所述可用线程的数量是否小于所述多个查询句柄对应的数量。如果所述可用线程的数量大于或等于所述多个查询句柄对应的数量，则分配单元将所述多个查询句柄中的每个查询句柄分配给所述可用线程中的一个线程。
在该实施方式中，句柄分配模块还可以包括线程创建单元。线程创建单元用于如果所述可用线程的数量小于所述多个查询句柄对应的数量，于所述线程池中创建目标数量的线程，其中，所述目标数量与所述可用线程的数量的和值大于或等于多个查询句柄对应的数量。
在该实施方式中，线程创建单元可以具体用于：获取处理器的负载率； 如果所述负载率小于指定负载率，于所述线程池中创建目标数量的线程。
所属领域的技术人员可以清楚地了解到，为描述的方便和简洁，上述描述装置和模块的具体工作过程，可以参考前述方法实施例中的对应过程，在此不再赘述。
在本申请所提供的几个实施例中，模块相互之间的耦合可以是电性，机械或其它形式的耦合。
另外，在本申请各个实施例中的各功能模块可以集成在一个处理模块中，也可以是各个模块单独物理存在，也可以两个或两个以上模块集成在一个模块中。上述集成的模块既可以采用硬件的形式实现，也可以采用软件功能模块的形式实现。
综上所述，本申请提供的方案，通过获取业务端发送的数据查询请求，根据该查询请求，获取数据查询结果对应的多个查询句柄，将多个查询句柄中的每个查询句柄分配给一个线程，其中，每个查询句柄对应的线程不同，然后每个线程根据对应的查询句柄，拉取数据查询结果的同时将数据查询结果发送至业务端，从而可以实现并行的拉取和发送数据，提升数据推送的速度。
请参考图6，其示出了本申请实施例提供的一种服务器的结构框图。该服务器100可以是传统服务器、云服务器等能够运行应用程序的服务器。本申请中的服务器100可以包括一个或多个如下部件：处理器110、存储器120、触摸屏130以及一个或多个应用程序，其中一个或多个应用程序可以被存储在存储器120中并被配置为由一个或多个处理器110执行，一个或多个程序配置用于执行如前述方法实施例所描述的方法。
处理器110可以包括一个或者多个处理核。处理器110利用各种接口和线路连接整个服务器100内的各个部分，通过运行或执行存储在存储器120内的指令、程序、代码集或指令集，以及调用存储在存储器120内的数据，执行服务器100的各种功能和处理数据。可选地，处理器110可以采用数字信号处理(Digital Signal Processing，DSP)、现场可编程门阵列(Field－Programmable Gate Array，FPGA)、可编程逻辑阵列(Programmable Logic Array，PLA)中的至少一种硬件形式来实现。处理器110可集成中央处理器(Central Processing Unit，CPU)、图像处理器(Graphics Processing Unit，GPU)和调制解调器等中的一种或几种的组合。其中，CPU主要处理操作系统、用户界面和应用程序等；GPU用于负责显示内容的渲染和绘制；调制解调器用于处理无线通信。可以理解的是，上述调制解调器也可以不集成到处理器110中，单独通过一块通信芯片进行实现。
存储器120可以包括随机存储器(Random Access Memory，RAM)，也可以包括只读存储器(Read-Only Memory)。存储器120可用于存储指令、程序、代码、代码集或指令集。存储器120可包括存储程序区和存储数据区，其中，存储程序区可存储用于实现操作系统的指令、用于实现至少一个功能的指令(比如触控功能、声音播放功能、图像播放功能等)、 用于实现下述各个方法实施例的指令等。存储数据区还可以存储服务器100在使用中所创建的数据(比如电话本、音视频数据、聊天记录数据)等。
请参考图7，其示出了本申请实施例提供的一种计算机可读存储介质的结构框图。该计算机可读介质800中存储有程序代码，所述程序代码可被处理器调用执行上述方法实施例中所描述的方法。
计算机可读存储介质800可以是诸如闪存、EEPROM(电可擦除可编程只读存储器)、EPROM、硬盘或者ROM之类的电子存储器。可选地，计算机可读存储介质800包括非易失性计算机可读介质(non-transitory computer-readable storage medium)。计算机可读存储介质800具有执行上述方法中的任何方法步骤的程序代码810的存储空间。这些程序代码可以从一个或者多个计算机程序产品中读出或者写入到这一个或者多个计算机程序产品中。程序代码810可以例如以适当形式进行压缩。
最后应说明的是：以上实施例仅用以说明本申请的技术方案，而非对其限制；尽管参照前述实施例对本申请进行了详细的说明，本领域的普通技术人员当理解：其依然可以对前述各实施例所记载的技术方案进行修改，或者对其中部分技术特征进行等同替换；而这些修改或者替换，并不驱使相应技术方案的本质脱离本申请各实施例技术方案的精神和范围。

Claims (20)

  1. 一种数据推送方法，其特征在于，所述方法包括：
    获取业务端发送的数据查询请求；
    根据所述数据查询请求，获取数据查询结果对应的多个查询句柄；
    将所述多个查询句柄中的每个查询句柄分配给一个线程，每个查询句柄对应的线程不同；
    每个线程根据对应的查询句柄，拉取数据查询结果的同时将所述数据查询结果发送至所述业务端。
  2. 根据权利要求1所述的方法，其特征在于，所述根据所述数据查询请求，获得多个查询句柄，包括：
    根据所述数据查询请求，确定数据查询结果所对应的多个分片；
    确定每个分片所对应的部分数据查询结果的查询句柄，获得多个查询句柄。
  3. 根据权利要求2所述的方法，其特征在于，所述数据查询请求中携带有查询条件，所述根据所述数据查询请求，确定数据查询结果所对应的多个分片，包括：
    根据所述数据查询请求，获取满足所述查询条件的数据查询结果所在的分片。
  4. 根据权利要求2或3所述的方法，其特征在于，所述根据所述数据查询请求，确定数据查询结果所对应的多个分片，包括：
    根据所述数据查询请求，确定数据查询结果对应的索引；
    获取所述索引对应的多个分片。
  5. 根据权利要求2-4任一项所述的方法，其特征在于，在所述根据所述数据查询请求，确定数据查询结果所对应的多个分片之前，所述方法还包括：
    对存储至数据库的数据进行分片。
  6. 根据权利要求2-5任一项所述的方法，其特征在于，所述确定每个分片所对应的部分数据查询结果的查询句柄，获得多个查询句柄，包括：
    获取每个分片所对应的部分数据查询结果的标识信息，将所述标识信息作为查询句柄，获得多个查询句柄。
  7. 根据权利要求1-6任一项所述的方法，其特征在于，所述每个线程根据对应的查询句柄，拉取数据查询结果的同时将所述数据查询结果发送至所述业务端，包括：
    每个线程根据对应的查询句柄，在拉取预设数据量的数据后，将拉取到的数据发送至所述业务端；
    重复所述每个线程根据对应的查询句柄，在拉取预设数据量的数据后，将拉取到的数据发送至所述业务端，直至所述数据查询结果对应的数据全 部发送至所述业务端。
  8. 根据权利要求7所述的方法，其特征在于，所述每个线程根据对应的查询句柄，拉取数据查询结果的同时将所述数据查询结果发送至所述业务端，还包括：
    获取当前的剩余内存值；
    根据所述剩余内存值，确定所述预设数据量。
  9. 根据权利要求8所述的方法，其特征在于，所述根据所述剩余内存值，确定所述预设数据量，包括：
    判断所述剩余内存值是否大于指定内存值；
    如果所述剩余内存值大于所述指定内存值，则将第一数据量作为所述预设数据量；
    如果所述剩余内存值小于所述指定内存值，则将第二数据量作为所述预设数据量，所述第二数据量小于所述第一数据量。
  10. 根据权利要求8所述的方法，其特征在于，所述每个线程根据对应的查询句柄，拉取数据查询结果的同时将所述数据查询结果发送至所述业务端，还包括：
    根据所述数据查询结果的总数据量，以及指定拉取次数，获取所述总数据量与所述指定拉取次数的比值，将所述比值作为所述预设数据量。
  11. 根据权利要求1-10任一项所述的方法，其特征在于，所述每个线程根据对应的查询句柄，拉取数据查询结果的同时将所述数据查询结果发送至所述业务端，包括：
    每个线程根据对应的查询句柄，通过ElasticSearch的scroll游标功能实现拉取数据查询结果的同时将所述数据查询结果发送至所述业务端。
  12. 根据权利要求1-11任一项所述的方法，其特征在于，在所述根据所述数据查询请求，获取数据查询结果对应的多个查询句柄之前，所述方法还包括：
    判断所述数据查询结果的总数据量是否大于指定数据量；
    如果所述总数据量大于所述指定数据量，则根据所述数据查询请求，获取数据查询结果对应的多个查询句柄。
  13. 根据权利要求12所述的方法，其特征在于，所述方法还包括：
    如果所述总数据量小于或等于所述指定数据量，则从数据库中读取全部的数据查询结果之后，将数据查询结果发送至所述业务端。
  14. 根据权利要求1-13任一项所述的方法，其特征在于，所述将所述多个查询句柄中的每个查询句柄分配给一个线程，包括：
    获取线程池中的可用线程；
    将所述多个查询句柄中的每个查询句柄分配给所述可用线程中的一个线程。
  15. 根据权利要求14所述的方法，其特征在于，所述将所述多个查询句柄中的每个查询句柄分配给所述可用线程中的一个线程，包括：
    判断所述可用线程的数量是否小于所述多个查询句柄对应的数量；
    如果所述可用线程的数量大于或等于所述多个查询句柄对应的数量，则将所述多个查询句柄中的每个查询句柄分配给所述可用线程中的一个线程。
  16. 根据权利要求15所述的方法，其特征在于，所述将所述多个查询句柄中的每个查询句柄分配给所述可用线程中的一个线程，还包括：
    如果所述可用线程的数量小于所述多个查询句柄对应的数量，于所述线程池中创建目标数量的线程，其中，所述目标数量与所述可用线程的数量的和值大于或等于多个查询句柄对应的数量。
  17. 根据权利要求16所述的方法，其特征在于，在所述于所述线程池中创建目标数量的线程之前，所述方法还包括：
    获取处理器的负载率；
    如果所述负载率小于指定负载率，于所述线程池中创建目标数量的线程。
  18. 一种数据推送装置，其特征在于，所述装置包括：请求获取模块、句柄获取模块、句柄分配模块以及数据发送模块，其中，
    所述请求获取模块用于获取业务端发送的数据查询请求；
    所述句柄获取模块用于根据所述数据查询请求，获取数据查询结果对应的多个查询句柄；
    所述句柄分配模块用于将所述多个查询句柄中的每个查询句柄分配给一个线程，每个查询句柄对应的线程不同；
    所述数据发送模块用于每个线程根据对应的查询句柄，拉取数据查询结果的同时将所述数据查询结果发送至所述业务端。
  19. 一种服务器，其特征在于，包括：
    一个或多个处理器；
    存储器；
    一个或多个应用程序，其中所述一个或多个应用程序被存储在所述存储器中并被配置为由所述一个或多个处理器执行，所述一个或多个程序配置用于执行如权利要求1-17任一项所述的方法。
  20. 一种计算机可读取存储介质，其特征在于，所述计算机可读取存储介质中存储有程序代码，所述程序代码可被处理器调用执行如权利要求1-17任一项所述的方法。
PCT/CN2020/078634 2020-03-10 2020-03-10 数据推送方法、装置、服务器及存储介质 WO2021179170A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
CN202080094645.XA CN115004168A (zh) 2020-03-10 2020-03-10 数据推送方法、装置、服务器及存储介质
PCT/CN2020/078634 WO2021179170A1 (zh) 2020-03-10 2020-03-10 数据推送方法、装置、服务器及存储介质

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2020/078634 WO2021179170A1 (zh) 2020-03-10 2020-03-10 数据推送方法、装置、服务器及存储介质

Publications (1)

Publication Number Publication Date
WO2021179170A1 true WO2021179170A1 (zh) 2021-09-16

Family

ID=77670355

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/078634 WO2021179170A1 (zh) 2020-03-10 2020-03-10 数据推送方法、装置、服务器及存储介质

Country Status (2)

Country Link
CN (1) CN115004168A (zh)
WO (1) WO2021179170A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20230089667A1 (en) * 2021-09-17 2023-03-23 International Business Machines Corporation Problem solving in a database

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115952202B (zh) * 2023-02-24 2023-08-22 北京锐服信科技有限公司 一种基于高速缓存的大数据分片查询方法

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20020046230A1 (en) * 1998-04-29 2002-04-18 Daniel J. Dieterich Method for scheduling thread execution on a limited number of operating system threads
CN1704931A (zh) * 2004-05-27 2005-12-07 华为技术有限公司 网管系统对外提供信息查询的方法及装置
WO2018005613A1 (en) * 2016-06-28 2018-01-04 Solano Labs, Inc. Systems and methods for efficient distribution of stored data objects
CN110222075A (zh) * 2019-04-25 2019-09-10 视联动力信息技术股份有限公司 一种响应数据查询的方法、视联网系统以及mserver系统

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20020046230A1 (en) * 1998-04-29 2002-04-18 Daniel J. Dieterich Method for scheduling thread execution on a limited number of operating system threads
CN1704931A (zh) * 2004-05-27 2005-12-07 华为技术有限公司 网管系统对外提供信息查询的方法及装置
WO2018005613A1 (en) * 2016-06-28 2018-01-04 Solano Labs, Inc. Systems and methods for efficient distribution of stored data objects
CN110222075A (zh) * 2019-04-25 2019-09-10 视联动力信息技术股份有限公司 一种响应数据查询的方法、视联网系统以及mserver系统

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20230089667A1 (en) * 2021-09-17 2023-03-23 International Business Machines Corporation Problem solving in a database

Also Published As

Publication number Publication date
CN115004168A (zh) 2022-09-02

Similar Documents

Publication Publication Date Title
WO2022062537A1 (zh) 数据压缩方法、装置及计算机可读存储介质
CN110879800B (zh) 数据写入、压缩和读取方法、数据处理方法及装置
WO2019205371A1 (zh) 服务器、消息分配的方法及存储介质
WO2017107414A1 (zh) 文件操作方法和装置
WO2021072716A1 (zh) 内存管理方法、装置、电子设备和计算机可读介质
US20120303905A1 (en) Method and apparatus for implementing cache
WO2021179170A1 (zh) 数据推送方法、装置、服务器及存储介质
CN109240607B (zh) 一种文件读取方法和装置
CN110765031B (zh) 数据存储方法、装置、移动终端及存储介质
JP2006351004A (ja) 携帯端末機のメモリ管理方法
WO2021093365A1 (zh) 一种gpu显存管理控制方法及相关装置
WO2021057623A1 (zh) 内存回收方法、装置、电子设备及存储介质
CN106960054B (zh) 数据文件的存取方法及装置
CN114546295B (zh) 一种基于zns固态硬盘的智能写分配方法和装置
CN110955704A (zh) 一种数据管理方法、装置、设备及存储介质
CN115039091A (zh) 多键值命令的处理方法、装置、电子设备及存储介质
WO2022111699A1 (zh) 数据传输方法、电子设备及计算机可读存储介质
CN112328592A (zh) 数据存储方法、电子设备及计算机可读存储介质
CN110045924B (zh) 分级存储方法、装置、电子设备及计算机可读存储介质
CN111857992B (zh) 一种Radosgw模块中线程资源分配方法和装置
WO2021207923A1 (zh) 集群扩容方法、装置、存储介质及电子设备
CN112433848A (zh) 资源处理方法、装置、存储介质和缓存服务器
CN110674086A (zh) 数据合并方法、装置、电子设备及存储介质
CN110825732A (zh) 数据查询方法、装置、计算机设备和可读存储介质
CN115970295A (zh) 请求处理方法、装置和电子设备

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20924904

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 20.02.2023)

122 Ep: pct application non-entry in european phase

Ref document number: 20924904

Country of ref document: EP

Kind code of ref document: A1