CN115004168A

CN115004168A - Data pushing method and device, server and storage medium

Info

Publication number: CN115004168A
Application number: CN202080094645.XA
Authority: CN
Inventors: 程际
Original assignee: Guangdong Oppo Mobile Telecommunications Corp Ltd; Shenzhen Huantai Technology Co Ltd
Current assignee: Guangdong Oppo Mobile Telecommunications Corp Ltd; Shenzhen Huantai Technology Co Ltd
Priority date: 2020-03-10
Filing date: 2020-03-10
Publication date: 2022-09-02
Also published as: WO2021179170A1

Abstract

The application discloses a data pushing method, a device, a server and a storage medium, wherein the data pushing method comprises the following steps: acquiring a data query request sent by a service end; acquiring a plurality of query handles corresponding to a data query result according to the data query request; allocating each query handle in the plurality of query handles to a thread, wherein the thread corresponding to each query handle is different; and each thread pulls the data query result and simultaneously sends the data query result to the service end according to the corresponding query handle. The method can realize the quick pushing of the query result to the service end.

Description

Data pushing method and device, server and storage medium

Technical Field

The present application relates to the field of data processing technologies, and in particular, to a data pushing method, an apparatus, a server, and a storage medium.

Background

With the rapid development of network information technology, information recommendation technology based on big data technology is also developed. In the information recommendation technology, related information is recommended mainly according to tag data of a user, a business party usually queries the tag data in the information recommendation, and much time is needed for pulling and sending the data from a database.

Disclosure of Invention

In view of the foregoing problems, the present application provides a data pushing method, apparatus, server and storage medium.

In a first aspect, an embodiment of the present application provides a data pushing method, where the method includes: acquiring a data query request sent by a service end; acquiring a plurality of query handles corresponding to a data query result according to the data query request; allocating each query handle in the plurality of query handles to a thread, wherein the thread corresponding to each query handle is different; and each thread pulls a data query result and simultaneously sends the data query result to the service end according to the corresponding query handle.

In a second aspect, an embodiment of the present application provides a data pushing apparatus, where the apparatus includes: the system comprises a request acquisition module, a handle distribution module and a data sending module, wherein the request acquisition module is used for acquiring a data query request sent by a service end; the handle acquisition module is used for acquiring a plurality of query handles corresponding to a data query result according to the data query request; the handle allocation module is used for allocating each query handle in the plurality of query handles to a thread, and the thread corresponding to each query handle is different; the data sending module is used for pulling the data query result and sending the data query result to the service end according to the corresponding query handle by each thread.

In a third aspect, an embodiment of the present application provides a server, including: one or more processors; a memory; one or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs configured to perform the data pushing method provided by the first aspect above.

In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, where a program code is stored in the computer-readable storage medium, and the program code may be called by a processor to execute the data pushing method provided in the first aspect.

According to the scheme, the data query request sent by the service end is obtained, the plurality of query handles corresponding to the data query result are obtained according to the query request, each query handle in the plurality of query handles is allocated to one thread, the threads corresponding to the query handles are different, and then each thread pulls the data query result and sends the data query result to the service end according to the corresponding query handle, so that parallel pulling and data sending can be realized, and the data pushing speed is improved.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.

Fig. 1 shows a flow chart of a data pushing method according to an embodiment of the application.

Fig. 2 shows a flow chart of a data pushing method according to another embodiment of the present application.

Fig. 3 shows a flow chart of a data push method according to yet another embodiment of the present application.

Fig. 4 shows a flow chart of a data pushing method according to yet another embodiment of the present application.

FIG. 5 shows a block diagram of a data pushing device according to an embodiment of the present application.

Fig. 6 is a block diagram of a server according to an embodiment of the present application for executing a data push method according to an embodiment of the present application.

Fig. 7 is a storage unit for storing or carrying program codes for implementing a data pushing method according to an embodiment of the present application.

Detailed Description

In order to make the technical solutions better understood by those skilled in the art, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application.

In information push, a business side generally inquires the label data, and a data end pulls data from a database according to the data requirement of the business side and sends the data to the business side. Therefore, the service side can determine the push content and the push object according to the obtained data.

In a conventional data pulling and sending scheme, all data is usually read from a database to a memory, and then sent from the memory to a service party. The scheme is simple and clear, and the data export is realized.

The inventor finds that, through long-term research, in a traditional data exporting scheme, as the number and the demand of business parties continuously increase, data required by the business parties are more and more, and a data end reads massive query results into a memory, so that the time is consumed, and the memory overflow is easily triggered. For example: a business party needs to export users with applications A and B, the conditions are that more than 2 hundred million users can be easily obtained, the export of user information in hundred million levels is very slow, and the current technical scheme has a downtime risk.

In view of the above problems, the inventors propose a data pushing method, an apparatus, a server, and a storage medium provided in this embodiment of the present application, where multiple query handles corresponding to a data query result are fragmented to different threads, and each thread concurrently pulls data and sends the data to a service end according to the fragmented query handles. The specific data pushing method is described in detail in the following embodiments.

Referring to fig. 1, fig. 1 is a schematic flowchart illustrating a data pushing method according to an embodiment of the present application. In a specific embodiment, the data pushing method is applied to the data pushing device 400 shown in fig. 5 and the server 100 (fig. 6) configured with the data pushing device 400. The specific flow of the embodiment will be described below by taking a server as an example, and it is understood that the server applied in the embodiment may be a conventional server, a cloud server, and the like, and is not limited herein. As will be described in detail with respect to the flow shown in fig. 1, the data pushing method may specifically include the following steps:

step S110: and acquiring a data query request sent by a service end.

In the embodiment of the application, when the service end needs to query data, the service end can send a data query request to the server. Correspondingly, the server can receive the data query request sent by the service end, respond to the data query request, and return corresponding data to the service end.

In some embodiments, the data query request may carry a data requirement of a service party corresponding to the service end, for example, may carry a query condition. One or more conditions may be included in the query. For example, a business party wants to advertise to users who have application 1 and application 2 installed at the same time but have no application 3 installed, and the set query condition is users who have application 1 and application 2 installed at the same time but have no application 3 installed; for another example, a game company wants to award a game prize to a relatively active game player in the near term in order to increase the remaining rate of the relatively active game player in the near term, and the query condition is set to a user who has a game duration exceeding 24 hours or a login frequency exceeding 10 times in the last week.

In some embodiments, the data query request sent by the service end may be a request for obtaining tag data, or may be a request for obtaining other types of data, for example, obtaining document data. The type of data that the data query request is for obtaining may not be limiting.

Step S120: and acquiring a plurality of query handles corresponding to the data query result according to the data query request.

In the embodiment of the application, after the server acquires the data query request, the server may determine the data query result according to the data query request. The data query result may include data corresponding to the data query request. That is, the server may determine which data satisfies the demand information according to the demand information carried in the data query request. For example, the data query request is a request for querying tag data, and the data query request carries users installed with the application a and the application B, the server may determine which users installed with the application a and the application B are. For another example, the data query request is a request for querying document data, and the data query request carries keywords a and B, so that the server can determine which documents include the keywords a and B.

In the embodiment of the application, after the server determines the data query result according to the data query request, the server may determine the query handle corresponding to the data query result according to the data query result. The query handle may be information for reading a data query result. The query handle may include a storage location, an identification of the data, a field, and the like. According to the query handle, a corresponding data query result can be read. The server may store query handles corresponding to each data in advance, and the server may determine the query handle corresponding to the data query result according to the data included in the data query result.

In some embodiments, the server may divide the data query result into multiple parts of data query results according to the data query result, and determine the query handle corresponding to the data query result of each part, so that multiple query handles corresponding to the data query result may be obtained. It can be understood that the obtained multiple query handles may be used to read data, and the data read according to the multiple query handles may form a complete data query result.

As an embodiment, the server may divide the data query result into a plurality of portions according to the location of the data distribution in the data query result, and determine the query handle corresponding to each portion. For example, the data corresponding to the data query result is stored in the database cluster, so that a plurality of databases storing the data query result can be determined, and the query handle is determined according to the data query result included in the plurality of databases. For another example, the data corresponding to the data query result is stored in different segments in the database, so that a plurality of segments in which the data query result is located can be determined, and the query handle is determined according to the data query result existing in the plurality of segments.

As another embodiment, the server may also divide the data query result according to different types of data in the data query result, and determine the query handle corresponding to each part of the data query result according to the divided result.

Of course, the specific manner of obtaining the multiple query handles corresponding to the data query result may not be limited.

Step S130: and allocating each query handle in the plurality of query handles to one thread, wherein the thread corresponding to each query handle is different.

In the embodiment of the application, after determining a plurality of query handles corresponding to a data query result, the server may allocate a different thread for each query handle, so as to read data from the database and send the data to the service end.

In some embodiments, a thread pool may be created in the server, and the server may allocate a different thread from the thread pool for each query handle, where each thread is used to read data of the data query result corresponding to its allocated query handle from the database separately.

Step S140: and each thread pulls the data query result and simultaneously sends the data query result to the service end according to the corresponding query handle.

In the embodiment of the application, after the server allocates one thread to each query handle, each thread can pull the data query result from the database according to the allocated query handle, and send the data to the service end while pulling the data, so that the phenomenon that the occupied memory is too much and the memory overflows due to sending after the data is pulled once is avoided. For example, when the service end is kafka, each thread pulls data and then sends the data to kafka.

According to the data pushing method provided by the embodiment of the application, the data query request sent by the service end is obtained, the plurality of query handles corresponding to the data query result are obtained according to the query request, each query handle in the plurality of query handles is allocated to one thread, the threads corresponding to the query handles are different, and then each thread pulls the data query result and sends the data query result to the service end according to the corresponding query handle, so that parallel data pulling and sending can be realized, and the data pushing speed is improved. And when each thread pulls and sends data, a mode of pulling and sending simultaneously is adopted, so that excessive memory is not occupied, and the stable operation of the program is ensured.

Referring to fig. 2, fig. 2 is a schematic flowchart illustrating a data pushing method according to another embodiment of the present application. The data pushing method is applicable to the server, and will be described in detail with respect to the flow shown in fig. 2, and the data pushing method may specifically include the following steps:

step S210: and acquiring a data query request sent by a service end.

In the embodiment of the present application, step S210 may refer to the contents of the foregoing embodiments, which are not described herein again.

Step S220: and determining a plurality of fragments corresponding to the data query result according to the data query request.

In the embodiment of the application, when the data is stored in the database, the data can be fragmented. Specifically, the fragmentation function of elastic search (es) may be utilized to perform fragmentation of data. The Elastic Search (ES) is a distributed search engine, where an ES cluster is composed of one or more nodes, and an ES instance is a node. One index in the ES is a document set, each index has one or more fragments, and the indexed data is distributed and stored on each fragment. A shard may be understood as the smallest unit block of data, with data written into the database being allocated to be stored on the shard. Specifically, an index may be created in the ES, each index may correspond to multiple shards, the multiple shards may be at the same node, or may be at different nodes, for example, if the index IndexA has 2 shards, 10 pieces of data (10 documents) are inserted into the IndexA, then the 10 pieces of data may be divided into 5 pieces and stored in the first shard, and the remaining 5 pieces of data may be stored in another shard. The data corresponding to the index can be stored in the corresponding fragment, and when the data needs to be queried, the data can be queried from the fragment corresponding to the index.

In the embodiment of the application, when the server determines a plurality of query handles corresponding to the data query result, the server may determine the fragments corresponding to the data in the data query result. In some embodiments, the data query request sent by the service end may carry a query condition. The server can extract the characteristics in the query condition according to the query condition and determine the data matched with the characteristics as the data query result. After the data query result is determined, the server determines the fragment where the data is located in the data query result. It can be understood that the server may store information of each data and a segment where each data is located, and based on this, the server may determine a data query result that satisfies the query condition according to the query condition, and determine the segment where the data is located.

In some embodiments, the server may determine, according to the data query request, an index corresponding to the data query result, and obtain a plurality of segments corresponding to the index. It can be understood that the server may search the index corresponding to the data query result according to the determined data query result, and determine the plurality of segments corresponding to the index according to the information of the index. For example, the data query result includes index 1 and index 2, and the IDs of the fragments of index 1 and index 2 are determined.

Step S230: and determining the query handle of the partial data query result corresponding to each fragment, and obtaining a plurality of query handles.

In the embodiment of the application, after determining the plurality of fragments corresponding to the data query result, the server may determine the query handle of the partial data query result corresponding to each fragment, and obtain the plurality of query handles. The query handle may be information for reading a data query result. The query handle may include a storage location, an identification of the data, a field, and the like.

In some embodiments, the identification information of the partial data query result corresponding to each fragment is obtained, and the identification information is used as a query handle to obtain a plurality of query handles. It will be appreciated that by identifying the message, data may be read from the corresponding fragment. The data extracted from the corresponding fragment by using each query handle can form a complete data query result.

Step S240: and allocating each query handle in the plurality of query handles to one thread, wherein the thread corresponding to each query handle is different.

In the embodiment of the present application, step S240 may refer to the contents of the foregoing embodiments, which are not described herein again.

Step S250: and each thread pulls the data with preset data volume according to the corresponding query handle, and then sends the pulled data to the service end.

In the embodiment of the application, after the server allocates one thread to each query handle, each thread can pull the data query result from the database according to the allocated query handle, and send the data to the service end while pulling the data. Specifically, each thread may pull data from the database multiple times. Each thread can pull data from the corresponding fragment according to the query handle, that is, each query handle corresponds to a partial data query result in one fragment, so that when pulling data according to the query handle, each thread pulls data from the allocation corresponding to the query handle. In particular, the fetched fragment may be decided according to the reference parameter, and if the parameter starts with _ shares, this means that only the specified fragment is queried, so that each thread may pull data from the corresponding allocation by defining the parameter starting with _ shares.

In some embodiments, each thread can pull the data query result and send the data query result to the service end through a scroll cursor function of an elastic search according to the corresponding query handle. It can be understood that if we want to take a large amount of data out of the ES cluster, one-time taking out will result in too much IO pressure and performance is easy to be problematic, while paging query is easy to cause a deep paging problem, so the scroll query of ES can be used, and a batch of queries can be performed until all data are queried. For example, the queried data is the user id, each thread queries a certain amount of user ids by using its own assigned query handle, and forms a piece of message to be sent to Kafka, thereby implementing batch sending in each fragmented query handle.

In some embodiments, each thread may send the pulled data to the service end after pulling the data of the preset data amount according to the corresponding query handle. That is, after each thread pulls a batch of data, the pulled data is stored in the memory, and then the pulled data is sent from the memory.

In this embodiment, the server may further determine the data amount to be pulled each time before pulling the data, that is, the preset data amount.

As an implementation manner, the server may obtain a current remaining memory value, and determine the preset data amount according to the remaining memory value. It can be understood that, by using a manner of pulling while sending, pulling and sending data in batches, the problem that the operation is crashed due to the fact that the memory of the server may be occupied by the pulled data is mainly considered. Therefore, the preset data amount pulled each time can be determined according to the current residual memory value of the server.

In this embodiment, the determining, by the server, the preset data amount according to the current remaining memory value may include: judging whether the residual memory value is larger than a specified memory value or not; if the residual memory value is larger than the specified memory value, taking a first data volume as the preset data volume; and if the residual memory value is smaller than the specified memory value, taking a second data volume as the preset data volume, wherein the second data volume is smaller than the first data volume. The specified memory value can be used as a standard for judging whether the current memory usage of the server occupies more, if the remaining memory value is larger than the specified memory value, the current remaining memory of the server is more, and therefore, a first data volume which is larger than a second data volume can be selected as a preset data volume; if the remaining memory value is less than or equal to the specified memory value, it means that the server has less current remaining memory and the memory is more intense, so that the second data size smaller than the first data size can be used as the preset data size.

Furthermore, both the first data volume and the second data volume are smaller than the specified memory value. When the first data volume and the second data volume are determined, the server may determine, according to the remaining memory value, the allocable memory if the remaining memory value is greater than the specified memory value, calculate a ratio of the allocable memory to the number of threads corresponding to the plurality of query handles, and use the ratio as the first data volume. The server may further determine, according to the remaining memory value, an allocable memory when the remaining memory value is less than or equal to the specified memory value, calculate a ratio of the allocable memory to the number of threads corresponding to the plurality of query handles, and use the ratio as the first data size. It is to be understood that the remaining memory value should be greater than the memory allocable if the remaining memory value is greater than the specified memory value, and should be greater than the memory allocable if the remaining memory value is less than or equal to the specified memory value, and thus the determined first data amount should be greater than the second data amount.

Of course, the first data volume and the second data volume may be preset in the server. In addition, the first amount of data should satisfy: when the threads corresponding to the query handles pull data, the total data amount stored in the memory is smaller than the designated memory value; the second amount of data should satisfy: when the threads corresponding to the query handles pull data, the total data amount stored in the memory is smaller than the residual memory value.

In this embodiment, the determining, by the server, the preset data amount according to the current remaining memory value may include: and acquiring the preset data volume corresponding to the residual memory value according to the preset corresponding relation between the memory value and the preset data volume. The cache data amount that can be supported by different memory values can be determined in advance according to experiments, so as to establish the corresponding relation between the memory values and the preset data amount. It should be noted that the preset data amount determined according to the corresponding relationship can satisfy the normal operation of the program in the server.

In the embodiment, the preset data volume is determined by the memory value of the server, and on the basis of pulling the data volume in batches and avoiding the problem of excessive memory occupation, the problem of excessive memory occupation of the server is further ensured, and the normal operation of the server is ensured.

In some embodiments, the server may also obtain a ratio of the total data amount to the specified pulling times according to the total data amount of the data query result and the specified pulling times, and use the ratio as the preset data amount. It can be understood that, a specified number of pulling times may be set in the server, where the specified number of pulling times is the total number of times that each thread pulls data from the fragment. The setting of appointed number of times of drawing can satisfy the demand of different scenes, also can guarantee that the number of times of drawing of data can not be too much, and lead to the whole time of exporting data overlength. The number of pulling times may be set for the user, or may be determined according to the current read data and the speed of transmission.

Step S260: and repeating the process of each thread according to the corresponding query handle, after pulling the data with the preset data volume, sending the pulled data to the service end until all the data corresponding to the data query result are sent to the service end.

In the embodiment of the present application, step S250 is repeated until all the data in the data query result are read, and the data is sent to the service end, so that the data is pulled and sent in batches, an effect of pulling the data while sending the data can be achieved, and the problem that the pulled data occupies a large amount of the memory is avoided.

The data pushing method provided by the embodiment of the application determines a plurality of fragments corresponding to a data query result according to a data query request sent by a service end, then determines a query handle of a partial data query result corresponding to each fragment to obtain a plurality of query handles, and then allocates each query handle in the plurality of query handles to one thread, wherein the thread corresponding to each query handle is different, then each thread pulls data in batches according to the corresponding query handle and sends the data to the service end until all data in the data query result are sent to the service end, so that parallel pulling and sending of the data can be realized, the data pushing speed is increased, and when each thread pulls and sends the data, a mode of pulling and sending while not occupying too much memory is adopted, and the stable running of a program is ensured, therefore, the exporting speed of the data is greatly improved overall, in practical application, the exporting time of hundred million-level data can be shortened from half an hour to 2 minutes, and all the data are sent to a service end.

Referring to fig. 3, fig. 3 is a schematic flow chart illustrating a data pushing method according to another embodiment of the present application. The data pushing method may be applied to the server, and as will be described in detail with respect to the flow shown in fig. 3, the data pushing method may specifically include the following steps:

step S310: and acquiring a data query request sent by a service end.

Step S320: and judging whether the total data volume of the data query result is larger than the specified data volume.

In this embodiment of the present application, after the server obtains the data query request, the server may determine the data query result according to the data query request, and then determine the total data size of the data query result, that is, the data size of all data in the data query result. And the server compares the total data volume of the data query result with the specified data volume, and determines whether the total data volume is greater than the specified data volume according to the comparison result. The specified data size is used as a standard for determining whether the data size in the data query result is larger, and if the total data size is larger than the specified data size, it indicates that the data size to be pushed is larger, so steps S340 to S360 can be performed, that is, after a plurality of query handles are determined, data are read in batches from the partition by using a plurality of threads according to the allocated query handles and then sent to the service end, so that parallel data pulling is realized, the data export speed is increased, and the data are sent while being sent, thereby avoiding the problem of excessive memory occupation. If the total data amount is less than or equal to the specified data amount, it indicates that the data amount required to be pushed is not large, so step S330 may be executed, that is, after all the data query results are read from the database, the data is stored in the memory, and then the data is sent to the service end from the memory. The specific numerical value of the specified data volume is not limited, and it should be understood that the total data volume is less than or equal to the specified data volume, and all data of the data query result is stored in the memory, so that the situation that the operation is crashed due to excessive memory occupation is avoided.

Step S330: and after all the data query results are read from the database, sending the data query results to the service end.

If the total data amount is less than or equal to the specified data amount, step S330 is performed.

Step S340: and acquiring a plurality of query handles corresponding to the data query result according to the data query request.

If the total data amount is less than or equal to the specified data amount, steps S340 to S360 are performed.

Step S350: and allocating each query handle in the plurality of query handles to one thread, wherein the thread corresponding to each query handle is different.

Step S360: and each thread pulls a data query result and simultaneously sends the data query result to the service end according to the corresponding query handle.

In the embodiment of the present application, steps S330 to S360 may refer to the contents of the foregoing embodiments, and are not described herein again.

The data pushing method provided by the embodiment of the application judges whether the total data volume of the data query result is larger than the specified data volume or not by acquiring the data query request sent by the service end according to the query request, if so, acquires a plurality of query handles corresponding to the data query result, allocates each query handle in the plurality of query handles to one thread, wherein the thread corresponding to each query handle is different, then each thread pulls the data query result and simultaneously sends the data query result to the service end according to the corresponding query handle, thereby realizing parallel pulling and sending of data when the total data volume of the data query result is larger than the specified data volume, improving the speed of data pushing, and adopting a mode of pulling while sending when pulling and sending data by each thread, not occupying too much memory, and ensuring the stable operation of the program.

Referring to fig. 4, fig. 4 is a schematic flowchart illustrating a data pushing method according to still another embodiment of the present application. The data pushing method may be applied to the server, and as will be described in detail with respect to the flow shown in fig. 4, the data pushing method may specifically include the following steps:

step S410: and acquiring a data query request sent by a service end.

Step S420: and acquiring a plurality of query handles corresponding to the data query result according to the data query request.

In the embodiment of the present application, step S410 and step S420 may refer to the contents of the foregoing embodiments, and are not described herein again.

Step S430: and acquiring the available threads in the thread pool.

In an embodiment of the present application, the server may utilize a thread pool to assign threads to query handles. The thread pool is a multi-thread processing form, tasks can be distributed to the threads, and the threads can automatically execute the tasks. The server may create a thread pool in advance, or create a thread pool when data push is needed. For example, the server may create the thread pool by using a threadpoolsexcecutor class, or may create the thread pool by using an Executors class, and a specific manner of creating the thread pool may not be limited.

In some embodiments, the server may obtain available threads in the thread pool, where the available threads may refer to threads that are in an idle state, i.e., threads that have no task to execute. The available threads may be used to assign query handles to perform tasks that pull data from the partition.

Step S440: assigning each query handle of the plurality of query handles to one of the available threads.

In this embodiment of the present application, after determining an available thread from a thread pool, a server may assign a query handle to a thread in the available threads, where each query handle should be guaranteed to be assigned to an available thread, so as to guarantee that all data query results can be pulled out for sending.

In some embodiments, after determining the available threads, the server may further determine a number of the available threads, compare the number with a number corresponding to the plurality of query handles, and determine whether the number of the available threads is less than the number corresponding to the plurality of query handles based on the comparison. If the number of available threads is greater than or equal to the number corresponding to the plurality of query handles, the indication of the currently available threads may ensure that each query handle is assigned to a different thread, and thus each query handle of the plurality of query handles may be assigned to a thread of the available threads. If the number of available threads is less than the number corresponding to the plurality of query handles, then this indicates that the currently available threads cannot guarantee that each query handle is assigned to a different thread. At this time, it is necessary to ensure that all query handles can be allocated to the threads to complete the pulling and sending of the entire data query result.

For one embodiment, if the number of available threads is less than the number corresponding to the plurality of query handles, the number of threads in the thread pool can be used to create a target number, wherein the sum of the target number and the number of available threads is greater than or equal to the number corresponding to the plurality of query handles. The server can request to allocate the available resources of the processor to create a target number of new threads into the thread pool, and the sum of the target number and the number of the available threads is greater than or equal to the number corresponding to the plurality of query handles, so that after the threads are created, the number of the available threads can ensure that each query handle is allocated to a different thread.

In this embodiment, before creating the thread into the thread pool, the server may further obtain a load rate of the processor, compare the load rate with a specified load rate, and determine whether the load rate is less than the specified load rate according to a comparison result. Wherein the specified load rate is used to determine whether the load of the processor is high, if the load rate is less than the specified load rate, the load of the processor is not high, so that only the target number of threads can be created in the thread pool. If the load rate is greater than or equal to the specified load rate, it indicates that the load of the processor is high and it is not suitable to create a new thread into the thread pool.

In some embodiments, if the load rate is greater than or equal to the specified load rate, the server may adopt a mode of respectively giving at least two query handles to the same thread, so as to ensure that all the query handles are distributed to the threads, and data corresponding to the data query result can be all pulled and sent.

As an embodiment, if the load rate is greater than or equal to the specified load rate, the server may determine a data amount of the partial data query result corresponding to each query handle in the plurality of query handles, and then respectively give at least two query handles to the same thread according to the data amount, and make the size of the data amount pulled and sent by each thread substantially consistent.

For example, there are 8 query handles, which are query handle 1, query handle 2, query handle 3, query handle 4, query handle 5, query handle 6, query handle 7, and query handle 8, where the data volume corresponding to query handle 1 is 100 megabytes (Mb), the data volume corresponding to query handle 2 is 200Mb, the data volume corresponding to query handle 3 is 400Mb, the data volume corresponding to query handle 4 is 300Mb, the data volume corresponding to query handle 5 is 200Mb, the data volume corresponding to query handle 6 is 150Mb, the data volume corresponding to query handle 7 is 150Mb, and the data volume corresponding to query handle 8 is 110 Mb. The query handle 1 and the query handle 4 may be allocated to the same thread, the query handle 2 and the query handle 5 may be allocated to the same thread, the query handle 4 may be allocated to the same thread, and the query handle 6, the query handle 7, and the query handle 8 may be allocated to the same thread, so that the amount of data that each thread needs to pull is about 400 Mb.

In some embodiments, if the load rate is greater than or equal to the specified load rate, the server may also determine the priority of the task executed by the current non-idle thread, may allocate a query handle to the thread corresponding to the task with a low priority to perform data pulling and sending, and after the data pushing of this time is completed, the threads execute the previous task again. The priority may be determined according to the real-time performance of the task, for example, according to the data type.

Step S450: and each thread pulls a data query result and simultaneously sends the data query result to the service end according to the corresponding query handle.

In the embodiment of the present application, step S450 may refer to the contents of the foregoing embodiments, and is not described herein again.

According to the data pushing method provided by the embodiment of the application, the data query request sent by the service end is obtained, the plurality of query handles corresponding to the data query result are obtained according to the query request, then the available threads in the thread pool are obtained, each query handle in the plurality of query handles is distributed to one thread in the available threads, the threads corresponding to each query handle are different, then each thread pulls the data query result and sends the data query result to the service end according to the corresponding query handle, and therefore parallel pulling and data sending can be achieved, and the data pushing speed is improved. And when each thread pulls and sends data, a mode of pulling and sending simultaneously is adopted, so that excessive memory is not occupied, and the stable operation of the program is ensured.

Referring to fig. 5, a block diagram of a data pushing apparatus 400 according to an embodiment of the present disclosure is shown. The data pushing apparatus 400 applies the above-mentioned server, and the data pushing apparatus 400 includes: a request acquisition module 410, a handle acquisition module 420, a handle assignment module 430, and a data transmission module 440. The request obtaining module 410 is configured to obtain a data query request sent by a service end; the handle obtaining module 420 is configured to obtain a plurality of query handles corresponding to a data query result according to the data query request; the handle allocating module 430 is configured to allocate each query handle in the multiple query handles to a thread, where the thread corresponding to each query handle is different; the data sending module 440 is configured to send the data query result to the service end while pulling the data query result according to the corresponding query handle by each thread.

In some implementations, the handle acquisition module 420 may include: an allocation determination unit and a handle determination unit. The fragment determining unit is used for determining a plurality of fragments corresponding to a data query result according to the data query request; the handle determining unit is used for determining the query handle of the partial data query result corresponding to each fragment, and obtaining a plurality of query handles.

In this embodiment, the slice determining unit may be specifically configured to: and acquiring the fragment where the data query result meeting the query condition is located according to the data query request.

In this embodiment, the slice determining unit may be specifically configured to: determining an index corresponding to a data query result according to the data query request; and acquiring a plurality of fragments corresponding to the index.

In this embodiment, the data pushing apparatus 400 may further include: and a data slicing module. The data fragmentation module may be configured to fragment data stored to the database.

In this embodiment, the handle determining unit may be specifically configured to: and acquiring identification information of a partial data query result corresponding to each fragment, and taking the identification information as a query handle to acquire a plurality of query handles.

In some embodiments, the data sending module 440 may be specifically configured to: each thread pulls data with a preset data volume according to the corresponding query handle, and then sends the pulled data to the service end; and repeating the process of sending the pulled data to the service end after pulling the data with the preset data volume according to the corresponding query handle until all the data corresponding to the data query result are sent to the service end.

In some embodiments, the data sending module 440 may further be configured to: acquiring a current residual memory value; and determining the preset data volume according to the residual memory value.

In this embodiment, the determining, by the data sending module 440, the preset data amount according to the remaining memory value may include: judging whether the residual memory value is larger than a specified memory value or not; if the residual memory value is larger than the specified memory value, taking a first data volume as the preset data volume; and if the residual memory value is smaller than the specified memory value, taking a second data volume as the preset data volume, wherein the second data volume is smaller than the first data volume.

In some embodiments, the data sending module 440 may further be configured to: and acquiring the ratio of the total data volume to the specified pulling times according to the total data volume of the data query result and the specified pulling times, and taking the ratio as the preset data volume.

In some embodiments, the data sending module 440 may be specifically configured to: and each thread pulls a data query result and simultaneously sends the data query result to the service end through a scroll cursor function of an ElasticSearch according to the corresponding query handle.

In some embodiments, the data pushing apparatus 400 may further include a data amount determination module. The data volume judging module is used for judging whether the total data volume of the data query result is larger than the specified data volume before the plurality of query handles corresponding to the data query result are obtained according to the data query request. If the total data amount is greater than the specified data amount, the handle obtaining module 420 obtains a plurality of query handles corresponding to the data query result according to the data query request.

In some embodiments, the data sending module 440 may be further configured to, if the total data amount is less than or equal to the specified data amount, send the data query result to the service end after reading all the data query results from the database.

In some implementations, the handle assignment module can include: a thread acquisition unit and an allocation unit. The thread obtaining unit is used for obtaining available threads in the thread pool; an allocation unit is to allocate each of the plurality of query handles to one of the available threads.

In this embodiment, the handle assignment module may further include: and a thread judgment unit. And the thread judging unit is used for judging whether the number of the available threads is less than the number corresponding to the plurality of query handles. If the number of the available threads is larger than or equal to the number corresponding to the plurality of query handles, the allocation unit allocates each query handle in the plurality of query handles to one thread in the available threads.

In this embodiment, the handle assignment module may further include a thread creation unit. The thread creating unit is used for creating a target number of threads in the thread pool if the number of the available threads is less than the number corresponding to the plurality of query handles, wherein the sum of the target number and the number of the available threads is greater than or equal to the number corresponding to the plurality of query handles.

In this embodiment, the thread creating unit may specifically be configured to: acquiring the load rate of a processor; and if the load rate is less than the specified load rate, creating a target number of threads in the thread pool.

It can be clearly understood by those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described apparatuses and modules may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.

In the several embodiments provided in the present application, the coupling between the modules may be electrical, mechanical or other type of coupling.

In addition, functional modules in the embodiments of the present application may be integrated into one processing module, or each of the modules may exist alone physically, or two or more modules are integrated into one module. The integrated module can be realized in a hardware mode, and can also be realized in a software functional module mode.

To sum up, according to the scheme provided by the application, the data query request sent by the service end is obtained, the plurality of query handles corresponding to the data query result are obtained according to the query request, each query handle in the plurality of query handles is allocated to one thread, wherein the threads corresponding to each query handle are different, and then each thread pulls the data query result and sends the data query result to the service end according to the corresponding query handle, so that parallel pulling and sending of data can be realized, and the data pushing speed is improved.

Referring to fig. 6, a block diagram of a server according to an embodiment of the present disclosure is shown. The server 100 may be a conventional server, a cloud server, or the like capable of running an application. The server 100 in the present application may include one or more of the following components: a processor 110, a memory 120, a touch screen 130, and one or more applications, wherein the one or more applications may be stored in the memory 120 and configured to be executed by the one or more processors 110, the one or more programs configured to perform the methods as described in the aforementioned method embodiments.

Processor 110 may include one or more processing cores. The processor 110 connects various parts within the overall server 100 using various interfaces and lines, performs various functions of the server 100 and processes data by executing or executing instructions, programs, code sets, or instruction sets stored in the memory 120, and calling data stored in the memory 120. Alternatively, the processor 110 may be implemented in hardware using at least one of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). The processor 110 may integrate one or more of a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), a modem, and the like. Wherein, the CPU mainly processes an operating system, a user interface, an application program and the like; the GPU is used for rendering and drawing display content; the modem is used to handle wireless communications. It is understood that the modem may not be integrated into the processor 110, but may be implemented by a communication chip.

The Memory 120 may include a Random Access Memory (RAM) or a Read-Only Memory (Read-Only Memory). The memory 120 may be used to store instructions, programs, code sets, or instruction sets. The memory 120 may include a stored program area and a stored data area, wherein the stored program area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playing function, an image playing function, etc.), instructions for implementing various method embodiments described below, and the like. The storage data area may also store data created by the server 100 in use (such as phone books, audio and video data, chat log data), and the like.

Referring to fig. 7, a block diagram of a computer-readable storage medium according to an embodiment of the present application is shown. The computer-readable medium 800 has stored therein a program code that can be called by a processor to execute the method described in the above-described method embodiments.

The computer-readable storage medium 800 may be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read only memory), an EPROM, a hard disk, or a ROM. Alternatively, the computer-readable storage medium 800 includes a non-volatile computer-readable storage medium. The computer readable storage medium 800 has storage space for program code 810 to perform any of the method steps of the method described above. The program code can be read from or written to one or more computer program products. The program code 810 may be compressed, for example, in a suitable form.

Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit the same; although the present application has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; such modifications and substitutions do not necessarily depart from the spirit and scope of the corresponding technical solutions in the embodiments of the present application.

Claims

A method for pushing data, the method comprising:

acquiring a data query request sent by a service end;

acquiring a plurality of query handles corresponding to a data query result according to the data query request;

allocating each query handle in the plurality of query handles to a thread, wherein the thread corresponding to each query handle is different;

and each thread pulls a data query result and simultaneously sends the data query result to the service end according to the corresponding query handle.
The method of claim 1, wherein obtaining a plurality of query handles according to the data query request comprises:

determining a plurality of fragments corresponding to a data query result according to the data query request;

and determining the query handle of the partial data query result corresponding to each fragment, and obtaining a plurality of query handles.
The method according to claim 2, wherein the data query request carries a query condition, and the determining, according to the data query request, the plurality of segments corresponding to the data query result includes:

and acquiring the fragment where the data query result meeting the query condition is located according to the data query request.
The method according to claim 2 or 3, wherein the determining, according to the data query request, the plurality of segments corresponding to the data query result includes:

determining an index corresponding to a data query result according to the data query request;

and acquiring a plurality of fragments corresponding to the index.
The method according to any of claims 2-4, wherein before said determining, according to the data query request, a plurality of segments corresponding to a data query result, the method further comprises:

and slicing the data stored in the database.
The method according to any one of claims 2 to 5, wherein the determining the query handle of the partial data query result corresponding to each segment obtains a plurality of query handles, including:

and acquiring identification information of a partial data query result corresponding to each fragment, and acquiring a plurality of query handles by taking the identification information as the query handles.
The method according to any one of claims 1 to 6, wherein each thread pulls a data query result and sends the data query result to the service end according to the corresponding query handle, and the method comprises:

each thread pulls data with a preset data volume according to the corresponding query handle, and then sends the pulled data to the service end;

and repeating the process of each thread according to the corresponding query handle, after pulling the data with the preset data volume, sending the pulled data to the service end until all the data corresponding to the data query result are sent to the service end.
The method of claim 7, wherein each thread pulls a data query result according to a corresponding query handle and simultaneously sends the data query result to the service end, further comprising:

acquiring a current residual memory value;

and determining the preset data volume according to the residual memory value.
The method according to claim 8, wherein the determining the predetermined amount of data according to the remaining memory value comprises:

judging whether the residual memory value is larger than a specified memory value or not;

if the residual memory value is larger than the specified memory value, taking a first data volume as the preset data volume;

and if the residual memory value is smaller than the specified memory value, taking a second data volume as the preset data volume, wherein the second data volume is smaller than the first data volume.
The method of claim 8, wherein each thread pulls a data query result according to a corresponding query handle and simultaneously sends the data query result to the service end, further comprising:

and acquiring the ratio of the total data volume to the specified pulling times according to the total data volume of the data query result and the specified pulling times, and taking the ratio as the preset data volume.
The method according to any one of claims 1 to 10, wherein each thread, according to the corresponding query handle, pulls the data query result and simultaneously sends the data query result to the service end, including:

and each thread pulls a data query result and simultaneously sends the data query result to the service end through a scroll cursor function of an ElasticSearch according to the corresponding query handle.
The method according to any one of claims 1-11, wherein before the obtaining a plurality of query handles corresponding to data query results according to the data query request, the method further comprises:

judging whether the total data volume of the data query result is larger than a specified data volume;

and if the total data volume is larger than the specified data volume, acquiring a plurality of query handles corresponding to the data query result according to the data query request.
The method of claim 12, further comprising:

and if the total data volume is less than or equal to the specified data volume, reading all data query results from the database, and then sending the data query results to the service end.
The method of any of claims 1-13, wherein assigning each query handle of the plurality of query handles to a thread comprises:

acquiring available threads in a thread pool;

assigning each query handle of the plurality of query handles to one of the available threads.
The method of claim 14, wherein assigning each of the plurality of query handles to one of the available threads comprises:

judging whether the number of the available threads is smaller than the number corresponding to the plurality of query handles;

and if the number of the available threads is larger than or equal to the number corresponding to the plurality of query handles, allocating each query handle in the plurality of query handles to one thread in the available threads.
The method of claim 15, wherein assigning each of the plurality of query handles to one of the available threads further comprises:

and if the number of the available threads is less than the number corresponding to the plurality of query handles, creating a target number of threads in the thread pool, wherein the sum of the target number and the number of the available threads is greater than or equal to the number corresponding to the plurality of query handles.
The method of claim 16, wherein prior to said creating a target number of threads in the thread pool, the method further comprises:

acquiring the load rate of a processor;

and if the load rate is less than the specified load rate, creating a target number of threads in the thread pool.
A data pushing apparatus, the apparatus comprising: a request acquisition module, a handle allocation module and a data transmission module, wherein,

the request acquisition module is used for acquiring a data query request sent by a service end;

the handle acquisition module is used for acquiring a plurality of query handles corresponding to a data query result according to the data query request;

the handle allocation module is used for allocating each query handle in the plurality of query handles to a thread, and the thread corresponding to each query handle is different;

and the data sending module is used for pulling the data query result and sending the data query result to the service end according to the corresponding query handle by each thread.
A server, comprising:

one or more processors;

a memory;

one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, the one or more programs configured to perform the method of any of claims 1-17.
A computer-readable storage medium, characterized in that a program code is stored in the computer-readable storage medium, which program code can be called by a processor to execute the method according to any of claims 1-17.