WO2022206474A1

WO2022206474A1 - Data acquisition method and apparatus, electronic device, and computer-readable storage medium

Info

Publication number: WO2022206474A1
Application number: PCT/CN2022/082205
Authority: WO
Inventors: 杨武
Original assignee: 北京字节跳动网络技术有限公司
Priority date: 2021-03-30
Filing date: 2022-03-22
Publication date: 2022-10-06
Also published as: CN115145956A

Abstract

A data acquisition method and apparatus, an electronic device, and a computer-readable storage medium. The data acquisition method comprises: acquiring a first data source, wherein the first data source is a data source satisfying a preset condition; acquiring, from a data storage area, according to a first preset time interval, data generated by the first data source, and caching the data into a first cache area, data in the first cache area being stored in the form of data blocks, and each data block comprising data generated within a unit time; receiving a data acquisition request sent by a client; acquiring a first data block from the first cache area; and sending the first data block to the client.

Description

Data acquisition method, apparatus, electronic device, and computer-readable storage medium

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is based on the Chinese application with the application number of 202110337448.2 and the filing date of March 30, 2021, and claims its priority. The disclosure of the Chinese application is hereby incorporated into this application as a whole.

technical field

The present disclosure relates to the field of data acquisition, and in particular, to a data acquisition method, apparatus, electronic device, and computer-readable storage medium.

Background technique

In a live broadcast scenario, real-time interaction is required between viewers and viewers, and between viewers and anchors. Due to the rapid development of the live broadcast business, the number of viewers in the live broadcast room is increasing day by day, which leads to the exponential growth of CPU computing resources. An optimization solution is needed to improve the performance of the server, thereby increasing the utilization rate of CPU computing resources.

SUMMARY OF THE INVENTION

This Summary is provided to introduce concepts in a simplified form that are described in detail in the Detailed Description section that follows. This summary section is not intended to identify key features or essential features of the claimed technical solution, nor is it intended to be used to limit the scope of the claimed technical solution.

In order to solve the above technical problems, the embodiments of the present disclosure propose the following technical solutions.

In a first aspect, an embodiment of the present disclosure provides a data acquisition method, including:

acquiring a first data source, wherein the first data source is a data source that satisfies a preset condition;

Acquire the data generated by the first data source from the data storage area according to the first preset time interval and cache the data in the first cache area; wherein, the data in the first cache area is divided into data blocks It is stored in the form of data block, and the data block includes the data generated in unit time;

Receive the data acquisition request sent by the client;

Acquire the first data block corresponding to the data acquisition request from the first cache area;

Sending the first data block to the client.

In some embodiments, the preset condition includes: the number of clients connected to the data source is greater than a first threshold; or, the amount of data generated by the data source within a preset time is greater than a second threshold.

In some embodiments, acquiring the data generated by the first data source from a data storage area according to a first preset time interval and caching the data in the first cache area includes:

In response to the elapse of a first preset time interval, acquiring data generated by the first data source during the duration of the first preset time interval from the data storage area; wherein the data includes a data timestamp;

Organize data with data timestamps within the same unit of time into data blocks;

The data block is cached in the first cache area.

In some embodiments, the data acquisition request includes a mark of the first data source and a data cursor, and the acquiring a first data block corresponding to the data acquisition request from the first cache area includes: :

determining a first cache area according to the mark of the first data source;

The first data block corresponding to the data cursor is acquired from the first cache area according to the data cursor.

In some embodiments, the data cursor includes a flag to start data or a timestamp of the first data block.

In some embodiments, acquiring the first data block corresponding to the data cursor from the first cache area according to the data cursor includes:

The first data block with the flag of the start data is queried from the first cache area or the first data block with the timestamp of the first data block is queried from the first cache area.

In some embodiments, obtaining the first data block corresponding to the data obtaining request from the first cache area according to the data cursor includes:

If the data cursor is the default cursor, acquire the latest first data block from the first cache area; the default cursor indicates that the data acquisition request is the first request by the client for the first data source data acquisition request.

In some embodiments, the acquiring the data generated by the first data source and caching the data in the first cache area includes:

obtaining data generated by the first data source;

compressing the data to obtain compressed data;

The compressed data is cached in the first cache area.

In some embodiments, the sending the first data block to the client comprises:

Generate the next data cursor;

The next data cursor and the first data block are sent to the client.

In a second aspect, an embodiment of the present disclosure provides a data acquisition device, including:

a data source acquisition module, configured to acquire a first data source, wherein the first data source is a data source that satisfies a preset condition;

A data cache module, configured to acquire the data generated by the first data source from the data storage area at preset time intervals and cache the data in the first cache area; wherein, the data in the first cache area The data is stored in the form of data blocks, and the data blocks include data generated in a unit time;

The request receiving module is used to receive the data acquisition request sent by the client;

a data block obtaining module, configured to obtain the first data block corresponding to the data obtaining request from the first cache area;

A data block sending module, configured to send the first data block to the client.

In some embodiments, the data cache module is further used for:

The data block is cached in the first cache area.

In some embodiments, the data block obtaining module is also used to:

determining a first cache area according to the mark of the first data source;

In some embodiments, the data block obtaining module is further configured to:

The first data block with the flag of the start data is queried from the first cache or the first data block with the timestamp of the first data block is queried from the first cache.

In some embodiments, the data block obtaining module is further configured to:

In some embodiments, the data cache module is further used for:

obtaining data generated by the first data source;

compressing the data to obtain compressed data;

The compressed data is cached in the first cache area.

In some embodiments, the data block sending module is further configured to:

Generate the next data cursor;

The next data cursor and the first data block are sent to the client.

In a third aspect, an embodiment of the present disclosure provides an electronic device, including:

memory for storing computer-readable instructions; and

A processor, configured to execute the computer-readable instructions, so that the electronic device implements the method described in any one of the above-mentioned first aspects.

In a fourth aspect, embodiments of the present disclosure provide a non-transitory computer-readable storage medium for storing computer-readable instructions, and when the computer-readable instructions are executed by a computer, enable the computer to implement the first aspect above. The method of any one.

In a fifth aspect, an embodiment of the present disclosure further provides a computer program product bearing program code, wherein the instructions included in the program code, when executed by the computer, cause the computer to implement the method described in any one of the first aspects above.

In a sixth aspect, an embodiment of the present disclosure further provides a computer program, where the instructions included in the computer program, when executed by the computer, cause the computer to implement the method described in any one of the foregoing first aspects.

The above description is only an overview of the technical solutions of the present disclosure. In order to understand the technical means of the present disclosure more clearly, it can be implemented according to the content of the description, and to make the above and other purposes, features and advantages of the present disclosure more obvious and easy to understand , the following specific preferred embodiments, and in conjunction with the accompanying drawings, are described in detail as follows.

Description of drawings

The above and other features, advantages and aspects of various embodiments of the present disclosure will become more apparent when taken in conjunction with the accompanying drawings and with reference to the following detailed description. Throughout the drawings, the same or similar reference numbers refer to the same or similar elements. It should be understood that the drawings are schematic and that the originals and elements are not necessarily drawn to scale.

FIG. 1 is a schematic flowchart of a data acquisition method provided by an embodiment of the present disclosure;

FIG. 2 is a further schematic flowchart of a data acquisition method provided by an embodiment of the present disclosure;

FIG. 3 is a schematic diagram of an application scenario of the data acquisition method provided by an embodiment of the present disclosure.

FIG. 4 is a schematic structural diagram of an embodiment of a data acquisition apparatus provided by an embodiment of the present disclosure;

FIG. 5 is a schematic structural diagram of an electronic device provided according to an embodiment of the present disclosure.

Detailed ways

Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While certain embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be construed as limited to the embodiments set forth herein, but rather are provided for the purpose of A more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes, and are not intended to limit the protection scope of the present disclosure.

It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and/or in parallel. Furthermore, method embodiments may include additional steps and/or omit performing the illustrated steps. The scope of the present disclosure is not limited in this regard.

As used herein, the term "including" and variations thereof are open-ended inclusions, ie, "including but not limited to". The term "based on" is "based at least in part on." The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions of other terms will be given in the description below.

It should be noted that concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or interdependence.

It should be noted that the modifications of "a" and "a plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless the context clearly indicates otherwise, they should be understood as "one or a plurality of". multiple".

There are generally two ways to obtain interactive messages in the live broadcast room: one is that the client actively polls and pulls from the server, and the other is that the server pushes new messages to the client. For the first type, the client requests messages from the server at regular intervals. After the server determines the message sent to the client, it will compress the returned data, and the client will decompress the data after getting the data. Compression is a relatively large overhead for server resources. In the case of many clients, the server needs to compress data for each request, which will consume a lot of cpu (central processing unit) resources, so bandwidth and cpu resources There is a contradiction between them. If you want to obtain benefits on one side, you must increase the cost of the other.

The embodiments of the present disclosure disclose a data acquisition method, an apparatus, an electronic device, and a computer-readable storage medium. The data acquisition method includes: acquiring a first data source, wherein the first data source is a data source that satisfies a preset condition; acquiring the first data source from a data storage area according to a first preset time interval and cache the data in the first cache area; wherein, the data in the first cache area is stored in the form of data blocks, and the data blocks include data generated in a unit time; receiving customers the data acquisition request sent by the client; acquire the first data block corresponding to the data acquisition request from the first cache area; and send the first data block to the client. The above method solves the technical problem of wasting computing resources caused by acquiring data by periodically reading the data of the data source into the cache area, and obtaining data from the cache area to the client after receiving the data acquisition request.

1 is a flowchart of an embodiment of a data acquisition method provided by an embodiment of the present disclosure. The data acquisition method provided by this embodiment may be executed by a data acquisition device, and the data acquisition device may be implemented as software, or as software and A combination of hardware, the data acquisition device can be integrated in a device in a data acquisition system, such as a data acquisition service terminal. As shown in Figure 1, the method includes the following steps:

Step S101: Obtain a first data source, wherein the first data source is a data source that satisfies a preset condition.

In this embodiment, the data to be acquired by the client is generated by the data source. Exemplarily, the data source is a live broadcast room in a live broadcast scenario, the live broadcast room includes a host client and a viewer client, and the host client and the audience client interact in the live broadcast room to generate a message, and the message is The data generated by the live studio as the data source.

In this step, a first data source is acquired, and the first data source is a data source that satisfies a preset condition. The preset condition includes: the number of clients connected to the data source is greater than the first threshold; or, the amount of data generated by the data source within the preset time is greater than the second threshold. Exemplarily, if the data source is a live broadcast room, the preset condition may be that the number of clients connected to the live broadcast room is greater than the first threshold. For example, if there are more than 1,000 clients connected to the live broadcast room, it is a popular live broadcast. time, that is, the first data source; or, the preset condition may be that more than 1,000 messages are generated in a unit time. For example, a live room with more than 1,000 messages generated within one minute is a popular live room. , that is, the first data source. It can be understood that the above preset conditions are only examples and do not constitute a limitation of the present disclosure. As long as the preset conditions of data sources that generate more data can be determined, they can be set as preset conditions, which will not be repeated here.

Returning to FIG. 1, the data acquisition method further includes step S102: acquiring data generated by the first data source from a data storage area according to a first preset time interval and caching the data in the first cache area wherein, the data in the first cache area is stored in the form of data blocks, and the data blocks include data generated in a unit time.

The data generated by the first data source is stored in a data storage area. Exemplarily, the data storage area may be a storage area of a server, such as a memory in a server, or a separate storage area, such as a storage area with a server. connected network storage, etc. The data generated by the first data source may be stored in the data storage area in a suitable manner, eg in real time, ie as soon as it is generated.

In some embodiments, as shown in FIG. 2 , the step S102 includes:

Step S201, in response to passing a first preset time interval, obtain data generated by the first data source in the duration of the time interval from a data storage area; wherein, the data includes a data timestamp;

Step S202, organize the data of the data time stamp within the same unit time into data blocks;

Step S203, the data block is cached in the first cache area.

In this step S201, periodically acquiring data generated by a first data source from the data storage area according to a first preset time interval. As an example, the data generated by the first data source is acquired from the data storage area every first preset time interval, whereby the data acquisition can be performed periodically. The data stored in the storage area includes a time stamp, and the time stamp may be the generation time of the data or the storage time of the data. In particular, whenever data is to be acquired, data within a preset time interval before or after the current time/moment at which the data is acquired may be acquired. For example, the time when data is to be acquired may be the current system time when data is to be acquired, or data within a preset time interval before or after the current system time. As an example, the server asynchronously acquires, from the storage area according to the first preset time interval, the data generated by the first data source between the time when the data was acquired last time and the time when the data was acquired this time. data. Exemplarily, if the current system time is 9:00:01, and the first preset time interval is 1 second, the server obtains the timestamp from the storage area between 9:00:00 and 9:00:01 between the data generated by the first data source.

In step S202, after the data generated by the first data source is acquired, the data is firstly organized according to unit time to generate data blocks. The duration of the unit time may be any duration, which is exemplary, but if the unit time is 1 second, the data with timestamps within the same 1 second are organized into data blocks. For example, organize data with timestamps between 10:00:00 and 10:00:01 (excluding 10:00:01) into data blocks, and mark the data block with a timestamp of 10:00:00; Organize data between 10:00:01 and 10:00:02 (excluding 10:00:02) into data blocks, and mark the data block with a timestamp of 10:00:01, and so on. It can be understood that the unit time is only an example, in fact, any unit time length may be used as a unit for organizing data blocks, which will not be repeated here. The first preset time interval may be the same as the length of the unit time, and the data obtained each time is directly organized into data blocks; or, the first preset time interval may be greater than the length of the unit time. , the data obtained each time is divided according to the units and then organized into data blocks.

In step S203, the data block is cached in the first cache area. The first cache area is a different storage area from the data storage area, and the first cache area may be constituted by a cache, so that the data reading speed is faster.

obtaining data generated by the first data source;

compressing the data to obtain compressed data;

The compressed data is cached in the first cache area.

In this optional embodiment, after the data is compressed, the compressed data is cached in the first cache area. It can be understood that the compression operation may be performed after the data is organized into data blocks, or may be performed before the data is organized into data blocks.

In one embodiment, the step S102 is performed by a data proxy instance, and the data proxy instance is generated by the proxy layer of the server. For the first data source, multiple data proxy instances may be generated, each data proxy instance corresponds to a first cache area, and each data proxy instance executes the step S102 to asynchronously acquire the first cache area from the storage area. The data generated by the data source are respectively stored in the corresponding multiple first cache areas; or, the multiple data proxy instances correspond to one first cache area, and one of the data proxy instances asynchronously retrieves the data from the storage area. The data generated by the first data source is acquired and stored in the first cache area, and multiple data proxy instances share the data in the first cache area. It can be understood that the first cache area is marked with the first data source, or the data or data blocks generated by the first data source are marked with the first data source.

Returning to FIG. 1 , the data acquisition method further includes step S103 : receiving a data acquisition request sent by the client.

In this step, the server periodically receives the data acquisition request sent by the client.

The client obtains data from the server in a polling manner, and the polling period is a second preset time interval; the second preset time interval may be the same as the first preset time interval, or may be the same as the The first preset time intervals are different.

In one embodiment, a data proxy instance is used to receive a data acquisition request sent by a client. Since there are multiple clients connected to the server, each client sends a data acquisition request. In order to reduce the burden of some data proxy instances, you can The data acquisition request sent by the client is evenly distributed to the data proxy instance through the load balancing layer. Optionally, the load balancing layer randomly assigns the data acquisition request to a data proxy instance or allocates the data acquisition request to a data proxy instance according to a preset policy, which will not be repeated here.

Returning to FIG. 1, the data acquisition method further includes step S104: acquiring a first data block corresponding to the data acquisition request from the first cache area.

In some embodiments, the data acquisition request includes a mark of the first data source and a data cursor. The mark of the first data source is used to indicate the source of the data requested by the client, and the data cursor is used to mark the starting position of acquiring data, that is, the starting point of the data requested this time, and the mark of the first data source and data cursors are used to find the data requested by the client.

In some embodiments, the step S104 includes:

determining a first cache area according to the mark of the first data source;

In this embodiment, the first cache area includes a mark of the first data source, indicating that the data stored in the first cache area is the data generated by the first data source, and the data cursor is used to mark the data requested by the data request. For the acquired data, the data cursor includes a flag of starting data or a timestamp of the first data block.

The obtaining the first data block corresponding to the data cursor from the first cache area according to the data cursor includes: querying the first data block with the flag of the start data from the first cache Or query the first data block with the timestamp of the first data block from the first cache.

When the data cursor is the mark of starting data, the first cache area storing the data of the first data source is queried through the mark of the first data source, and then the first data block to be acquired this time is queried through the mark of starting data. For example, in a live broadcast scenario, each message has a globally unique number, and the number of the message can be used as a data cursor. In one embodiment, the data cursor is a flag of the first data in the first data block, and the flag can be used as an index of the first data block; or, the data cursor can also be the last acquired data The identifier of the last data in the data block acquired at the time, through which the last acquired data block can be queried, and the last acquired data block can be used to determine the data block to be acquired this time, that is, the last acquisition The first data block after the data block is the first data block to be acquired this time.

When the data cursor is the timestamp of the first data block, the first cache area storing the data of the first data source is queried through the mark of the first data source, and then the time stamp of the first data block is used to query the data that needs to be obtained this time. the first data block. For example, in the live broadcast scenario, when the message broker instance obtains the message in the live broadcast room, the server generates the data block of the message and stamps the data block with a timestamp; when the message broker instance receives the message acquisition request, according to the Query the timestamp of the data block in the first cache from the timestamp of the data block in the message acquisition request. If the timestamp of the data block is the same as the timestamp in the message acquisition request, the message broker instance acquires the data block. Obtain the first data block corresponding to the request as a local message. In some embodiments, the time corresponding to the timestamp of the first data block in the data request is earlier than the current system time. For example, if the client sends a data acquisition request every 1 second, the current system time of the server is 10:00 :00, the timestamp of the first data block in the data acquisition request is 09:59:59, and the server reads the first data block with a timestamp of 09:59:59 from the first cache After 1 second, that is, when the system time is 10:00:01, the server receives the next data acquisition request, and the timestamp of the first data block is 10:00:00. At this time, the server starts from the first data block. Read out the first data block with a timestamp of 10:00:00 in a cache. If the time corresponding to the timestamp of the first data block is the same as the current system time, it may happen that the data at the current time has not been completely written to the first cache, and the obtained data may be incomplete. Therefore, set the The time corresponding to the timestamp of the first data block is earlier than the current system time, which can ensure that the new data generated at the current time can be completely stored in the first cache, and the data read by the server is also a complete unit of time. data generated within.

In some embodiments, the step S104 includes: if the data cursor is a default cursor, obtaining the latest first data block from the first cache area; the default cursor indicates that the data obtaining request is the The client makes a first data acquisition request for the first data source. In this optional embodiment, if the data cursor in the data acquisition request is the default cursor, it means that the data acquisition request is the first data acquisition request sent by a client, and at this time, the data is obtained from the first cache area. Get the latest first data block in it. For example, in the live broadcast scenario, the messages between the client and the server need to be highly real-time. Therefore, when the client connects to the live room and starts to obtain the messages sent by other clients in the live room, the server obtains the latest currently stored message. .

Returning to FIG. 1, the data acquisition method further includes step S105: sending the first data block to the client.

In this step, the server sends the acquired first data block to the client that sends the data request.

In some embodiments, the step S105 includes:

Generate the next data cursor;

The next data cursor and the first data block are sent to the client.

Before sending the first data block to the client, the server may generate a data cursor to be used by the client when sending a data request next time as the above-mentioned next data cursor. As mentioned above, the next data cursor may also be used. Including the sign of the next start data or the next time stamp of the first data block, wherein the sign of the next start data is the sign of the last data in the first data block obtained by the server this time, and the server is in the next After receiving the data acquisition request of the client for the second time, determine the first data block to be acquired according to the data flag; or, the next timestamp may be after the timestamp of the first data block acquired this time and separated by a specific time The timestamp of the interval, the specific interval can be appropriately set, for example, it can be the aforementioned preset time interval, unit time length, any other suitable setting value, etc., so that the next timestamp of the first data block is passed The first data block corresponding to the next timestamp in the first cache is queried.

The server sends the next data cursor to the client while sending the first data block to the client, so that the client can use the data cursor when sending a data acquisition request next time .

Through the above steps S101 to S105, the server asynchronously obtains the data generated by the first data source from the data storage area according to the first preset time interval, and caches the data in the first cache area. The data in the first cache area It is stored in the form of data blocks, so that every time a client sends a data acquisition request, the data blocks sent by the server are the same data blocks, that is, the segments of the data blocks obtained by the client are the same, unlike In the prior art, the data obtained by each client may be different, so that the server does not need to obtain data from the storage area separately for each client, but can obtain the data requested by the client from the cache. The number of times of fetching data from the data storage area is increased, and the data hit rate and CPU usage efficiency when fetching data are improved.

The above embodiment discloses a data acquisition method, the data acquisition method includes: acquiring a first data source, wherein the first data source is a data source that satisfies a preset condition; Obtain the data generated by the first data source and cache the data in the first cache area; wherein, the data in the first cache area is stored in the form of data blocks, and the data blocks include data generated in a unit time; receive a data acquisition request sent by the client; acquire the first data block corresponding to the data acquisition request from the first cache area; send the first data block to the client end. The above method solves the technical problem of wasting computing resources caused by acquiring data by periodically reading the data of the data source into the cache area, and obtaining data from the cache area to the client after receiving the data acquisition request.

FIG. 3 is a schematic diagram of an application scenario of the data acquisition method described in the embodiment of the present disclosure. As shown in FIG. 3, it is a live broadcast scene, wherein 301 is a live broadcast server and terminal devices 3021 to 302n; wherein a server is installed in the live broadcast server 301, and a client is installed in the terminal device, and a user can create a live broadcast room through the client Live broadcast as the host, or enter the live broadcast room through the client, and watch the live broadcast as a viewer; during the live broadcast, the client sends messages to other clients in the live broadcast room. The messages sent by the client are all sent to the data storage area of the server, and then the client polls the server at 1-second intervals to obtain messages sent by other clients in the live room. A server is installed in the server 301, wherein the server includes a load balancing layer and a message broker layer, wherein the load balancing layer is used to receive a message acquisition request sent by the client through polling, that is, the same client is received every second The sent message acquisition request is then randomly sent to a message broker instance in the message broker layer; the message broker instance in the message broker layer is used to process the message acquisition request and acquire a message corresponding to the message acquisition request. There are multiple message broker instances, and each message broker instance can independently execute the above data acquisition method, that is, the message broker instance obtains a high-volume live broadcast room, such as a live broadcast room with more than 1,000 people online, or the number of messages generated per minute. More than 1000 live broadcast rooms and so on. After determining the hot live broadcast room, each message broker instance asynchronously acquires the data generated in the live broadcast room in the last second from the data storage area every second, and divides the data into 1 second time length. The data block is stored in the cache area of the message broker instance. Each message broker instance corresponds to a cache area. The cache area described in Figure 3 is the cache area of a message broker instance, including 3 seconds. message, where the data block at 10:00:00 includes messages 0, 1, and 2; the data block at 10:10:01 includes

messages

3, 4, 5, 6, and 7; the data block at 10:10:02 Contains

messages

8, 9, 10, 11. In this application scenario, the time interval for client polling, the time interval for message broker instances to obtain messages, and the time length for dividing data blocks are the same, so that all clients obtain the same data block segmentation each time. After a message acquisition request is sent to the message broker instance, the message broker instance parses the data cursor, which includes the timestamp of the data block to be acquired this time. In order to complete the message reading, the timestamp of the data block in the data cursor is the first 1 second of the current system time, for example, the current time is 10:10:01. When a message acquisition request is received, the timestamp is parsed from the data cursor. It is 10:10:00. At this time, the message broker instance reads all messages from the cache block with a timestamp of 10:10:00, that is, messages 0, 1, and 2 in Figure 3, and marks it in the returned data cursor. The timestamp is 10:10:01; when the next message acquisition request arrives at the server again, the time should be after 10:10:01 (because the client polls every second), then the timestamp is 10:10 All messages in the cache block of :01 are returned to the client, namely

messages

3, 4, 5, 6, and 7 in Figure 3.

Due to the above mechanism, the data cursor in each client's message acquisition request is the timestamp in the corresponding data block every 1 second (the first 1 second of the system time), that is, the data cursors of all message acquisition requests are processed. In order to align, the returned messages are also separated by blocks, that is to say, the data segments obtained by each client are the same. Cache the compressed message, which can greatly improve the cache hit rate; and because each compressed message can be forwarded to multiple clients, it is not necessary to go to the data storage area to obtain the message and perform the compression operation for each message acquisition request. Therefore, the CPU consumption due to compression can be greatly reduced.

In the above, although the steps in the above method embodiments are described in the above order, it should be clear to those skilled in the art that the steps in the embodiments of the present disclosure are not necessarily executed in the above order, and may also be performed in reverse order, parallel, interleaved and other steps are performed in other order, and, on the basis of the above steps, those skilled in the art can also add other steps, these obvious modifications or equivalent replacement modes should also be included within the protection scope of the present disclosure, and will not be repeated here. .

FIG. 4 is a schematic structural diagram of an embodiment of a data acquisition apparatus according to an embodiment of the present disclosure. As shown in FIG. 4 , the apparatus 400 includes: a data source obtaining module 401, a data buffering module 402, a request receiving module 403, a data block obtaining module 404 and a data block sending module 405. in,

A data source acquisition module 401, configured to acquire a first data source, wherein the first data source is a data source that satisfies a preset condition;

A data cache module 402, configured to acquire data generated by the first data source from a data storage area at preset time intervals and cache the data in the first cache area; wherein, the first cache area The data is stored in the form of data blocks, and the data blocks include the data generated in unit time;

A request receiving module 403, configured to receive a data acquisition request sent by the client;

A data block obtaining module 404, configured to obtain the first data block corresponding to the data obtaining request from the first cache area;

A data block sending module 405, configured to send the first data block to the client.

In some embodiments, the data caching module 402 is further configured to:

The data block is cached in the first cache area.

In some embodiments, the data block obtaining module 404 is further configured to:

determining a first cache area according to the mark of the first data source;

In some embodiments, the data caching module 402 is further configured to:

obtaining data generated by the first data source;

compressing the data to obtain compressed data;

The compressed data is cached in the first cache area.

In some embodiments, the data block sending module 405 is further configured to:

Generate the next data cursor;

The next data cursor and the first data block are sent to the client.

The apparatus shown in FIG. 4 may execute the method of the embodiment shown in FIG. 1 . For the parts not described in detail in this embodiment, reference may be made to the related description of the embodiment shown in FIG. 1 . For the execution process and technical effect of the technical solution, refer to the description in the embodiment shown in FIG. 1 , which will not be repeated here.

Referring next to FIG. 5 , it shows a schematic structural diagram of an electronic device 500 suitable for implementing an embodiment of the present disclosure. Terminal devices in the embodiments of the present disclosure may include, but are not limited to, such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablets), PMPs (portable multimedia players), vehicle-mounted terminals (eg, mobile terminals such as in-vehicle navigation terminals), etc., and stationary terminals such as digital TVs, desktop computers, and the like. The electronic device shown in FIG. 5 is only an example, and should not impose any limitation on the function and scope of use of the embodiments of the present disclosure.

As shown in FIG. 5 , an electronic device 500 may include a processing device (eg, a central processing unit, a graphics processor, etc.) 501 that may be loaded into random access according to a program stored in a read only memory (ROM) 502 or from a storage device 508 Various appropriate actions and processes are executed by the programs in the memory (RAM) 503 . In the RAM 503, various programs and data required for the operation of the electronic device 500 are also stored. The processing device 501, the ROM 502, and the RAM 503 are connected to each other through a bus 504. An input/output (I/O) interface 505 is also connected to bus 504 .

Typically, the following devices may be connected to the I/O interface 505: input devices 506 including, for example, a touch screen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; including, for example, a liquid crystal display (LCD), speakers, vibration An output device 507 such as a computer; a storage device 508 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 509 . Communication means 509 may allow electronic device 500 to communicate wirelessly or by wire with other devices to exchange data. While FIG. 5 shows electronic device 500 having various means, it should be understood that not all of the illustrated means are required to be implemented or provided. More or fewer devices may alternatively be implemented or provided.

In particular, according to embodiments of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a non-transitory computer readable medium, the computer program containing program code for performing the method illustrated in the flowchart. In such an embodiment, the computer program may be downloaded and installed from the network via the communication device 509, or from the storage device 508, or from the ROM 502. When the computer program is executed by the processing apparatus 501, the above-mentioned functions defined in the methods of the embodiments of the present disclosure are executed.

It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the above two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or a combination of any of the above. More specific examples of computer readable storage media may include, but are not limited to, electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable Programmable read only memory (EPROM or flash memory), fiber optics, portable compact disk read only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing. In this disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the present disclosure, however, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave with computer-readable program code embodied thereon. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device . Program code embodied on a computer readable medium may be transmitted using any suitable medium including, but not limited to, electrical wire, optical fiber cable, RF (radio frequency), etc., or any suitable combination of the foregoing.

In some embodiments, the client and server can use any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol) to communicate, and can communicate with digital data in any form or medium Communication (eg, a communication network) interconnects. Examples of communication networks include local area networks ("LAN"), wide area networks ("WAN"), the Internet (eg, the Internet), and peer-to-peer networks (eg, ad hoc peer-to-peer networks), as well as any currently known or future development network of.

The above-mentioned computer-readable medium may be included in the above-mentioned electronic device; or may exist alone without being assembled into the electronic device.

The above computer readable medium carries one or more programs, when the one or more programs are executed by the electronic device, the electronic device causes the electronic device to: execute the data acquisition method in the above embodiment.

Computer program code for performing operations of the present disclosure may be written in one or more programming languages, including but not limited to object-oriented programming languages—such as Java, Smalltalk, C++, and This includes conventional procedural programming languages - such as the "C" language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (eg, using an Internet service provider through Internet connection).

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code that contains one or more logical functions for implementing the specified functions executable instructions. It should also be noted that, in some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It is also noted that each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented in dedicated hardware-based systems that perform the specified functions or operations , or can be implemented in a combination of dedicated hardware and computer instructions.

The units involved in the embodiments of the present disclosure may be implemented in a software manner, and may also be implemented in a hardware manner. Among them, the name of the unit does not constitute a limitation of the unit itself under certain circumstances.

The functions described herein above may be performed, at least in part, by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: Field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), Systems on Chips (SOCs), Complex Programmable Logical Devices (CPLDs) and more.

In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in connection with the instruction execution system, apparatus or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media may include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media would include one or more wire-based electrical connections, portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), fiber optics, compact disk read only memory (CD-ROM), optical storage, magnetic storage, or any suitable combination of the foregoing.

According to one or more embodiments of the present disclosure, there is provided an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores data that can be accessed by the at least one processor Instructions executed by at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform any one of the data acquisition methods of the foregoing first aspect.

According to one or more embodiments of the present disclosure, a non-transitory computer-readable storage medium is provided, wherein the non-transitory computer-readable storage medium stores computer instructions for causing a computer to execute the foregoing Any one of the data acquisition methods in the first aspect.

The above description is merely a preferred embodiment of the present disclosure and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of the disclosure involved in the present disclosure is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, and should also cover, without departing from the above-mentioned disclosed concept, the technical solutions formed by the above-mentioned technical features or Other technical solutions formed by any combination of its equivalent features. For example, a technical solution is formed by replacing the above features with the technical features disclosed in the present disclosure (but not limited to) with similar functions.

Claims

A data acquisition method, including:

acquiring a first data source, wherein the first data source is a data source that satisfies a preset condition;

Acquire the data generated by the first data source from the data storage area according to the first preset time interval and cache the data in the first cache area; wherein, the data in the first cache area is divided into data blocks It is stored in the form of data block, and the data block includes the data generated in unit time;

Receive the data acquisition request sent by the client;

Acquire the first data block corresponding to the data acquisition request from the first cache area;

Sending the first data block to the client.
The data acquisition method according to claim 1, wherein the preset condition comprises: the number of clients connected to the data source is greater than the first threshold; or, the amount of data generated by the data source within the preset time is greater than second threshold.
The data acquisition method according to claim 1, wherein the data generated by the first data source is acquired from a data storage area according to a first preset time interval and the data is cached in the first cache area ,include:

In response to the elapse of a first preset time interval, acquiring data generated by the first data source during the duration of the first preset time interval from the data storage area; wherein the data includes a data timestamp;

Organize data with data timestamps within the same unit of time into data blocks;

The data block is cached in the first cache area.
The data acquisition method according to claim 1, wherein the data acquisition request includes a mark of the first data source and a data cursor, and the acquisition from the first cache area corresponds to the data acquisition request The first block of data includes:

determining a first cache area according to the mark of the first data source;

The first data block corresponding to the data cursor is acquired from the first cache area according to the data cursor.
The data acquisition method of claim 4, wherein the data cursor includes a flag of starting data or a time stamp of the first data block.
The data acquisition method according to claim 5, wherein the acquiring the first data block corresponding to the data cursor from the first cache area according to the data cursor comprises:

The first data block with the flag of the start data is queried from the first cache or the first data block with the timestamp of the first data block is queried from the first cache.
The data acquisition method according to claim 4, wherein the acquiring the first data block corresponding to the data acquisition request from the first cache area comprises:

If the data cursor is the default cursor, acquire the latest first data block from the first cache area; the default cursor indicates that the data acquisition request is the first request by the client for the first data source data acquisition request.
The data acquisition method according to claim 1, wherein the acquiring the data generated by the first data source and caching the data in the first cache area comprises:

obtaining data generated by the first data source;

compressing the data to obtain compressed data;

The compressed data is cached in the first cache area.
The data acquisition method of claim 1, wherein the sending the first data block to the client comprises:

Generate the next data cursor;

The next data cursor and the first data block are sent to the client.
A data acquisition device, comprising:

a data source acquisition module, configured to acquire a first data source, wherein the first data source is a data source that satisfies a preset condition;

A data cache module, configured to acquire the data generated by the first data source from the data storage area at preset time intervals and cache the data in the first cache area; wherein, the data in the first cache area The data is stored in the form of data blocks, and the data blocks include data generated in a unit time;

The request receiving module is used to receive the data acquisition request sent by the client;

a data block obtaining module, configured to obtain the first data block corresponding to the data obtaining request from the first cache area;

A data block sending module, configured to send the first data block to the client.
An electronic device comprising:

memory for storing computer-readable instructions; and

A processor for executing the computer-readable instructions to cause the electronic device to implement the method according to any one of claims 1-9.
A computer-readable storage medium for storing computer-readable instructions, which, when executed by a computer, cause the computer to implement the method of any one of claims 1-9.
A computer program product carrying program code comprising instructions, when executed by a computer, that cause the computer to perform the method of any of claims 1-9.
A computer program comprising instructions which, when executed by a computer, cause the computer to perform the method of any of claims 1-9.