CN116932594A

CN116932594A - Data acquisition method

Info

Publication number: CN116932594A
Application number: CN202310890408.XA
Authority: CN
Inventors: 张慧超; 王星芳
Original assignee: Beijing Jingwei Hirain Tech Co Ltd
Current assignee: Beijing Jingwei Hirain Tech Co Ltd
Priority date: 2023-07-19
Filing date: 2023-07-19
Publication date: 2023-10-24

Abstract

The application discloses a data acquisition method, and belongs to the technical field of data processing. And acquiring a query request carrying the data identifier, and searching target data corresponding to the data identifier from the cache region. If the target data is found, the target data is returned from the cache area in response to the query request. If the target data is not found, the data batch containing the target data stored in the back-end equipment is copied to the buffer area preferentially. And then responding to the query request, and returning the target data from the cache area. It can be seen that the target data can be acquired by the head-end equipment. Before the target data is acquired, part of the target data is stored in a buffer area of the front-end equipment. When the target data of the specific data identification is acquired, the target data can be preferentially searched from the cache region, whether the target data is acquired from the back-end equipment or not is determined based on the searching result, frequent interaction with the back-end equipment is not needed, and efficiency is improved.

Description

Data acquisition method

Technical Field

The application belongs to the technical field of data processing, and particularly relates to a data acquisition method.

Background

In general, before the application of models such as a deep learning model and a computer vision model, and in the application process, the models need to be trained through data samples, so that the performance of the models is improved. For supervised learning, after the data sample is collected, the user is often required to label the data sample, so that the model can be trained based on the labeled data sample.

The user typically annotates the data sample based on a browser configured within the front-end device. However, since the cache space of the browser is small, typically only 3 to 5 Megabytes (MB), it is difficult to store a large number of data samples, and thus the data samples are often stored in the storage component of the back-end device. Thus, a user often needs to obtain a data sample before annotating the data sample.

In the prior art, when acquiring a data sample, a front-end device generally sends a request carrying a data identifier to a back-end device, so as to acquire the data sample from the back-end device.

Obviously, each time the user obtains a data sample, a request needs to be sent to the back-end device, and the back-end device waits for the data sample to be returned. Moreover, since a certain time is often required for the data sample to be sent from the back-end device to the front-end device, in the prior art, the efficiency of acquiring the data sample by the user is low.

Disclosure of Invention

The embodiment of the application provides a data acquisition method, which can improve the efficiency of acquiring target data.

In one aspect, an embodiment of the present application provides a data acquisition method, including:

acquiring a query request carrying a data identifier;

Searching target data corresponding to the data identification from the cache area;

under the condition that the target data is determined to be searched, responding to the query request, and returning the target data;

and copying the data batch containing the target data stored at the back-end equipment into the cache area under the condition that the target data is not found, and returning the target data stored in the cache area in response to the query request.

Optionally, the data batch carries a batch identifier;

copying the data batch containing the target data stored at the back-end equipment into the cache area, and responding to the query request, returning the target data from the cache area, wherein the method comprises the following steps:

according to the data identification, determining a batch identification corresponding to the data identification;

according to the batch identifier, acquiring a data batch corresponding to the batch identifier from a back-end device associated with the front-end device, wherein the data batch comprises the target data;

storing the data batch into the cache area;

and responding to the query request, and returning the target data stored in the cache area.

Optionally, the data identifiers are sequentially increased from the first value, the batch identifiers are sequentially increased from the second value, and the data batch with smaller batch identifiers preferentially configures the data with smaller target data;

the step of determining the batch identifier corresponding to the data identifier according to the data identifier comprises the following steps:

according to the data identifier, the first numerical value, the second numerical value and the B, determining a batch identifier of a data batch containing target data corresponding to the data identifier through the following formula:

wherein Y is a batch identifier, X is a data identifier, C is a first value, D is a second value, and N is the maximum value of the data identifier.

Optionally, according to the lot identifier, acquiring, from a back-end device associated with the front-end device, a data lot corresponding to the lot identifier includes:

sending a target request for acquiring the data batch corresponding to the batch identifier to a back-end device;

receiving and storing the data batch returned by the back-end equipment;

and deleting H data batches from the data batches stored in the buffer area according to the first-in first-out sequence under the condition that the number of the data batches stored in the buffer area is larger than B, wherein H is a positive integer and H is smaller than B.

Optionally, the data batch carries a status identifier, where the status identifier at least includes cached, in-request, and uncached;

the sending, to the back-end device, a target request for obtaining a data batch corresponding to the batch identifier, including:

acquiring a status identifier of the batch of data corresponding to the batch identifier from the back-end equipment;

waiting for the backend device to send the data batch if the status identification is determined to be requesting;

and under the condition that the state identifier is not cached, sending a target request for acquiring the data batch corresponding to the batch identifier to the back-end equipment.

the method further comprises the steps of:

stopping receiving the data batch sent by the back-end equipment under the condition that the state identification of the data batch is determined to be in a request;

and restoring the receiving of the data batch which is stopped to be received under the condition that the data batch which contains the target data corresponding to the data identification is determined to be stored.

Optionally, the data batch carries a status identifier, where the status identifier at least includes cached, in-request and uncached, and the status identifier of the data batch in the initial state is uncached;

The method further comprises the steps of:

under the condition that a target request carrying a batch identifier is sent to the back-end equipment, a state modification request carrying the batch identifier is sent to the back-end equipment, and the state identifier carried by the data batch corresponding to the batch identifier is modified to be a request;

under the condition that the data batch is confirmed to be stored in the buffer area, a state modification request carrying the batch identification is sent to the back-end equipment, and the state identification of the data batch is modified to be cached;

and under the condition that the data batch in the cache area is deleted, sending a state modification request carrying the batch identifier to the back-end equipment, and modifying the state identifier of the data batch into uncached.

Optionally, the data batch carries a batch identifier, and the method further includes:

based on a first-in first-out principle, determining sequence identifiers of the data batches in the cache region respectively;

according to the data identification of the target data in the data batch corresponding to the data identification, determining a first demarcation point and a second demarcation point through the following formula:

wherein K is ₁ K is the first demarcation point ₂ M is the smallest data mark in the data batch, and L is the largest data mark in the data batch;

Upon determining that the data identity is in interval [ M, K ] ₁ ]In the case, determining a target batch identifier according to the batch identifier of the data batch, wherein the target batch identifier is the difference between the batch identifier and a third numerical value;

upon determining that the data identification is in interval [ K ] ₂ ，L]When the data batch is in the process, determining a target batch identifier according to the batch identifier of the data batch, wherein the target batch identifier is the sum of the batch identifier and a third numerical value;

and acquiring the data batch with the same batch identification as the target batch identification.

In another aspect, an embodiment of the present application provides another data acquisition method, including:

determining the target quantity of the target data storable in the buffer area according to the size of the buffer area configured in the service area of the front-end equipment;

determining a total number of target data;

determining the total number of the data batches according to a preset second threshold and the total number, wherein the second threshold is the number of target data which can be contained in the data batches at most through the following formula:

wherein P is the total number of data batches, Q is the total number of target data, and B is a second threshold;

determining a first threshold according to the target number and the second threshold, wherein the first threshold is the number of data batches which can be stored in the cache area at most and is represented by the letter A;

And sending the A data batches to the front-end equipment, so that the front-end equipment stores the A data batches into the cache area so as to acquire target data from the cache area.

receiving a target request carrying a batch identifier sent by the front-end equipment;

and responding to the target request, and sending the data batch corresponding to the batch identification to the front-end equipment.

In another aspect, an embodiment of the present application provides another data acquisition apparatus, including:

the acquisition unit is used for acquiring a query request carrying a data identifier;

the searching unit is used for searching target data corresponding to the data identifier from the cache area;

the first determining unit is used for responding to the query request and returning the target data under the condition that the target data is determined to be found;

and the second determining unit is used for copying the data batch containing the target data stored at the back-end equipment into the cache area under the condition that the target data is not found, and returning the target data stored in the cache area in response to the query request.

In another aspect, an embodiment of the present application provides a data acquisition apparatus, including:

the first determining unit is used for determining the target quantity of the storable target data of the buffer area according to the size of the buffer area configured in the service area of the front-end equipment;

a second determining unit configured to determine a total number of target data;

the calculating unit is configured to determine the total number of data batches according to a preset second threshold and the total number, where the second threshold is the number of target data that can be included in the data batches at most by the following formula:

a third determining unit, configured to determine a first threshold according to the target number and the second threshold, where the first threshold is a number of data batches that can be stored in the buffer at most, and is denoted by a letter a;

and the storage unit is used for sending the A data batches to the front-end equipment, so that the front-end equipment stores the A data batches into the cache area so as to acquire target data from the cache area.

In still another aspect, an embodiment of the present application provides a data acquisition apparatus, including:

A processor and a memory storing computer program instructions;

the processor executes the computer program instructions to implement the data acquisition method according to the above aspect, and it should be noted that, in one or more embodiments of the present application, the data acquisition device is a front-end device or a back-end device.

In yet another aspect, an embodiment of the present application provides a computer storage medium having stored thereon computer program instructions which, when executed by a processor, implement a data acquisition method according to any one of the above aspects.

In yet another aspect, an embodiment of the present application provides a computer program product, where instructions in the computer program product, when executed by a processor of an electronic device, cause the electronic device to implement a data acquisition method according to any one of the above aspects.

The data acquisition method, the device, the equipment and the computer storage medium acquire target data through the front-end equipment. Before the target data is acquired, part of the target data is stored in a buffer area of the front-end equipment. When the target data of the specific data identification is acquired, the target data can be preferentially searched from the cache region, whether the target data is acquired from the back-end equipment or not is determined based on the searching result, frequent interaction with the back-end equipment is not needed, and efficiency is improved.

Drawings

In order to more clearly illustrate the technical solution of the embodiments of the present application, the drawings that are needed to be used in the embodiments of the present application will be briefly described, and it is possible for a person skilled in the art to obtain other drawings according to these drawings without inventive effort.

FIG. 1 is a flow chart of a data acquisition method according to an embodiment of the present application;

FIG. 2 is a schematic diagram of a change in status identification provided by one embodiment of the present application;

FIG. 3 is a schematic diagram of data batch aliquots provided by one embodiment of the present application;

FIG. 4 is a flowchart of a data acquisition method according to another embodiment of the present application;

FIG. 5 is a schematic diagram of data lot partitioning according to one embodiment of the present application;

FIG. 6 is a schematic diagram of a data acquisition device according to an embodiment of the present application;

FIG. 7 is a schematic diagram of a data acquisition device according to another embodiment of the present application;

fig. 8 is a schematic structural diagram of a data acquisition device according to an embodiment of the present application.

Detailed Description

Features and exemplary embodiments of various aspects of the present application will be described in detail below, and in order to make the objects, technical solutions and advantages of the present application more apparent, the present application will be described in further detail below with reference to the accompanying drawings and the detailed embodiments. It should be understood that the particular embodiments described herein are meant to be illustrative of the application only and not limiting. It will be apparent to one skilled in the art that the present application may be practiced without some of these specific details. The following description of the embodiments is merely intended to provide a better understanding of the application by showing examples of the application.

It is noted that relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising … …" does not exclude the presence of other like elements in a process, method, article or apparatus that comprises the element.

As described in the background, in general, since a user marks a data sample based on a browser, the memory of the browser is often small, and it is difficult to store a large amount of data samples, the data samples are often stored in a storage component of the back-end device. Thus, the user typically obtains a data sample to be annotated from the backend device during the annotation process. However, the browser is usually disposed in the front-end device, and a certain time is often required for the front-end device to interact with the back-end device. Obviously, in the prior art, the user frequently obtains the behavior of the data sample to be marked from the back-end equipment, so that the efficiency of obtaining the data is reduced, and the efficiency of marking the data sample is further reduced.

In order to solve the problems in the prior art, the embodiment of the application provides a data acquisition method. The following first describes a data acquisition method provided by an embodiment of the present application.

Fig. 1 is a flow chart of a data acquisition method according to an embodiment of the present application. As shown in fig. 1, the data acquisition method provided by the embodiment of the application includes the following steps: s101 to S104.

S101: and acquiring a query request carrying the data identifier.

In one or more embodiments of the application, the data acquisition method may be performed by a head-end. Of course, the electronic device may be a computer, a mobile phone, a tablet computer, etc., and the electronic device is specifically a device, which is not limited herein, and may be set as required. It should be noted that, there is a communication connection between the front-end device and the back-end device, and the front-end device may perform data interaction with the back-end device through the communication connection.

In order to reduce the number of data interaction between the front-end equipment and the back-end equipment and improve the data acquisition efficiency, the front-end equipment is configured with a buffer area, and the buffer area has a certain storage space. The buffer area is different from the buffer space in the browser, and the data capacity of the buffer area is larger than the buffer space in the browser, but the buffer area is also limited, that is, the buffer area can store target data of certain data, but can not store all target data. For example, the size of the target data is 3MB, the size of the buffer space in the browser is 5MB, the size of the buffer area is 200MB, and the total amount of the target data is 1000. Therefore, the cache space of the browser can only store 1 target data, and each time the target data is acquired, data interaction with the back-end equipment is required. And 66 target data can be stored in the buffer area, and after the front-end equipment and the back-end equipment interact once, 66 target data can be obtained without frequently interacting with the back-end equipment.

The buffer area may be a partial area in the memory of the front-end device, and the data in the buffer area are all stored in a List (List). The buffer may also be a partial area in a storage component connected to the front end. The buffer area is specifically a region, the application is not limited, and the buffer area can be set according to the requirement. And, in determining the buffer area, it can be determined by the user by writing a service code. That is, in one or more embodiments of the present application, the buffer is a buffer corresponding to the memory of the service code of the front-end device.

In one or more embodiments of the present application, a certain amount of target data may be preferentially stored in the buffer of the front-end device. In addition, since the front-end device needs a certain time for data interaction with the back-end device, when a certain amount of unobtained (not labeled) target data still exist in the buffer area, the front-end device can perform data interaction with the back-end device in an asynchronous communication mode, namely, store part of target data in the buffer area under the condition of not influencing the acquisition of the target data.

Continuing with the above example, the front-end device may store 66 target data, and the front-end device may use the stored 66 target data as target data for two batches, the first batch containing 33 target data and the second batch containing 33 target data. When the front-end equipment determines that all the 33 target data contained in the first batch are acquired, the front-end equipment can delete the first batch, namely delete the 33 acquired target data, and perform data interaction with the back-end equipment in an asynchronous request mode to acquire target data of a third batch, wherein the third batch also contains the 33 target data. Therefore, the influence of data interaction between the front-end equipment and the back-end equipment on data acquisition is further reduced, and the data acquisition efficiency is improved.

Specifically, in one or more embodiments of the present application, before the front-end device acquires the target data, the front-end device may perform data interaction with the back-end device preferentially, and receive a data batches sent by the back-end device, where each data batch includes at most B target data. A and B are positive integers, A is smaller than a preset first threshold value, and B is smaller than a preset second threshold value. The first threshold and the second threshold are determined according to the storage capacity of the buffer area and the size of the target data, and the memory space occupied by the a data batches is smaller than the storage capacity of the buffer area. It should be noted that, each target data carries a data identifier, which is used for distinguishing from other target data, that is, the data identifier carried by each target data is different from the data identifiers carried by other target data.

Wherein in one or more embodiments of the present application, the target data is an image frame that constitutes a video image, and the data identification of the target data is a frame number of the target data in the video image.

The front-end equipment can respond to the operation of a user, determine the data identification of target data to be acquired, and determine a query request carrying the data identification. The front-end equipment can display the data identification of a plurality of target data to be acquired through a configured browser or other software, and respond to the selection operation of a user to determine the data identification selected by the user as the data identification of the target data to be acquired.

In this way, the front-end device can determine the data identifier of the target data to be acquired, so as to acquire the target data based on the data identifier.

S102: and searching target data corresponding to the data identification from the cache area.

In one or more embodiments of the present application, after the front-end device determines a search request carrying a data identifier, the front-end device may search for target data corresponding to the data identifier from the cache.

Specifically, the front-end device may search, according to the data identifier carried by the query request, target data corresponding to the data identifier from the cache region. Because the method for searching the data based on the identification of the data is mature in the prior art, how to search is specific, and for brevity, the application is not described here again.

By adopting the mode, the electronic equipment can search the data to be acquired in the buffer area preferentially without directly carrying out data interaction with the back-end equipment, and the data acquisition efficiency is improved.

S103: and under the condition that the target data is determined to be searched, returning the target data in response to the query request.

In one or more embodiments of the present application, when the front-end device finds the target data corresponding to the data identifier in the buffer, the target data may be acquired.

Specifically, in the case that the front-end device determines that the target data matching the data identifier is found in the buffer, the front-end device may return the target data, and display the target data to the user, so that the user obtains the target data.

By adopting the mode, the front-end equipment can return the target data to the user under the condition of not carrying out data interaction with the back-end equipment, so that the user can acquire the target data.

S104: and copying the data batch containing the target data stored at the back-end equipment into the cache area under the condition that the target data is not found, and returning the target data stored in the cache area in response to the query request.

In one or more embodiments of the present application, if the front-end device does not find the target data matching the data identifier in the cache region, the front-end device may perform data interaction with the back-end device based on the data identifier, so as to store the data batch including the target data in the cache region of the front-end device, and then return the target data.

Specifically, the front-end device may determine, according to the data identifier, a lot identifier corresponding to the data identifier when it is determined that the target data is not found from the cache region. And according to the batch identifier, acquiring a data batch corresponding to the batch identifier from a rear-end device associated with the front-end device, and storing the data batch in the buffer. And finally, responding to the query request, and returning the target data stored in the cache area.

Each data batch carries a batch identifier, and when the front-end equipment determines the batch identifier corresponding to the data identifier, the front-end equipment can determine the batch identifier according to the rule corresponding to the data identifier and the batch identifier. Or when determining the data identifiers, each data identifier can be determined to contain the batch identifier of the data batch to which the target data belongs, and the front-end equipment can directly determine the batch identifier according to the data identifier.

In addition, in one or more embodiments of the present application, the data identification of the target data, as well as the lot identification of the data lot, may be user-set. Therefore, in one or more embodiments of the present application, before the front-end device determines the lot identifier according to the rule corresponding to the data identifier and the lot identifier, the rule corresponding to the data identifier and the lot identifier needs to be set by itself.

Thus, in one or more embodiments of the application, the head-end equipment may determine that the data identification is incremented sequentially from a first value and that the lot identification is incremented sequentially from a second value. And the data batch priority configuration data with the smaller batch identification identifies the smaller target data until the number of target data contained in the data batch reaches B.

Wherein B is the number of data batches up to the target data contained. The first value and the second value are positive integers. For example, the first and second values are 1, b is 100, and the total number of target data is 432. Thus, the data identification of the target data is 1, 2, 3, … …, 432. The lot identities are 1, 2, 3, 4 and 5. The target lot with lot identification 1 contains the target data with the data identifications 1, 2, 3, … …, 100, the target lot with the lot identification 2 contains the target data with the data identifications 101, 102, 103, … …, 200, the target lot with the lot identification 3 contains the target data with the data identifications 201, 202, 203, … …, 300, the target lot with the lot identification 4 contains the target data with the data identifications 301, 302, 303, … …, 400, and the target lot with the lot identification 5 contains the target data with the data identifications 401, 402, 403, … …, 432.

Therefore, the front-end equipment can determine the batch identification of the data batch containing the target data corresponding to the data identification according to the data identification, the first value, the second value and the B through the following formula:

wherein Y is a lot identity, X is a data identity, C is a first value, D is a second value, N is a maximum target identity, and in one or more embodiments of the application, when C is 1, N is also the total number of target data. The rule between the data identifier and the batch identifier is determined based on the mode, so that the front-end equipment can determine the batch identifier corresponding to the data identifier based on the rule after determining the data identifier of any target data. The efficiency of acquiring the target data is further improved.

In addition, in one or more embodiments of the present application, when the front-end device acquires the data batch corresponding to the batch identifier from the back-end device, the following manner may be adopted for acquisition.

The front-end device may send a target request carrying a lot identification to the back-end device. The target request is used for acquiring the data batch corresponding to the batch identifier. And receiving the data batch returned by the back-end equipment in response to the target request. The data batch is stored in the buffer.

And, since the number of data batches that can be stored in the buffer is limited, when the number of data batches stored in the buffer is greater than a, the front-end device can delete a portion of the data batches in the buffer. Specifically, when the front-end device determines that the number of data batches stored in the buffer is greater than a, the front-end device may delete H data batches from the data batches stored in the buffer according to a first-in first-out order, that is, according to the order in which the data batches are stored in the buffer, where H is a positive integer and H is less than a.

In addition, in one or more embodiments of the present application, since the front-end device needs a certain time to interact with the back-end device, the user has autonomy and may not wait. That is, when the user determines to acquire the target data corresponding to a certain data identifier and the target data is not yet returned to the user, the user may acquire the target data corresponding to other data identifiers. Moreover, the target data acquired twice by the user may correspond to the same data batch, which easily causes the front-end device to request the back-end device for the same data batch twice.

For example, the user determines the target data corresponding to the acquired data identifier 134. The lot identifier of the data lot to which the target data belongs is 2, and then the front-end device sends a target request carrying the lot identifier 2 to the back-end device. If the user does not acquire the target data corresponding to the data identifier 134, the user determines to acquire the target data corresponding to the data identifier 158, and the lot identifier of the data lot to which the target data belongs is also 2. The head-end in turn sends a target request carrying lot identification 2 to the back-end.

Obviously, this is not reasonable. Thus, to circumvent this occurrence, the electronic device may determine that each data batch carries a status identifier that characterizes the status of the data batch. The status identification includes cached, requesting, uncached, etc. When a data batch is stored in the buffer, the status of the data batch is identified as cached. When the front-end equipment sends a target request carrying a certain batch identifier to the back-end equipment and the data batch corresponding to the batch identifier is not stored in the buffer area, the state identifier of the data batch is the request. When a certain batch of data is not stored in the buffer, and the front-end equipment does not send a target request carrying a batch identifier of the batch of data to the back-end equipment, the state identifier of the batch of data is not cached. The change of the status identifier may be implemented by a function in a computer program running in the back-end device, and since in the prior art, the related art is mature, and thus, a specific process of updating the status identifier and how the status identifier exists are not described herein for brevity.

In one or more embodiments of the present application, when the front-end device determines that the status identifier of a certain data batch needs to be updated, the status identifier may be updated by sending a status modification request to the back-end device. Specifically, by default, the status identifiers of the data batches stored in the front-end device before step S101 are cached, and the status identifiers of the other data batches are not cached.

Then, when the front-end device determines to send the target request carrying the lot identifier to the back-end device, a state modification request carrying the lot identifier may be sent to the back-end device, and the state identifier carried by the data lot corresponding to the lot identifier may be modified to be requested. And under the condition that the data batch is determined to be stored in the buffer, sending a state modification request carrying the batch identifier to the back-end equipment, and modifying the state identifier of the data batch into the cached state. And under the condition that the data batch in the buffer is deleted, sending a state modification request carrying the batch identifier to the back-end equipment, and modifying the state identifier of the data batch into uncached.

Fig. 2 is a schematic diagram of a change of state identification according to an embodiment of the present application, as shown in fig. 2. Wherein, when the front-end device initiates a target request for a certain data batch to the back-end device, the status identifier of the data batch is changed from uncached 201 to being requested 202. When the target request fails, the status identification of the data batch is changed from being requested 202 to uncached 201. When the target request is successful, the status identification of the data batch is changed from being requested 202 to cached 203. When a cached data batch is removed from the cache, the status identifier of the data batch is changed from cached 203 to uncached 201.

Therefore, when the front-end equipment sends a target request carrying a certain batch identifier to the back-end equipment, the front-end equipment can preferentially determine the state identifier of the data batch corresponding to the batch identifier, so as to determine whether to send the target request. Specifically, the front-end device may obtain, from the back-end device, the status identifier of the data lot corresponding to the lot identifier before sending, to the back-end device, a target request for obtaining the data lot corresponding to the lot identifier.

In the case where the front-end device determines that the status identifier is requesting, it may not be necessary to send the target request, and may wait for the back-end device to send the data batch. And if the front-end equipment determines that the state identifier is not cached, a target request for acquiring the data batch corresponding to the batch identifier can be sent to the back-end equipment. In the event that the head-end equipment determines that the status identification is cached, the target request may not need to be sent. By adopting the mode, the front-end equipment can reduce the occurrence of the condition of requesting the same data batch twice from the back-end equipment, and further improve the acquisition efficiency of the target data.

Further, as described in step S101, after determining that the target data of some data batches are all acquired, the front-end device may delete the data batches and request a part of the data batches from the back-end device by means of asynchronous communication. When determining the data identifier carried by the query request, the front-end device may request the data batch corresponding to the target data from the back-end device if the target data corresponding to the data identifier is not found in the buffer. Thus, the data batch requested from the back-end device by way of asynchronous communication may be the same as or may be different from the data batch requested from the back-end device based on the query request. In order to further improve the efficiency of acquiring the target data, the front-end equipment can process the target data in different ways according to different situations.

Specifically, the front-end device may determine a lot identifier corresponding to the data identifier and obtain a status identifier of the lot identifier from the back-end device when determining that the data identifier carried by the query request does not find the target data corresponding to the data identifier from the buffer.

In the event that the front-end device determines that the status identification is requesting or cached, there is no need to send a target request to the back-end device.

And in the case that the front-end equipment determines that the state identifier is not cached, a target request carrying the batch identifier can be sent to the back-end equipment. Also, in one or more embodiments of the application, the front-end device may be interacting with the back-end device as a result of the data. Thus, to further increase the efficiency of acquiring the target data, the front-end device may determine whether the front-end device is in data interaction with the back-end device. In the event that the head-end equipment determines that there is no status identification of the data batch as being requested, no further operations may be performed. When the front-end device determines that the status identifier of the data batch exists as being requested, the receiving of the data batch sent by the back-end device may be stopped. And receives a data batch associated with the query request. In the event that a determination is made that a data batch has been stored with the query request, the head-end equipment may resume receiving the data batch for which reception has ceased.

For example, the target identifier carried by the query request is 154, and the corresponding lot identifier is 2. After the front-end equipment sends a target request carrying the batch identifier 2 to the back-end equipment, the state identifier of the data batch corresponding to the batch identifier 4 is determined to be the request. That is, if it is determined that the front-end device is receiving the data lot with lot id 4, the front-end device may stop receiving the data lot with lot id 4, and return to receiving the data lot with lot id 4 after storing the data lot with lot id 2 in the buffer.

In addition, in one or more embodiments of the application, the target data contained by the data batch has a certain regularity due to certain circumstances. Therefore, after the last target data of a certain data batch is acquired, there is a certain probability that the target data included in the next data batch adjacent to the certain data batch will be acquired.

The head-end may then determine whether to store certain data batches in the buffer by.

Specifically, the front-end device may determine a target identifier carried by the query request and a data identifier corresponding to the target identifier. And determining a first demarcation point and a second demarcation point according to the data identification of each target data in the data batch corresponding to the data identification by the following formula:

Wherein K is ₁ K is the first demarcation point ₂ M is the smallest data mark in the data batch, L is the largest data mark in the data batch;

at the time of determining that the data mark is in the interval [ M, K ] ₁ ]In the case, a target lot identification is determined from the lot identification of the data lot, the target lot identification being the difference between the lot identification and a third value.

Upon determining that the data identification is in interval [ K ₂ ，L]And when the data batch is in the process, determining a target batch identifier according to the batch identifier of the data batch, wherein the target batch identifier is the sum of the batch identifier and a third numerical value. Wherein the third value is a positive integer, preferably the third value is 1.

FIG. 3 is a schematic diagram of data batch aliquots provided by one embodiment of the present application, as shown in FIG. 3. Wherein M301 is the smallest target identifier in the data batch, and L304 is the largest target identifier in the data batch. Intervals [ M, K ] ₁ ]There is 1/3 of the data identifier in the data batch, interval [ K ] ₂ ，L]There is also 1/3 of the data batchAnd (5) data identification. K (K) ₁ 302 is identified as K ₂ 303 target identification +.>If the target mark X belongs to the interval [ M, K ] ₁ ]The target lot identification may be determined as the difference between the lot identification and the third value.

Finally, in the case that the front-end equipment determines that the target lot identifier exists, the front-end equipment can send a target request carrying the target lot identifier to the back-end equipment, and the data lot with the lot identifier identical to the target lot identifier is stored in the buffer area. By adopting the mode, the front-end equipment can further improve the efficiency of acquiring the target data.

By adopting the mode, the front-end equipment can store the data batch containing the target data in the buffer area under the condition that the target data is not searched from the buffer area, and then return the target data from the buffer area.

The above is a specific implementation manner of the data acquisition method provided in the embodiment of the present application, and it can be seen that in the above embodiment, the target data can be acquired through the front-end device. Before the target data is acquired, part of the target data is stored in a buffer area of the front-end equipment. When the target data of the specific data identification is acquired, the target data can be preferentially searched from the cache region, whether the target data is acquired from the back-end equipment or not is determined based on the searching result, frequent interaction with the back-end equipment is not needed, and efficiency is improved.

The following first describes another data acquisition method provided in the embodiment of the present application.

Fig. 4 is a flow chart illustrating a data acquisition method according to an embodiment of the present application. As shown in fig. 4, the data acquisition method provided by the embodiment of the application includes the following steps: s101 to S104.

S401: and determining the target quantity of the target data which can be stored in the buffer area according to the size of the buffer area configured in the service area of the front-end equipment.

In one or more embodiments of the present application, the backend device may determine the target amount of storable target data in a buffer configured in a service area of the front-end device according to a size of the buffer and a size of the target data. Since the sizes of different target data may be different, the backend device may also estimate the target number of storable target data in the buffer according to the size of the buffer and the average value of the target data sizes.

S402: the total number of target data is determined.

In one or more embodiments of the application, the backend device may count the total number of target data stored within the storage component.

S403: determining the total number of the data batches according to a preset second threshold and the total number, wherein the second threshold is the number of target data which can be contained in the data batches at most through the following formula:

Wherein P is the total number of data batches, Q is the total number of target data, and B is a second threshold.

In one or more embodiments of the present application, the backend device may determine the total number of data batches through equation 4 based on the preset second threshold value and the total number. Wherein P is the total number of data batches, Q is the total number of target data, and B is a second threshold.

FIG. 5 is a schematic diagram of data lot partitioning according to one embodiment of the present application, as shown in FIG. 5. Wherein the Q target data are sequentially arranged in sequence, wherein the 1 st target data 501 to the B target data 502 are divided into the first data batch 505. The C-th target data 503 to Q-th target data 504 are divided into P-th data batches 506.

S404: and determining a first threshold according to the target quantity and the second threshold, wherein the first threshold is the quantity of data batches which can be stored in the buffer area at most and is indicated by a letter A.

In one or more embodiments of the present application, the backend device may determine a first threshold value according to the target number and the second threshold value, where the first threshold value is at most the number of data batches that can be stored in the buffer, and the number is denoted by the letter a.

S405: and sending the A data batches to the front-end equipment, so that the front-end equipment stores the A data batches into the cache area so as to acquire target data from the cache area.

In one or more embodiments of the present application, the electronic device may send a data batches to the front-end device at random, so that the front-end device may store a data batches in the buffer in advance, so as to obtain target data from the buffer.

In one or more embodiments of the present application, the backend device may also receive a target request carrying a lot identifier sent by the front-end device. And responding to the target request, and sending the data batch corresponding to the batch identifier to the front-end equipment.

In addition, the technical scheme of the application can acquire, store, use, process and the like the data, which accords with the relevant regulations of national laws and regulations.

Based on the data acquisition method provided by the embodiment, correspondingly, the application further provides a specific implementation mode of the data acquisition device. Please refer to the following examples.

Referring first to fig. 6, a data acquisition device provided in an embodiment of the present application includes the following units:

an obtaining unit 601, configured to obtain a query request carrying a data identifier;

a searching unit 602, configured to search target data corresponding to the data identifier from the cache area;

a first determining unit 603, configured to, in response to the query request, return the target data if it is determined that the target data is found;

and the second determining unit 604 is configured to copy, when it is determined that the target data is not found, a data batch including the target data stored at the back-end device into the cache area, and return the target data stored in the cache area in response to the query request.

According to the above embodiments, it can be seen that the data acquisition apparatus provided by the present application is capable of acquiring target data through a front-end device. Before the target data is acquired, part of the target data is stored in a buffer area of the front-end equipment. When the target data of the specific data identification is acquired, the target data can be preferentially searched from the cache region, whether the target data is acquired from the back-end equipment or not is determined based on the searching result, frequent interaction with the back-end equipment is not needed, and efficiency is improved.

As another implementation manner of the present application, in order to further improve efficiency, the apparatus may specifically further include: and a second determination subunit.

The second determining subunit is configured to determine, according to the data identifier, a batch identifier corresponding to the data identifier, obtain, according to the batch identifier, a data batch corresponding to the batch identifier from a back-end device associated with the front-end device, where the data batch includes the target data, store the data batch in the buffer area, and return, in response to the query request, the target data stored in the buffer area.

The second determining subunit is configured to determine, according to the data identifier, the first value, the second value, and the B, a lot identifier of a data lot including target data corresponding to the data identifier according to the following formula:

wherein Y is a batch identifier, X is a data identifier, C is a first value, and D is a second value.

The second determining subunit is configured to send a target request for obtaining a data batch corresponding to the batch identifier to a back-end device, receive and store the data batch returned by the back-end device, and delete H data batches from the data batches stored in the buffer according to a first-in first-out order when determining that the number of the data batches stored in the buffer is greater than B, where H is a positive integer and H is less than B.

The second determining subunit is configured to obtain, from the back-end device, a status identifier of the data batch corresponding to the batch identifier, wait for the back-end device to send the data batch when determining that the status identifier is being requested, and send, to the back-end device, a target request for obtaining the data batch corresponding to the batch identifier when determining that the status identifier is not cached.

And the second determining subunit is used for stopping receiving the data batch sent by the back-end equipment when the state identification of the data batch is determined to be in a request state, and recovering to receive the data batch stopping receiving when the data batch containing the target data corresponding to the data identification is determined to be stored.

The second determining subunit is configured to, when determining that a target request carrying a lot identifier is sent to the back-end device, send a status modification request carrying the lot identifier to the back-end device, modify a status identifier carried by a data lot corresponding to the lot identifier to be requested, when determining that the data lot is stored in the cache area, send a status modification request carrying the lot identifier to the back-end device, modify the status identifier of the data lot to be cached, and when determining that the data lot in the cache area is deleted, send a status modification request carrying the lot identifier to the back-end device, and modify the status identifier of the data lot to be uncached.

The second determining subunit is configured to determine, based on a first-in first-out principle, sequential identifiers of the plurality of data batches in the buffer, and determine, according to a data identifier of target data in a data batch corresponding to the data identifier, a first demarcation point and a second demarcation point according to the following formula:

wherein the method comprises the steps of，K ₁ K is the first demarcation point ₂ For the first demarcation point, M is the smallest data identifier in the data batch, N is the largest data identifier in the data batch, and when the data identifier is determined to be in the interval [ M, K ] ₁ ]In the case of the above, determining a target lot identification, which is a difference between the lot identification and a third value, from the lot identification of the data lot, and determining that the data identification is in the interval [ K ] ₂ ，N]And when the data batch is in the process, determining a target batch identifier according to the batch identifier of the data batch, wherein the target batch identifier is the sum of the batch identifier and a third numerical value, and acquiring the data batch with the same batch identifier as the target batch identifier.

As another implementation manner of the present application, in order to further improve efficiency, the apparatus may specifically further include: the subunit is acquired.

The acquisition subunit is configured to, for the target data, image frames that form a video image, and the data identifier is a frame number of the target data in the video image.

Referring first to fig. 7, the data acquisition device provided in the embodiment of the present application includes the following units:

a first determining unit 701, configured to determine, according to a size of a buffer configured in a service area of a front-end device, a target number of storable target data in the buffer;

a second determining unit 702 for determining the total number of target data;

a calculating unit 703, configured to determine the total number of data batches according to a preset second threshold and the total number, where the second threshold is the number of target data that can be included in the data batches at most by the following formula:

/>

A third determining unit 704, configured to determine a first threshold according to the target number and the second threshold, where the first threshold is a number of data batches that can be stored in the buffer at most, and is denoted by a letter a;

and a storage unit 705, configured to send a data batches to the front-end device, so that the front-end device stores the a data batches in the buffer area, so as to obtain target data from the buffer area.

As another implementation manner of the present application, in order to further improve efficiency, the apparatus may specifically further include: and a transmitting subunit.

The sending subunit is configured to receive a target request carrying a lot identifier sent by the front-end device, and send, in response to the target request, a data lot corresponding to the lot identifier to the front-end device.

Fig. 8 shows a schematic hardware structure of a data acquisition device according to an embodiment of the present application.

The data acquisition device may include a processor 801 and a memory 802 storing computer program instructions.

In particular, the processor 801 may include a Central Processing Unit (CPU), or an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), or may be configured as one or more integrated circuits that implement embodiments of the present application.

Memory 802 may include mass storage for data or instructions. By way of example, and not limitation, memory 802 may include a Hard Disk Drive (HDD), floppy Disk Drive, flash memory, optical Disk, magneto-optical Disk, magnetic tape, or universal serial bus (Universal Serial Bus, USB) Drive, or a combination of two or more of the above. Memory 802 may include removable or non-removable (or fixed) media, where appropriate. Memory 802 may be internal or external to the integrated gateway disaster recovery device, where appropriate. In a particular embodiment, the memory 802 is a non-volatile solid-state memory.

In particular embodiments, memory 802 may include Read Only Memory (ROM), random Access Memory (RAM), magnetic disk storage media devices, optical storage media devices, flash memory devices, electrical, optical, or other physical/tangible memory storage devices. Thus, in general, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software comprising computer-executable instructions and when the software is executed (e.g., by one or more processors) it is operable to perform the operations described with reference to methods in accordance with aspects of the present disclosure.

The processor 801 implements any of the data acquisition methods of the above embodiments by reading and executing computer program instructions stored in the memory 802.

In one example, the data acquisition device may also include a communication interface 803 and a bus 810. As shown in fig. 8, the processor 801, the memory 802, and the communication interface 803 are connected to each other via a bus 810 and perform communication with each other.

Communication interface 803 is primarily used to implement communication between modules, devices, units, and/or apparatuses in an embodiment of the present application.

Bus 810 includes hardware, software, or both, coupling components of the data acquisition device to each other. By way of example, and not limitation, the buses may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an infiniband interconnect, a Low Pin Count (LPC) bus, a memory bus, a micro channel architecture (MCa) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a video electronics standards association local (VLB) bus, or other suitable bus, or a combination of two or more of the above. Bus 810 may include one or more buses, where appropriate. Although embodiments of the application have been described and illustrated with respect to a particular bus, the application contemplates any suitable bus or interconnect.

The data acquisition device may perform the data acquisition method in the embodiment of the present application, thereby implementing the data acquisition method and apparatus described in connection with fig. 1, 5, 6 and 7.

In addition, in combination with the data acquisition method in the above embodiment, the embodiment of the present application may be implemented by providing a computer storage medium. The computer storage medium has stored thereon computer program instructions; the computer program instructions, when executed by a processor, implement any of the data acquisition methods of the above embodiments.

It should be understood that the application is not limited to the particular arrangements and instrumentality described above and shown in the drawings. For the sake of brevity, a detailed description of known methods is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method processes of the present application are not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between steps, after appreciating the spirit of the present application.

The functional blocks shown in the above-described structural block diagrams may be implemented in hardware, software, firmware, or a combination thereof. When implemented in hardware, it may be, for example, an electronic circuit, an Application Specific Integrated Circuit (ASIC), suitable firmware, a plug-in, a function card, or the like. When implemented in software, the elements of the application are the programs or code segments used to perform the required tasks. The program or code segments may be stored in a machine readable medium or transmitted over transmission media or communication links by a data signal carried in a carrier wave. A "machine-readable medium" may include any medium that can store or transfer information. Examples of machine-readable media include electronic circuitry, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio Frequency (RF) links, and the like. The code segments may be downloaded via computer networks such as the internet, intranets, etc.

It should also be noted that the exemplary embodiments mentioned in this disclosure describe some methods or systems based on a series of steps or devices. However, the present application is not limited to the order of the above-described steps, that is, the steps may be performed in the order mentioned in the embodiments, or may be performed in a different order from the order in the embodiments, or several steps may be performed simultaneously.

Aspects of the present disclosure are described above with reference to flowchart illustrations and/or block diagrams of methods, apparatus and computer program products according to embodiments of the disclosure. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions/acts specified in the flowchart and/or block diagram block or blocks. Such a processor may be, but is not limited to being, a general purpose processor, a special purpose processor, an application specific processor, or a field programmable logic circuit. It will also be understood that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware which performs the specified functions or acts, or combinations of special purpose hardware and computer instructions.

In the foregoing, only the specific embodiments of the present application are described, and it will be clearly understood by those skilled in the art that, for convenience and brevity of description, the specific working processes of the systems, modules and units described above may refer to the corresponding processes in the foregoing method embodiments, which are not repeated herein. It should be understood that the scope of the present application is not limited thereto, and any equivalent modifications or substitutions can be easily made by those skilled in the art within the technical scope of the present application, and they should be included in the scope of the present application.

Claims

1. The data acquisition method is characterized by being applied to front-end equipment, wherein A data batches are stored in a cache area of the front-end equipment, the data batches comprise B target data, the target data carry data identification, the cache area is a cache area corresponding to a memory to which a front-end equipment service code belongs, A and B are positive integers, A is a preset first threshold, and B is a preset second threshold, and the method comprises the following steps:

acquiring a query request carrying a data identifier;

2. The method of claim 1, wherein the data batch carries a batch identification;

storing the data batch into the cache area;

3. The method of claim 2, wherein the data identifications are incremented in sequence from a first value and the lot identifications are incremented in sequence from a second value, the data lot with smaller lot identifications preferentially configuring data with smaller target data;

wherein Y is a batch identifier, X is a data identifier, C is a first value, D is a second value, and N is a maximum data identifier.

4. The method of claim 2, wherein, based on the lot identification, obtaining the data lot corresponding to the lot identification from a backend device associated with the front-end device comprises:

receiving and storing the data batch returned by the back-end equipment;

5. The method of claim 4, wherein the data batch carries a status identification, the status identification including at least cached, in-request, and uncached;

6. The method of claim 2, wherein the data batch carries a status identification, the status identification including at least cached, in-request, and uncached;

the method further comprises the steps of:

7. The method of claim 1, wherein the data batch carries a status identifier, the status identifier at least including cached, in-request, and uncached, the status identifier of the data batch in the initial state being uncached;

the method further comprises the steps of:

8. The method of claim 1, wherein the data batch carries a batch identification, the method further comprising:

upon determining that the data identification is in interval [ K ] ₂ ，L]When the data batch is in the process, determining a target batch identifier according to the batch identifier of the data batch, wherein the target batch identifier is the batchIdentifying a sum of the third value;

9. A data acquisition method, applied to a front-end device, comprising:

determining a total number of target data;

10. The method of claim 9, wherein the data batch carries a batch identification, the method further comprising: