CN113110800A

CN113110800A - Data processing method and device, electronic equipment and storage medium

Info

Publication number: CN113110800A
Application number: CN202110389981.3A
Authority: CN
Inventors: 楚振江; 冀向阳; 宋晓东; 汪瑫
Original assignee: Beijing Baidu Netcom Science and Technology Co Ltd
Current assignee: Beijing Baidu Netcom Science and Technology Co Ltd
Priority date: 2021-04-12
Filing date: 2021-04-12
Publication date: 2021-07-13

Abstract

The disclosure provides a data processing method and device, electronic equipment and a storage medium, and relates to the technical field of computers, in particular to the technical fields of cloud computing, searching, information flow and the like. The specific implementation scheme is as follows: according to the service process of the target data downloading end, sending a first acquisition request of a first file block of a data file to a data providing end of a point-to-point network; acquiring a first file block of the data file provided by the data providing terminal according to the first acquisition request; and storing the first file block of the data file into a memory space corresponding to the business process, wherein the memory space corresponding to the business process is used for storing data directly called when the business process runs. The embodiment of the disclosure can improve the data downloading speed.

Description

Data processing method and device, electronic equipment and storage medium

Technical Field

The present disclosure relates to the field of computer technologies, and in particular, to the field of cloud computing, searching, information streaming, and the like.

Background

Under the general application background of the cloud native architecture technology, the service starting time of each data downloading end is increased linearly along with the deployment time brought by the service complexity.

With the development of networks, higher requirements are provided for the data downloading speed of the data downloading end, and it is urgently needed to integrally improve the speed of downloading data files of each data downloading end of the data source end under the cloud native architecture.

Disclosure of Invention

The disclosure provides a data processing method, apparatus, device and storage medium.

According to an aspect of the present disclosure, there is provided a data processing method including:

according to the service process of the target data downloading end, sending a first acquisition request of a first file block of a data file to a data providing end of a point-to-point network;

acquiring a first file block of a data file provided by a data providing end according to a first acquisition request;

and storing the first file block of the data file into a memory space corresponding to the business process, wherein the memory space corresponding to the business process is used for storing data directly called when the business process runs.

According to another aspect of the present disclosure, there is provided a data processing apparatus including:

the system comprises a first acquisition request sending module, a first file obtaining module and a second acquisition request sending module, wherein the first acquisition request sending module is used for sending a first acquisition request of a first file block of a data file to a data providing end of a point-to-point network according to a service process of a target data downloading end;

the first file block acquisition module is used for acquiring a first file block of a data file provided by a data providing terminal according to a first acquisition request;

and the first file block storing module is used for storing the first file block of the data file into a memory space corresponding to the business process, and the memory space corresponding to the business process is used for storing data directly called when the business process runs.

According to another aspect of the present disclosure, there is provided a data processing system comprising: the data downloading device comprises a data source end and a plurality of data downloading ends connected with the data source end; at least one of the plurality of data downloading terminals comprises the data processing device provided by any one of the embodiments of the present disclosure.

According to another aspect of the present disclosure, there is provided an electronic device including:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein,

the memory stores instructions executable by the at least one processor to enable the at least one processor to perform a method according to any one of the embodiments of the present disclosure.

According to another aspect of the present disclosure, there is provided a non-transitory computer readable storage medium having stored thereon computer instructions for causing a computer to perform a method in any of the embodiments of the present disclosure.

According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the method in any of the embodiments of the present disclosure.

According to the technical scheme of the disclosure, a first file block of a data file can be acquired through a data providing terminal in a peer-to-peer network, and a data providing terminal with requested data in a close distance can be selected for data acquisition; meanwhile, the acquired first file block is stored in the memory space corresponding to the business process without passing through a disk, so that the limitation of the disk writing speed is avoided, the data downloading speed and the data writing speed are improved, and the overall speed of data acquisition is further improved.

It should be understood that the statements in this section do not necessarily identify key or critical features of the embodiments of the present disclosure, nor do they limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description.

Drawings

The drawings are included to provide a better understanding of the present solution and are not to be construed as limiting the present disclosure. Wherein:

FIG. 1 is a schematic flow chart diagram of a data processing method according to an embodiment of the present disclosure;

FIG. 2 is a schematic diagram of a data source peer and a data download peer according to an example of the present disclosure;

FIG. 3 is a schematic flow chart diagram of a data processing method according to another embodiment of the present disclosure;

FIG. 4 is a schematic flow chart diagram of a data processing method according to yet another embodiment of the present disclosure;

FIG. 5 is a schematic diagram of a data processing apparatus according to an embodiment of the present disclosure;

FIG. 6 is a schematic diagram of a data processing apparatus according to another embodiment of the present disclosure;

FIG. 7 is a schematic diagram of a data processing apparatus according to yet another embodiment of the present disclosure;

FIG. 8 is a schematic diagram of a data processing apparatus according to yet another embodiment of the present disclosure;

FIG. 9 is a schematic diagram of a data processing apparatus according to yet another embodiment of the present disclosure;

FIG. 10 is a schematic diagram of a data processing apparatus according to yet another embodiment of the present disclosure;

FIG. 11 is a schematic diagram of a data processing system according to an embodiment of the present disclosure;

FIG. 12 is a schematic diagram of an application network of a data processing system according to an example of the present disclosure;

FIG. 13 is a block diagram of a data download side according to an example of the present disclosure;

fig. 14 is a block diagram of an electronic device for implementing a data processing method of an embodiment of the present disclosure.

Detailed Description

Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, in which various details of the embodiments of the disclosure are included to assist understanding, and which are to be considered as merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present disclosure. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.

An embodiment of the present disclosure first provides a data processing method, as shown in fig. 1, including:

step S11: according to the service process of the target data downloading end, sending a first acquisition request of a first file block of a data file to a data providing end of a point-to-point network;

step S12: acquiring a first file block of a data file provided by a data providing end according to a first acquisition request;

step S13: and storing the first file block of the data file into a memory space corresponding to the business process, wherein the memory space corresponding to the business process is used for storing data directly called when the business process runs.

In this embodiment, the target data downloading terminal may be one of a plurality of data downloading terminals that request to download a file block of the data file.

In a specific implementation manner, the target data download end may correspond to the data source end, as shown in fig. 2, in a data download network, there may be one data source end 21 and a plurality of data download ends 22, and the data source end and the data download ends 22 may be distributed in different regions, such as a region one, a region two, a region three, a region four, a region five, and the like. The data downloading terminal 22 can download the data from the data source terminal 21 to the data downloading terminal 22, so that the plurality of data downloading terminals 22 can provide the data to the terminals 23 within the territorial scope of the data downloading terminal 22. The data source end can be a general server for providing data files, and the data download end can be sub-servers which acquire the data files from the data source end and are distributed in a plurality of different regions. The data download port 22 is connected to the data source port 21 and may be connected to other data download ports.

In a specific implementation manner, the target data downloading terminal may run at least one service process, and the service process may use the data file. In case the data file is large, it may be divided into a plurality of file blocks, and the first file block may be one of at least one file block of the data file.

When the target data downloading end runs, the service process needs to use the data file, and the data file can be acquired from the data source end or other data downloading ends nearby. When the target data downloading terminal acquires the data file, the file blocks of the data file can be acquired in sequence.

In a specific implementation manner, the first obtaining request of the first file block of the data file is sent to the data providing end of the peer-to-peer network according to the service process of the target data downloading end, which may be sending the first obtaining request of the first file block of the data file to the data providing end of the peer-to-peer network according to the requirement of the service process of the target data downloading end on the data file.

In a specific implementation manner, the first obtaining request of the first file block of the data file is sent to the data providing end of the peer-to-peer network according to the service process of the target data downloading end, or the first obtaining request of the first file block of the data file is sent to the data providing end of the peer-to-peer network according to the operation information (for example, request information for reading data) of the service process of the target data downloading end.

In a possible implementation manner, the target data downloading end and the data providing end may both belong to the same Peer-to-Peer (P2P, Peer-to-Peer) network.

In a possible implementation manner, after receiving the first acquisition request, the data providing terminal may provide the first file block or the seed file of the first file block to the target data downloading terminal, so that the target data downloading terminal can directly download the first file block or download the first file block according to the seed file.

In a possible implementation manner, the first file block of the data file is stored in the memory space corresponding to the business process, which may be a memory space in which the first file block of the data file is stored in the program data of the business process and the dependent data, that is, a memory space in which the operating data of the business process is located.

In a specific implementation manner, the business process may be a business process of the target data downloading end underlying service module. According to the deployment time consumption statistics of the bottom service modules, the preparation time before starting caused by complex strategies and data is too long, wherein the data downloading usually occupies a relatively time-consuming stage. In the data downloading of a business process, if the size of the data file to be downloaded is too large, more time is needed. For example, a 7 GB-sized data file needs more than 16 minutes to be downloaded from a network data source, and the reason for analyzing the bottom layer is that cloud native containers (for example, a container of a search process, a container of a map process, and other different containers) on the same physical machine are deployed in a mixed manner, and the read-write rate of a Hard Disk Drive (Hard Disk Drive) is commonly used, while an online machine service for providing services to users provides stable services for ensuring multiple container services of the whole machine, and the write speed of a single Disk at a data download end is controlled at a rate of 20-40M/s.

Because the downloading of the large data file has a practical problem of long time consumption, and meanwhile, under the trend that the whole network bandwidth of the physical machine room is continuously increased in speed and expanded, the problem of how to increase the speed of the large data file downloading in a falling tray mode is gradually revealed. Generally, target physical hardware for downloading a data file of a business process is a disk under a default main partition, most of service storage is placed under the main partition by default, and under the condition that the single-disk writing speed of the disk adopted by the main partition is tens of MB/s, the situation that after network data reaches a physical machine, the network data is queued to wait for disk writing occurs. For such a situation, the time mismatch between the data downloading and the module service waiting (for example, the remote acquisition speed is fast, and the local writing speed is slow) brings about the problem of excessive time waiting, that is, the network downloading data is fast and the disk writing data is slow, which is extremely asymmetric.

According to the data processing method provided by the embodiment of the disclosure, the first file block of the data file can be acquired through the data providing terminal in the peer-to-peer network, and the data providing terminal with the requested data in a short distance can be selected for data acquisition during data acquisition; meanwhile, the acquired first file block is stored in the memory space corresponding to the business process without passing through a disk, so that the limitation of the disk writing speed is avoided, the data downloading speed and the data writing speed are improved, and the overall speed of data acquisition is further improved.

In one embodiment, sending a first obtaining request of a first file block of a data file to a data providing terminal of a peer-to-peer network according to a service process of a target data downloading terminal includes:

under the condition that the virtual file system of the target data downloading end determines that the first file block requested by the first acquisition request does not exist in the target data downloading end, acquiring a data providing end of a point-to-point network from a distribution server shared by a plurality of data downloading ends;

and sending a first acquisition request to a data providing end.

In a possible implementation manner, when a service process of a target data downloading end needs to acquire a data file, a query request may be initiated to a data service of the target data downloading end through a virtual system, and whether the data file that the service process needs to acquire is already stored in the target data downloading end is determined through the data service. Under the condition that the target data downloading end does not have the data file which needs to be acquired by the service process, inquiring a shared distribution server in the point-to-point network, determining which data downloading ends in the point-to-point network have the file blocks of the data file which needs to be acquired by the service process, preferentially acquiring the file blocks of the data file from other data downloading ends in the point-to-point network close to the target data downloading end, or preferentially downloading the data file from the data downloading end which can acquire the file blocks of the data file at the fastest speed.

In case that no other data download peer in the peer-to-peer network has a file block of the data file, the data source peer may be determined as the data provider peer.

In one possible implementation manner, the data file required by the business process may be divided into a plurality of file blocks, and the plurality of file blocks may be obtained from at least one data provider.

In one implementation, the data provider that obtains the peer-to-peer network from the distribution server shared by the plurality of data download terminals may be one or more data providers that obtain the peer-to-peer network.

In a specific implementation manner, the data service of the target data downloading end can be interacted with the virtual file system of the target data downloading end, the data service of the target data downloading end is interacted with the target data downloading end, and the downloading service of the target data downloading end is further interacted with a data providing end in a point-to-point network to obtain a first file block of the data file.

In this embodiment, the data providing end is determined by the distribution server in the peer-to-peer network, so that the data providing end with the highest downloading efficiency can be determined according to the distribution of each data downloading end in the peer-to-peer network, and the data downloading speed is integrally increased.

In one embodiment, the peer-to-peer network further comprises a data source; the data providing terminal is other data downloading terminals except the target data downloading terminal in the plurality of data downloading terminals, and the other data downloading terminals store a first file block of a data file directly or indirectly sourced from the data source terminal; or, the data providing end is the data source end.

In one implementation, the data downloading end capable of providing the file block fastest may be used as the data providing end. For example, if the first file block is stored in both of the two data downloading terminals, the first file block may be requested preferentially from the data downloading terminal that is closer to the distance or has a faster transmission speed, and the data downloading terminal that is closer to the distance or has a faster transmission speed may be used as the data providing terminal.

In another implementation manner, if none of the data downloading terminals in the peer-to-peer network downloads the first file block of the data file, the data source terminal may be used as the data providing terminal.

In this embodiment, a data source end or other data downloading end in the peer-to-peer network may be used as the data providing end, so that the device for acquiring the first file block may be determined flexibly, which is beneficial to increase the overall data file acquiring speed.

In one embodiment, before obtaining the data provider of the peer-to-peer network to the distribution server shared by the plurality of data download terminals, as shown in fig. 3, the method includes:

step S31: initiating a data service from a reading request to a target data downloading end by adopting a service process of the target data downloading end;

step S32: generating a reading interface by adopting the data service of the target data downloading end, and sending a reading request to the downloading service of the target data downloading end through the reading interface;

step S33: starting a point-to-point downloading client at the target data downloading end by adopting the downloading service of the target data downloading end;

step S34: and initiating a data providing terminal acquisition request of the first file block to the distribution server through the point-to-point downloading client.

In a possible implementation manner, the data service that the read request is sent to the target data download end is initiated by using a service process of the target data download end, and specifically, the data service that the read request is sent to the target data download end by using a virtual file system of the target data download end and generating the read request by using the service process may be included.

In a specific implementation manner, the peer-to-peer downloading client may be a client running at the target data downloading end and used for downloading the data file.

In a specific implementation manner, a data downloading end which specifically downloads a first file block of a data file can be queried through a distribution server, and the data downloading end which downloads the first file block and has a higher transmission speed is used as a data providing end.

In a specific implementation manner, the target data downloading terminal can be added to the peer-to-peer network when the target data downloading terminal needs to download the file blocks of the data file.

In this embodiment, the data service generates the read interface, and the read interface interacts with the download service, so that the business process can directly perform data interaction with the download service without passing through a magnetic disk of a target data download end, thereby realizing non-magnetic disk download and improving the data drop speed (write-in speed).

In one embodiment, storing a first file block of a data file into a memory space corresponding to a business process includes:

storing the first file blocks into a buffer storage area corresponding to the data file according to the sequence of the first file blocks in the data file;

under the condition that a buffer storage area corresponding to the data file is full, a first file block in the buffer storage area is moved to a memory space corresponding to a service process; or under the condition that the business process calls the first file block of the buffer storage area, moving the first file block called by the business process to the memory space corresponding to the business process.

In one possible implementation, the data file may be divided into a plurality of file blocks, the plurality of file blocks being distributed in the data file in a sequential order. For example, the data file size is 500M, the data of 1M to 100M, the data of 101M to 200M, the data of 201M to 300M, the data of 301M to 400M, and the data of 401M to 500M are divided into 5 file blocks, and the first file block may be any one of the 5 file blocks.

In an implementation manner, the first file block is stored in the buffer storage area corresponding to the data file according to the sequence of the first file block in the data file, specifically, according to the sequence of the first file block in the data file and the existing file block of the data file at the target data downloading end, it is determined that there is still a file block in sequence before the first file block which is not downloaded to the target data downloading end, and then downloading of the file block in sequence before the first file block may be waited first; or in case the buffer memory space size allows to put the first file block and the previous file block of the first file block (the first sequential file block of the data file), the file block before the first file block which is not finished to download is stored in the buffer memory together with the first file block. And after the buffer storage area is full or the downloading of the sequential file blocks before the first file block is finished, transferring the first file block of the buffer storage area into a memory corresponding to the service process when the service process starts to call the first file block.

In a possible implementation manner, the first file block is stored in the buffer storage area corresponding to the data file according to the sequence of the first file block in the data file, or the first file block is stored in the buffer storage area corresponding to the data file requested to be acquired by the service process when the first file block is a file block in the data file that is ordered first.

In a specific implementation manner, a buffer storage area may be set for each data file that needs to be acquired by the business process.

In this embodiment, the buffer storage area is set, so that the file blocks obtained by the target data downloading end can be sequentially called by the service process, and the service process can normally call the data files.

In one embodiment, as shown in fig. 4, the data file processing method further includes:

step S41: receiving a second acquisition request of file blocks of the data file sent by other data downloading terminals in the point-to-point network;

step S42: and sending the second file block of the data file stored on the target data downloading terminal to other data downloading terminals according to the second acquisition request.

In a specific implementation manner, the target data downloading end may become a data providing end after acquiring the first file block or other file blocks of the data file, and provide the file blocks of the data file for other data downloading ends in the peer-to-peer network.

In a specific implementation manner, the second file block may be the first file block, or may be another file block divided by the data file.

In this embodiment, the target data downloading end may become a data providing end after acquiring the file block of the data file, so that the data downloading efficiency in the whole peer-to-peer network can be improved.

In one embodiment, sending a first get request for a first file block of a data file to a data provider of a peer-to-peer network comprises:

acquiring a seed file of a first file block for downloading a data file through a downloading service of a target data downloading terminal;

and sending a first acquisition request of a first file block of the data file to a data providing end of the point-to-point network according to the seed file.

In a specific implementation manner, the seed file of the first file block may be obtained from a distribution server shared by a plurality of data downloading terminals in the peer-to-peer network, or may be obtained from other storage locations.

In a specific implementation manner, the seed file may be generated according to file blocks after the data file is divided by a distribution server shared by a plurality of data providing terminals of the peer-to-peer network.

In this embodiment, the first obtaining request of the first file block is sent to the data providing terminal according to the seed file, so that the first file block can be downloaded through the seed file.

In one embodiment, the seed file is generated by the distribution server according to the distribution strategy of the distribution management controller shared by the plurality of data downloading terminals and the data file.

In this embodiment, the torrent file is generated by the distribution server, so that after the torrent file is generated, the distribution server can distribute the torrent file according to a set distribution strategy, so that the data download terminal can obtain the file blocks of the data file according to the torrent file.

An embodiment of the present disclosure further provides a data processing apparatus, as shown in fig. 5, including:

a first obtaining request sending module 51, configured to send a first obtaining request of a first file block of a data file to a data providing end of a peer-to-peer network according to a service process of a target data downloading end;

a first file block obtaining module 52, configured to obtain a first file block of a data file provided by the data providing terminal according to the first obtaining request;

the first file block storing module 53 is configured to store the first file block of the data file into a memory space corresponding to the business process, where the memory space corresponding to the business process is used to store data directly called when the business process runs.

In one embodiment, as shown in fig. 6, the first obtaining request sending module includes:

a data provider acquiring unit 61, configured to acquire a data provider of the peer-to-peer network from a distribution server shared by the plurality of data downloading terminals when it is determined that the first file block requested by the first acquisition request does not exist at the target data downloading terminal through the virtual file system of the target data downloading terminal;

a sending unit 62, configured to send the first obtaining request to the data providing side.

In one embodiment, as shown in fig. 7, the data processing apparatus further includes:

a read request initiating module 71, configured to initiate a data service from a read request to a target data downloading end by using a service process of the target data downloading end;

the reading interface module 72 is configured to generate a reading interface by using the data service of the target data downloading end, and send a reading request to the downloading service of the target data downloading end through the reading interface;

a starting module 73, configured to start a point-to-point downloading client at a target data downloading end by using a downloading service of the target data downloading end;

and a provider-side obtaining request module 74, configured to initiate a data provider-side obtaining request of the first file block to the distribution server through the peer-to-peer downloading client.

In one embodiment, as shown in fig. 8, the first file block storing module includes:

the cache unit 81 is configured to store the first file blocks into the corresponding cache areas of the data files according to the sequence of the first file blocks in the data files;

a first transferring unit 82, configured to, when a buffer storage area corresponding to a data file is full, transfer a first file block in the buffer storage area to a memory space corresponding to a service process;

and/or the second transfer unit 83 is configured to, under the condition that the service process calls the first file block in the buffer storage area, move the first file block called by the service process to the memory space corresponding to the service process.

In one embodiment, as shown in fig. 9, the data processing apparatus further includes:

a second obtaining request receiving module 91, configured to receive a second obtaining request for a file block of a data file sent by another data downloading end in the peer-to-peer network;

and a second file block sending module 92, configured to send a second file block of the data file stored on the target data download end to another data download end according to the second obtaining request.

In one embodiment, as shown in fig. 10, the first obtaining request sending module includes:

a seed file unit 101, configured to obtain a seed file of a first file block for downloading a data file through a download service of a target data download end;

the torrent file processing unit 102 is configured to send a first obtaining request of a first file block of a data file to a data providing end of a peer-to-peer network according to the torrent file.

An embodiment of the present disclosure further provides a data processing system, as shown in fig. 11, including: a data source end 111 and a plurality of data download ends 112 connected with the data source end; at least one of the plurality of data downloading terminals includes a data processing device 113 as provided in any of the embodiments of the present disclosure.

In one implementation, at least one of the plurality of data download terminals in the data processing system includes the data service and the download service of any one of the embodiments of the present disclosure.

In one example of the present disclosure, a data processing system achieves acceleration of a data network based on data download diskless of a P2P network. After the data file of the data source section is updated, the auxiliary system informs each data downloading end of the event of updating the data file, and after the equipment of the data downloading end downloads the file block of the data file through the network, the equipment receives the data packet transmitted in the network and directly stores the data packet into the memory of the service process, and informs the service process of carrying out loading operation, thereby realizing the diskless remote data request and loading scheme.

In the disclosed example, as shown in fig. 12, the data acceleration network may mainly include an online data source 121 and a data downloading end, and the data downloading end may further include a data distribution management 122, a distribution server 123, a download service 124, and a data service 125. Therein, the data service 125 may correspond to a business APP, and the download service 124 corresponds to a data service. A plurality of data download terminals may share one distribution server 123 and one data distribution management 122, and one data service 125 may correspond to a plurality of download services 124.

Referring to fig. 12, the data acceleration network may perform container management through a PAAS (Platform As a Service) controller, where each container corresponds to a main business process. Meanwhile, an MIS (Management Information System) version Management module may be further used to manage the distribution policy generated by the data distribution Management 122 and notify the PAAS controller.

In one specific implementation, a Data-Service (Data-Service) is used to provide a Data-reading Service interface for a business process to perform interactive state synchronization with the business process. The business process may be a process of a business program (APP), and data of the business process at the data downloading end is obtained through a downloading or re-downloading operation.

The data service is also used for interacting with the downloading service, informing the data file to start downloading and starting, and acquiring the downloading state of the remote data file. Meanwhile, the data service is also used for performing persistent operation (such as writing in a disk) by dropping the asynchronous low-speed writing data file to the storage medium so as to realize a standby scheme of data downloading, and writing part of downloaded data into the disk so as to recover the downloaded data when the downloaded data is abnormal. By asynchronous low-speed writing of the data file, online acquisition of the data file and direct storage of the data file into a memory space corresponding to a business process are not affected.

In addition, the data downloading end provides a read proxy function for data service on the local storage medium, obtains a file name in the virtual file system, judges the distribution condition of each file block of the data file corresponding to the read file name and provides a quick reading capability for abnormal recovery.

In a specific example, a Download-Service (Download-Service) may also be disposed at the data Download end, and may be configured to interface with the data Service, receive a data file Download instruction sent from the data Service to the Download Service, and finally place the data file downloaded from the cloud P2P in the memory and deliver the data file to the data Service at the same time. The cloud end can comprise a data source end and other data downloading ends which have already downloaded data.

The download service may also be configured to interact with the distribution service, and obtain specific information of the data file to be downloaded and distribution information of file blocks of the data file from the distribution server. Meanwhile, the download service is also used for registering heartbeat information, and the heartbeat information indicates that the data download end has a continuous download function, so that the seed file of the download file P2P is acquired, and the download state is synchronized to the distribution server.

The download service may also be used to interact with the data source and peer data download nodes to exchange and download data blocks over the P2P network.

The distribution Server (Tracker-Server) may be set to be one for all data downloads in the peer-to-peer network, that is, all data downloads in the peer-to-peer network may share the distribution Server.

The distribution server is used for receiving and executing instructions of data distribution management, providing a data file meta information (including basic information such as names of data files) query function for the data downloading end, and feeding back summarized information to the management controller.

The distribution server can also be used for managing the downloading service state, collecting the downloading progress of the downloading service, completing the downloading example of the data file, and setting the data file example to be in the downloading completed state.

Meanwhile, the distribution server is also used for splitting file blocks of the data file, making seed files for the split file blocks and distributing the seed files to each data downloading end.

The function of Data distribution management (Data-Controller) is biased to background management and is used for realizing the functions of core decision and the like.

The data distribution management can be called as a data distribution management and controller, controls data file operation and manages the multi-region data files.

The data distribution management can be used for registration management of data files, detection of changes of the data files provided by the data source, and generation and management of data versions according to the data files provided by the data source.

The data distribution management can also be used for data monitoring and version report management, and is used for querying a data downloading end in the system.

In another example of the present application, in a disketted remote data acceleration network, a disk writing speed limit that affects a dictionary used by an online service is removed, and the overall time consumption for network data transmission to the online service to be read into a memory is reduced, wherein the dictionary may be a data file and further may belong to a part of dependent data of a business process. The main implementation mechanism is shown in fig. 13.

Referring to fig. 13, a virtual file system 131, an online business service program 132(Main App Process), which is equivalent to the business App in fig. 12, is started on a physical device at a data downloading end, and at least one business Process is started in the running Process.

Still referring to fig. 13, the online service program uses a Virtual File System 131(Virtual File System) to Mount the address of the data File in the data service 133 through a Soft Link (Soft Link), and places the directory name of the data File in a Mount Point directory.

The virtual file system 131 can reduce the cost of the user, and the read file storage address is changed from the disk to the virtual file system, so that the change of the read file is minimum.

Still referring to fig. 13, Meta-information (Meta-Info) management, seed file (feeding) management, and data persistence (persistence) are designed and implemented in the data service 133 of the data downloading end, when a service process initiates a Mount Point (Mount Point) remote dictionary reading state (read stat) operation, a reading action is sent to the data service 133, and when it is determined that data to be read does not exist locally at the data downloading end, the data service 133 starts a reading interface to initiate a data request to the downloading service 134.

Still referring to fig. 13, after receiving the instruction of reading the file, the download service 134 starts the P2P network download client (P2P download) to download the remote data (file block of the data file) through the seed file, the downloaded data is transmitted back to the download service client through the network, the download service sends the data to the data service in batches, the data service merges and processes the data, and returns the data to the service program, thereby completing a download process of the data acceleration network.

According to the method and the device, the remote data are directly downloaded and enter the memory space corresponding to the business process through the data service and the downloading service, the performance bottleneck limitation caused by writing in a disk is eliminated, and therefore the speed of remote data downloading and memory loading is remarkably improved.

For fast file transmission and downloading in a P2P network, a data file in the network is divided into a plurality of small data packets (i.e. data blocks in other embodiments), each packet calculates a hash value, and according to a policy generated by data distribution management, a plurality of peer nodes are concurrently requested as much as possible in the network as a receiving end of the data packet of the data file, and for the implemented data acceleration network, it is required to directly provide a data memory to enter a service program memory, so that the basic order of the file blocks needs to be ensured.

In the disclosed example, still referring to fig. 13, a buffer storage 135 for data files is added in the download service 134. The buffer memory 135 may download data in P2P or may download data in a backup cluster download for data recovery in case of an anomaly. When the download service 134 provides data to the data service 133, data transmission may be performed according to a set Speed Limit (Speed Limit).

In the computer program, each dictionary (data file) is correspondingly established with a buffer storage area, each program corresponds to a plurality of dictionaries, the buffer storage area can be used for storing data packets with the data files sorted in the front, and each data downloading end preferentially acquires data in the buffer storage area when downloading the data packets or data blocks of the data files and uses the data packets or data blocks in sequence when using the data packets or data blocks. According to the data in the P2P network, the window is sequentially obtained in a sliding way according to the size of the buffer storage area and the set window size of the file block, and finally the downloading service has the functions of sequentially obtaining and returning the data file. In the data service, the consumption range of machine resources can be limited by reading and controlling the downloading speed, and the resource usage amount of the computer CPU and the internal memory under different data downloading speeds can be controlled.

The embodiment of the disclosure realizes the ordered and rapid downloading and acquisition of network data downloading in a window by improving the downloading rate of the P2P network in a controlled manner and adding a file block sliding window technical scheme to the downloading service, thereby achieving a balance scheme of speed and resource consumption.

The embodiment of the disclosure can be applied to the fields of cloud computing, searching, information flow and the like.

When the method is applied to the field of information flow, modeling can be performed on a user in an off-line system, vectorization representation generation is performed on articles and videos of recommended content of the information flow, and finally an index data file is generated to serve as a data source of an on-line computer program. The online computer program carries out diskless network accelerated downloading on the required index data file through the invention, places the required data file into the program memory, and finally responds to the user access request so that the user obtains the recommended content of the information stream.

Meanwhile, the technical scheme provided by the embodiment of the disclosure effectively speeds up the downloading speed of the remote dictionary (data file), and establishes a set of diskless data downloading and loading scheme, so that the online service loads the data file to get rid of the problem of disk writing speed limit, and the deployment speed of the online service is greatly improved. The method is currently applied to a real online search engine system and an information flow recall system, the time consumption for downloading the original big data file is shortened to 50% -12%, the time consumption for downloading the search recommendation big dictionary is shortened to 10 minutes from more than 20 minutes, the downloading speed of the information flow vector index is shortened to 5 minutes from 42 minutes, and good business effect benefits are obtained.

The embodiments of the present disclosure can be applied to the technical field of computers, and particularly can be applied to the technical fields of cloud computing, big data, information flow, and the like.

Furthermore, the embodiments of the present disclosure can be applied to the acquisition of data files of servers corresponding to products such as a search engine online service system, an information flow recommendation system, related search products, demand guidance products, and input box prompt word products.

The functions of each unit, module or sub-module in each apparatus in the embodiments of the present disclosure may refer to the corresponding description in the above method embodiments, and are not described herein again.

The present disclosure also provides an electronic device, a readable storage medium, and a computer program product according to embodiments of the present disclosure.

FIG. 14 shows a schematic block diagram of an example electronic device 140 that may be used to implement embodiments of the present disclosure. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the disclosure described and/or claimed herein.

As shown in fig. 14, the electronic device 140 includes a computing unit 141, which can perform various appropriate actions and processes according to a computer program stored in a Read Only Memory (ROM)142 or a computer program loaded from a storage unit 148 into a Random Access Memory (RAM) 143. In the RAM 143, various programs and data required for the operation of the electronic apparatus 140 can also be stored. The calculation unit 141, the ROM 142, and the RAM 143 are connected to each other via a bus 144. An input/output (I/O) interface 145 is also connected to bus 144.

A number of components in the electronic device 140 are connected to the I/O interface 145, including: an input unit 146 such as a keyboard, a mouse, or the like; an output unit 147 such as various types of displays, speakers, and the like; a storage unit 148 such as a magnetic disk, optical disk, or the like; and a communication unit 149 such as a network card, modem, wireless communication transceiver, etc. The communication unit 149 allows the electronic device 140 to exchange information/data with other devices through a computer network such as the internet and/or various telecommunication networks.

Computing unit 141 may be a variety of general and/or special purpose processing components having processing and computing capabilities. Some examples of the computing unit 141 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various dedicated Artificial Intelligence (AI) computing chips, various computing units running machine learning model algorithms, a Digital Signal Processor (DSP), and any suitable processor, controller, microcontroller, and so forth. The calculation unit 141 performs the respective methods and processes described above, such as a data processing method. For example, in some embodiments, the data processing method may be implemented as a computer software program tangibly embodied in a machine-readable medium, such as storage unit 148. In some embodiments, part or all of the computer program may be loaded and/or installed onto electronic device 140 via ROM 142 and/or communications unit 149. When loaded into RAM 143 and executed by computing unit 141, a computer program may perform one or more steps of the data processing method described above. Alternatively, in other embodiments, the computing unit 141 may be configured to perform the data processing method by any other suitable means (e.g., by means of firmware).

Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuitry, Field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), system on a chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, receiving data and instructions from, and transmitting data and instructions to, a storage system, at least one input device, and at least one output device.

Program code for implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the program codes, when executed by the processor or controller, cause the functions/operations specified in the flowchart and/or block diagram to be performed. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package partly on the machine and partly on a remote machine or entirely on the remote machine or server.

In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic input, speech input, or tactile input.

The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), Wide Area Networks (WANs), and the Internet.

The computer system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

It should be understood that various forms of the flows shown above may be used, with steps reordered, added, or deleted. For example, the steps described in the present disclosure may be executed in parallel or sequentially or in different orders, and are not limited herein as long as the desired results of the technical solutions disclosed in the present disclosure can be achieved.

The above detailed description should not be construed as limiting the scope of the disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made in accordance with design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present disclosure should be included in the scope of protection of the present disclosure.

Claims

1. A method of data processing, comprising:

acquiring a first file block of the data file provided by the data providing terminal according to the first acquisition request;

2. The method as claimed in claim 1, wherein the sending a first obtaining request of a first file block of the data file to a data providing end of the peer-to-peer network according to the business process of the target data downloading end comprises:

under the condition that the virtual file system of the target data downloading end determines that the target data downloading end does not have the first file block requested by the first acquisition request, acquiring a data providing end of a point-to-point network from a distribution server shared by a plurality of data downloading ends;

and sending the first acquisition request to the data providing end.

3. The method of claim 2, wherein the peer-to-peer network further comprises a data source; the data providing terminal is other data downloading terminals except the target data downloading terminal in the data downloading terminals, and the other data downloading terminals store a first file block of a data file directly or indirectly sourced from the data source terminal; or, the data providing end is the data source end.

4. The method of claim 3, wherein the obtaining the data provider of the peer-to-peer network from the distribution server shared by the plurality of data downloaders comprises:

initiating a reading request to a data service of the target data downloading terminal by adopting the service process of the target data downloading terminal;

generating a reading interface by adopting the data service of the target data downloading end, and sending the reading request to the downloading service of the target data downloading end through the reading interface;

starting a point-to-point downloading client at the target data downloading end by adopting the downloading service of the target data downloading end;

and initiating a data providing terminal acquisition request of the first file block to the distribution server through the point-to-point downloading client.

5. The method of claim 1, wherein the storing the first file block of the data file into the memory space corresponding to the business process comprises:

under the condition that a buffer storage area corresponding to the data file is full, moving a first file block in the buffer storage area to a memory space corresponding to a service process; or under the condition that the business process calls the first file block of the buffer storage area, moving the first file block called by the business process to a memory space corresponding to the business process.

6. The method according to any one of claims 1-5, further comprising:

receiving a second acquisition request of file blocks of the data file sent by other data downloading terminals in the point-to-point network;

and sending a second file block of the data file stored on the target data downloading terminal to the other data downloading terminals according to the second acquisition request.

7. The method according to any one of claims 1 to 5, wherein the sending a first get request of a first file block of a data file to a data provider of a peer-to-peer network comprises:

acquiring a seed file of a first file block for downloading the data file through a downloading service of the target data downloading terminal;

8. The method of claim 7, wherein the seed file is generated by the distribution server according to a distribution policy of a distribution management controller shared by the plurality of data download terminals and the data file.

9. A data processing apparatus comprising:

a first file block obtaining module, configured to obtain a first file block of the data file provided by the data providing end according to the first obtaining request;

and the first file block storing module is used for storing the first file block of the data file into the memory space corresponding to the business process, and the memory space corresponding to the business process is used for storing the data directly called when the business process runs.

10. The apparatus of claim 9, wherein the first acquisition request sending module comprises:

a data provider acquiring unit, configured to acquire a data provider of a peer-to-peer network from a distribution server shared by multiple data download terminals when it is determined that the first file block requested by the first acquisition request does not exist at a target data download terminal through a virtual file system of the target data download terminal;

a sending unit, configured to send the first acquisition request to the data providing end.

11. The apparatus of claim 10, wherein the peer-to-peer network further comprises a data source; the data providing terminal is other data downloading terminals except the target data downloading terminal in the data downloading terminals, and the other data downloading terminals store a first file block of a data file directly or indirectly sourced from the data source terminal; or, the data providing end is the data source end.

12. The apparatus of claim 11, wherein the apparatus further comprises:

a read request initiating module, configured to initiate a data service from a read request to the target data downloading end by using the service process of the target data downloading end;

the reading interface module is used for generating a reading interface by adopting the data service of the target data downloading end and sending the reading request to the downloading service of the target data downloading end through the reading interface;

the starting module is used for starting the point-to-point downloading client at the target data downloading end by adopting the downloading service of the target data downloading end;

and the providing terminal acquisition request module is used for initiating a data providing terminal acquisition request of the first file block to the distribution server through the point-to-point downloading client.

13. The apparatus of claim 9, wherein the first file block logging module comprises:

the cache unit is used for storing the first file blocks into a buffer storage area corresponding to the data file according to the sequence of the first file blocks in the data file;

the first transfer unit is used for transferring the first file block in the buffer storage area to the memory space corresponding to the business process under the condition that the buffer storage area corresponding to the data file is full;

and/or the second transfer unit is used for transferring the first file block called by the service process to a memory space corresponding to the service process under the condition that the first file block of the buffer storage area is called by the service process.

14. The apparatus of any of claims 9-13, wherein the apparatus further comprises:

the second acquisition request receiving module is used for receiving a second acquisition request of a file block of a data file sent by other data downloading terminals in the peer-to-peer network;

and the second file block sending module is used for sending the second file block of the data file stored on the target data downloading terminal to the other data downloading terminals according to the second acquisition request.

15. The apparatus according to any one of claims 9-13, wherein the first obtaining request sending module includes:

a seed file unit, configured to obtain a seed file for downloading a first file block of the data file through a download service of the target data download end;

and the seed file processing unit is used for sending a first acquisition request of a first file block of the data file to a data providing end of the point-to-point network according to the seed file.

16. The apparatus of claim 15, wherein the seed file is generated by the distribution server according to a distribution policy of a distribution management controller shared by the plurality of data download ports and the data file.

17. A data processing system, comprising: the data downloading device comprises a data source end and a plurality of data downloading ends connected with the data source end; at least one of the plurality of data download ports comprises a data processing apparatus according to any of claims 9-16.

18. The system of claim 17, wherein at least one of the plurality of data download ports comprises the data service and download service of claim 4.

19. An electronic device, comprising:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein,

the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-8.

20. A non-transitory computer readable storage medium having stored thereon computer instructions for causing a computer to perform the method of any one of claims 1-8.

21. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1-8.