CN113364877B - Data processing method, device, electronic equipment and medium - Google Patents
Data processing method, device, electronic equipment and medium Download PDFInfo
- Publication number
- CN113364877B CN113364877B CN202110658310.2A CN202110658310A CN113364877B CN 113364877 B CN113364877 B CN 113364877B CN 202110658310 A CN202110658310 A CN 202110658310A CN 113364877 B CN113364877 B CN 113364877B
- Authority
- CN
- China
- Prior art keywords
- seed
- resource
- data
- key value
- index information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000003672 processing method Methods 0.000 title abstract description 30
- 238000003860 storage Methods 0.000 claims description 47
- 238000000034 method Methods 0.000 claims description 25
- 238000012545 processing Methods 0.000 claims description 14
- 238000011084 recovery Methods 0.000 claims description 6
- 230000004044 response Effects 0.000 claims description 3
- 230000005540 biological transmission Effects 0.000 abstract description 2
- 238000004590 computer program Methods 0.000 description 9
- 238000004891 communication Methods 0.000 description 8
- 238000010586 diagram Methods 0.000 description 7
- 238000013461 design Methods 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 230000008569 process Effects 0.000 description 3
- 230000033228 biological regulation Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 241001632422 Radiola linoides Species 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 230000014759 maintenance of location Effects 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 230000001105 regulatory effect Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000001953 sensory effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/104—Peer-to-peer [P2P] networks
- H04L67/1074—Peer-to-peer [P2P] networks for supporting data block transmission mechanisms
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/104—Peer-to-peer [P2P] networks
- H04L67/1074—Peer-to-peer [P2P] networks for supporting data block transmission mechanisms
- H04L67/1078—Resource delivery mechanisms
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The present disclosure provides a data processing method, an apparatus, an electronic device and a medium, which relate to the technical field of computers, and in particular to the technical field of P2P network transmission. The specific implementation scheme is as follows: determining index information corresponding to the seeds according to the resource identification of the resource aiming at each seed of the resource under the condition that the number of the seeds of the acquired resource is less than or equal to the number threshold; determining index information corresponding to the seeds according to the resource identifiers of the resources and the position identifiers of the seeds aiming at each seed of the resources under the condition that the number of the seeds of the resources is larger than a number threshold; storing seed data of each seed according to the index information corresponding to each seed; wherein the seed for the resource indicates a copy of the resource stored by any node in the peer-to-peer network.
Description
Technical Field
The present disclosure relates to the field of computer technologies, and in particular, to the field of P2P network transmission technologies.
Background
P2P (Peer to Peer) networks, i.e., peer-to-Peer computer networks. Each node in the P2P network is in a peer-to-peer status, and each node can serve as a server to provide services for other nodes, and can serve as a client to receive services provided by other nodes.
Disclosure of Invention
The disclosure provides a data processing method, apparatus, device and storage medium.
According to an aspect of the present disclosure, there is provided a data processing method including: under the condition that the number of the obtained seeds of the resource is less than or equal to a number threshold, determining index information corresponding to the seeds according to the resource identification of the resource aiming at each seed of the resource; determining index information corresponding to the seeds according to the resource identifiers of the resources and the position identifiers of the seeds aiming at each seed of the resources under the condition that the number of the seeds of the resources is larger than a number threshold value; storing the seed data of each seed according to the index information corresponding to each seed; wherein a seed for a resource indicates a copy of the resource stored by any node in a peer-to-peer network.
According to another aspect of the present disclosure, there is provided a data processing apparatus including: the first determining module is used for determining index information corresponding to the seeds according to the resource identifiers of the resources under the condition that the quantity of the acquired seeds of the resources is less than or equal to a quantity threshold value, wherein the seeds of the resources are copies of the resources stored by any node in a peer-to-peer network; a second determining module, configured to determine, for each seed of the resource, index information corresponding to the seed according to the resource identifier of the resource and the location identifier of the seed when the number of seeds of the resource is greater than a number threshold; and the storage module is used for storing the seed data of each seed according to the index information corresponding to each seed.
Another aspect of the present disclosure provides an electronic device including: at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of the embodiments of the present disclosure.
According to another aspect of the disclosed embodiments, there is provided a non-transitory computer readable storage medium having stored thereon computer instructions for causing the computer to perform the method shown in the disclosed embodiments.
According to another aspect of embodiments of the present disclosure, there is provided a computer program product, a computer program, which when executed by a processor, implements the method shown in embodiments of the present disclosure.
It should be understood that the statements in this section do not necessarily identify key or critical features of the embodiments of the present disclosure, nor do they limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description.
Drawings
The drawings are included to provide a better understanding of the present solution and are not to be construed as limiting the present disclosure. Wherein:
fig. 1 schematically shows an exemplary application scenario to which a data processing method according to an embodiment of the present disclosure may be applied;
FIG. 2 schematically shows a flow chart of a data processing method according to an embodiment of the present disclosure;
FIG. 3A schematically illustrates a flow diagram of a data processing method according to another embodiment of the present disclosure;
FIG. 3B schematically shows a flow diagram of a data processing method according to another embodiment of the present disclosure;
FIG. 4 schematically shows a flow chart of a data processing method according to another embodiment of the present disclosure;
FIG. 5 schematically shows a data processing method according to another embodiment of the present disclosure;
fig. 6 schematically shows a schematic block diagram of a data processing apparatus according to an embodiment of the present disclosure; and
FIG. 7 illustrates a schematic block diagram of an example electronic device that can be used to implement embodiments of the present disclosure.
Detailed Description
Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, in which various details of the embodiments of the disclosure are included to assist understanding, and which are to be considered as merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present disclosure. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
An application scenario of the data processing method according to the embodiment of the present disclosure is described in detail below with reference to fig. 1.
Fig. 1 schematically shows an exemplary application scenario 100 to which a data processing method may be applied according to an embodiment of the present disclosure. It should be noted that fig. 1 is only an example of an application scenario in which the embodiments of the present disclosure may be applied to help those skilled in the art understand the technical content of the present disclosure, but does not mean that the embodiments of the present disclosure may not be applied to other devices, systems, environments or scenarios.
As shown in fig. 1, an application scenario 100 according to an embodiment of the present disclosure may include a seed server 10, a database server 20, nodes 31, 32, and 33.
The nodes 31, 32, and 33 may be nodes in a P2P network, for example. Nodes 31, 32 and 33 may be used to store resources, which may be the same or different in each node. Illustratively, in this embodiment, the resources may include various types of data such as text files, multimedia files, executable files, compressed files, and so on.
According to an embodiment of the present disclosure, a copy of a resource may be stored in a plurality of nodes, respectively, and the copy of the resource stored in the nodes may be referred to as a seed of the resource. When downloading a resource, one or more of the nodes storing the resource may be selected, and the resource may be acquired from the selected node.
The seed server 10 is a management server for managing various seeds in the nodes 31, 32, and 33. The seed server 10 may interact with the nodes 31, 32, and 33 to receive or transmit data, etc. Database server 20 may be used to store data, and database server 20 may include a MySQL database, a Redis database, and the like. The seed server 10 may interact with the database server 20 to write data to the database server 20 or read data from the database server 20, etc.
According to the embodiment of the present disclosure, when the nodes 31, 32, and 33 possess the seed, the seed data of the seed is reported to the seed server 10, and the seed server 10 may forward the seed data to the database server 20. The subsequent nodes 31, 32 and 33 report the seed data to the seed server 10 at a preset time period to synchronize the seed data with the seed server 10. For example, the preset time period may be 1 hour. In this process, if the seed data in the seed server 10 is lost, the seed data can be obtained from the database server 20 to realize data recovery.
According to other embodiments of the present disclosure, the seed server 10, the database server 20, and the nodes 31, 32, and 33 may be physical servers or cloud servers. The cloud Server is also called a cloud computing Server or a cloud host, and is a host product in a cloud computing service system, so as to solve the defects of high management difficulty and weak service expansibility in the traditional physical host and VPS service ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server incorporating a blockchain.
It should be noted that the data processing method provided by the embodiment of the present disclosure may be generally executed by the seed server 10. Accordingly, the data processing device provided by the embodiment of the present disclosure may be generally disposed in the seed server 10. The data processing method provided by the embodiment of the present disclosure may also be performed by a server or a server cluster different from the seed server 10 and capable of communicating with the database server 20, the nodes 31, 32, 33 and/or the seed server 10. Accordingly, the data processing apparatus provided by the embodiment of the present disclosure may also be disposed in a server or a server cluster different from the seed server 10 and capable of communicating with the database server 20, the nodes 31, 32, 33 and/or the seed server 10.
It should be understood that the number of nodes, database servers, and seed servers in fig. 1 is merely illustrative. There may be any number of nodes, database servers, and seed servers, as desired for an implementation.
In the technical solution of the present disclosure, the acquisition, storage, application, and the like of the related data all conform to the regulations of the related laws and regulations, and do not violate the good custom of the public order.
The data processing method of the embodiment of the present disclosure is explained in detail below with reference to fig. 2.
Fig. 2 schematically shows a flow chart of a data processing method according to an embodiment of the present disclosure.
As shown in fig. 2, the data processing method 200 includes operations S210 to S250. The method 200 may be performed by, for example, the seed server shown above.
In operation S210, a seed number of a resource is acquired.
According to embodiments of the present disclosure, a resource may include, for example, various types of data such as text files, multimedia files, executable files, compressed files, and so forth. The resource may correspond to one or more copies that are respectively stored in nodes of the P2P network, i.e., as a seed for the resource.
In operation S220, it is determined whether the number of seeds of the acquired resource is greater than a number threshold. In case the number of seeds of the acquired resource is less than or equal to the number threshold, operation S230 is performed. In case the seed number of the resource is greater than the number threshold, operation S240 is performed.
According to the embodiments of the present disclosure, the number threshold may be set according to actual needs, and the present disclosure is not particularly limited thereto. For example, in the present embodiment, the number threshold may be set to 500.
In operation S230, for each seed of the resource, index information corresponding to the seed is determined according to the resource identification of the resource. Then, operation S250 is performed.
According to an embodiment of the present disclosure, the index information may be used to provide an index when querying the seed information. The resource identification is used to identify the resource, and may include, for example, an ID (identification number) of the resource.
In operation S240, for each seed of the resource, index information corresponding to the seed is determined according to the resource identifier of the resource and the location identifier of the seed.
According to the embodiment of the disclosure, the location identifier of the seed may be used to indicate the location of the node where the seed is located, and may include, for example, the IP of the node where the seed is located.
According to the embodiment of the disclosure, array data can be generated as index information according to the resource identifier and the position identifier.
In operation S250, seed data of each seed is stored according to the index information corresponding to each seed.
According to an embodiment of the present disclosure, the seed data may include, for example, various information related to the seed, such as IP information of a node where the seed is located, port information, network attributes, and the like.
According to an embodiment of the present disclosure, key value data may be generated according to index information and seed data. The key-value data is then stored to the target location. The key value data may include a key value and a key name. The target location may include, for example, a local memory, a remote memory, etc. The local storage is a storage provided locally, and may include, for example, a memory, a magnetic disk, and the like. The remote storage is a storage provided in another device than the seed server, for example, a storage of the database server.
In this embodiment, the key name may be determined according to the index information, and the key value may be determined according to the seed data.
Other types of data besides key-value data may also be employed to store seed data in accordance with other embodiments of the present disclosure.
According to the embodiment of the disclosure, different index information is adopted to store the seed data of the resource according to the seed number of the resource. For the resource with less seed number, the resource identification of the resource is used as the index information. For the resource with larger seed number, the resource identifier and the position identifier of the seed are used as index information. By the storage mode, for the resources with less seed quantity, only the seed data is required to be inquired according to the resource identification, and the position identification is not required to be inquired, so that the inquiry frequency is reduced, the access bandwidth is reduced, and the inquiry speed is increased.
Illustratively, in one embodiment of the present disclosure, the seed data may be stored in a memory of a database server. When the seed data needs to be read, the seed server accesses the database server to acquire the needed seed data from the database server. When the seed data are obtained in the mode, the access amount to the database server is large, and the seed data need to be transmitted remotely, so that the speed is low. Based on this, fig. 3A schematically shows a flow chart of a data processing method according to another embodiment of the present disclosure, which can reduce the amount of access and improve the data reading speed.
As shown in fig. 3A, the data processing method 300A includes operations S310A to S360A. The method 300A may be performed by, for example, the seed server shown above.
In operation S310A, a seed number of a resource is acquired.
In operation S320A, it is determined whether the number of seeds of the acquired resource is greater than a number threshold. In a case where the number of seeds of the acquired resource is equal to or less than the number threshold, operation S330A is performed. In case the seed number of the resource is greater than the number threshold, operation S340A is performed.
In operation S330A, for each seed of the resource, index information corresponding to the seed is determined according to the resource identifier of the resource. Then, operation S350A is performed.
In operation S340A, for each seed of the resource, index information corresponding to the seed is determined according to the resource identifier of the resource and the location identifier of the seed.
In operation S350A, key value data is generated according to the index information and the seed data, and the key value data is stored to the local memory.
According to an embodiment of the present disclosure, for example, reference may be made to the above in operations S310A to S350A, which are not described herein again.
According to the embodiment of the present disclosure, the local storage is a storage provided in the seed server, and may be, for example, a memory, a disk, or the like.
In operation S360A, in the case where the key value data in the local memory is changed, the changed portion of the key value data is stored to the remote memory.
According to an embodiment of the present disclosure, the remote storage is a storage provided in other devices than the seed server, for example, may be a storage provided in a database server.
According to the embodiment of the disclosure, by storing the seed data of the resource with higher access frequency in the local memory of the seed server, when the seed data needs to be read, the required seed data can be read from the local memory, and the speed is higher. In addition, under the condition that the key value data in the local memory are changed, the changed key value data are stored in the remote memory, so that the access amount of the remote memory can be reduced, and the access bandwidth of the remote memory is reduced.
In addition, fig. 3B schematically shows a flow chart of a data processing method according to another embodiment of the present disclosure, which may further reduce the amount of access.
As shown in FIG. 3B, the data processing method 300B includes operations S310B-S360B. The method 300B may be performed by the seed server shown above, for example.
In operation S310B, a seed number of a resource is acquired.
In operation S320B, it is determined whether the number of seeds of the acquired resource is greater than a number threshold. In a case where the number of seeds of the acquired resource is equal to or less than the number threshold, operation S330B is performed. In case the seed number of the resource is greater than the number threshold, operation S340B is performed.
In operation S330B, for each seed of the resource, index information corresponding to the seed is determined according to the resource identification of the resource. Then, operation S350B is performed.
In operation S340B, for each seed of the resource, index information corresponding to the seed is determined according to the resource identifier of the resource and the location identifier of the seed.
In operation S350B, key value data is generated according to the index information and the seed data, and the key value data is stored to the local memory.
According to an embodiment of the disclosure, for example, reference may be made to the above in operations S310B to S350B, which are not described herein again.
In operation S360B, under a condition that the access frequency of the resource is less than the preset frequency threshold, the key value data corresponding to the resource in the local memory is transferred to the remote memory, and the key value data in the local memory is deleted.
According to the embodiment of the disclosure, when the access frequency of the resource is low, the seed data of the resource in the local memory can be deleted, and the seed data can be transferred to a remote memory (for example, a memory of a database server) to be stored. With this design, the local memory may act as a cache for the remote memory, thereby reducing access to the remote memory.
Fig. 4 schematically shows a flow chart of a data processing method according to another embodiment of the present disclosure.
As shown in fig. 4, the data processing method 400 includes operations S410 to S470.
In operation S410, a seed number of a resource is acquired.
In operation S420, it is determined whether the number of seeds of the acquired resource is greater than a number threshold. In case the number of seeds of the acquired resource is less than or equal to the number threshold, operation S430 is performed. In case the seed number of the resource is greater than the number threshold, operation S440 is performed.
In operation S430, for each seed of the resource, index information corresponding to the seed is determined according to the resource identifier of the resource. Then, operation S450 is performed.
In operation S440, in a case that the number of seeds of the resource is greater than the number threshold, for each seed of the resource, index information corresponding to the seed is determined according to the resource identifier of the resource and the location identifier of the seed.
In operation S450, key value data is generated according to the index information and the seed data, and the key value data is stored in the first local memory.
For example, reference may be made to the above in operations S410 to S450 according to an embodiment of the present disclosure, and details are not described herein.
In operation S460, key value data in the first local memory is backed up to the second local memory at a preset time period.
According to an embodiment of the present disclosure, the first local storage and the second local storage are different local storages. Exemplarily, in this embodiment, the first local storage may be, for example, a memory, and the second local storage may be, for example, a disk.
According to the embodiment of the disclosure, the preset time period can be set according to actual needs. The shorter the preset time period is set, the higher the backup frequency of the key value data is, and the higher the frequency of operations such as reading and writing of the key value data is. Conversely, the longer the preset time period is set, the lower the backup frequency of the key value data is, and the lower the frequency of operations such as reading and writing of the key value data is.
According to the embodiment of the disclosure, in the process of backing up the key value data to the second local storage, when the second local storage does not have the backup data of the key value data stored therein, a copy of the key value data may be created as the backup data, and the backup data may be written into the second local storage. And under the condition that the backup data of the key value data are stored in the second local memory, updating the backup data according to the key value data.
According to the embodiment of the disclosure, when writing into the second memory, the writing speed can be regulated according to the current resource overhead (for example, the amount of read-write data) of the second memory, so that the resource overhead of the second memory during writing is reduced. Based on this, in the case that the read-write data amount of the second local memory is greater than the preset data amount threshold, a plurality of threads may be respectively invoked to write the backup data into the second local memory at preset time intervals, wherein each thread of the plurality of threads is used to write a portion of the backup data into the second local memory.
According to an embodiment of the present disclosure, the number of threads and the preset time interval may be set according to the resource overhead of the current second local memory. For example, the backup data to be written may be divided into 10 copies, and the data is written into the second local storage by 10 threads, with an interval of 10 milliseconds between each thread, and each thread writes 1 copy of the 10 copies.
In operation S470, in response to the data recovery instruction for the target key value data, the backup data of the target key value data in the second local storage is written into the first local storage.
According to the embodiment of the disclosure, when some key value data in the first local storage cannot be read due to data loss and the like, a data recovery instruction may be generated for the key value data which cannot be read (i.e., the target key value data). In response to the data recovery instruction, the backup data corresponding to the unreadable key value data may be searched in the second local storage, and the backup data is written into the first local storage, so as to implement data recovery.
According to an embodiment of the present disclosure, seed data in a first local storage (e.g., a memory) may be backed up to a second local storage (e.g., a disk) periodically. When a failure occurs, data can be quickly recovered from the second local storage, thereby preventing seed loss. In addition, when recovering data, the database server does not need to be accessed, so that the access amount of the database server can be reduced.
The data processing method is further described with reference to fig. 5 in conjunction with specific embodiments. Those skilled in the art will appreciate that the following example embodiments are only for the understanding of the present disclosure, and the present disclosure is not limited thereto.
Fig. 5 schematically shows a data processing method according to another embodiment of the present disclosure.
As shown in fig. 5, in this embodiment, the seeds corresponding to the resources with smaller quantity scale (i.e., the resources with the quantity of the seeds smaller than or equal to the quantity threshold) are referred to as fast seeds 511. The seeds corresponding to the resource with the larger size (i.e. the resource with the seed number larger than the number threshold) are called as the common seeds 512.
For each fast seed 511, key value data is generated by using the seed data of the fast seed 511 as a key value according to the resource identifier fgid of the resource corresponding to the fast seed 511 as a key name. The key-value data may then be stored to a Redis server (i.e., database server) 530.
For each common seed 512, key value data is generated by taking the seed data of the common seed 512 as a key value according to the resource identifier fgid of the resource corresponding to the common seed 512 and the geographical location identifier ipcode of the common seed 512 as key names. The key-value data may then be stored to the Redis server 530.
According to the embodiment of the disclosure, the seed data of the resource is stored in the key-value data with different structures according to the seed number of the resource. When the seed data of the common seed 512 is queried, all the ipcodes corresponding to fgid may be queried according to fgid, then the ipcode of the common seed 512 is queried in all the ipcodes, and then the seed data corresponding to the ipcode of the common seed 512 is obtained. When the seed data of the fast seed 511 is queried, all seed data corresponding to fgid can be queried according to the resource identifier fgid, and then the required seed data is obtained from the seed data, so that the query times of the ipcode are reduced, and the access to the Redis server 530 is reduced.
In addition, seed data of resources with access frequency (or update frequency) higher than a preset frequency threshold may be stored in the memory 520, where the seed data may be seed data of the fast seed 511 or seed data of the normal seed 512. For seed data in memory 520, the changed portions are periodically synchronized to Redis server 530. In addition, when the access frequency (or the update frequency) of the resource becomes lower than the preset frequency threshold, the seed data of the resource may be removed from the memory and transferred to the Redis server 530 for storage. Access to the Redis server 530 may be further reduced by this multi-level cache design.
Seed data stored in the memory 520 in the form of key-value data may be saved to disk 540 periodically. When the seed data cannot be obtained due to the failure of the memory 520, the data can be quickly recovered from the disk 540, so that the seed data is prevented from being lost, and the access amount of the Redis server 530 can be reduced. When storing the key value data to the disk 540, the speed of writing the data into the disk 540 can be controlled according to the resource overhead of the current disk 540, so that the occupation of disk resources is reduced.
Fig. 6 schematically shows a schematic block diagram of a data processing apparatus according to an embodiment of the present disclosure.
As shown in fig. 6, the data processing apparatus 600 includes a first determining module 610, a second determining module 620, and a storing module 630.
The first determining module 610 may be configured to determine, for each seed of the resource, index information corresponding to the seed according to the resource identifier of the resource when the number of the seeds of the acquired resource is less than or equal to the number threshold.
According to embodiments of the present disclosure, the seed for a resource may be a copy of the resource stored by any node in the peer-to-peer network.
The second determining module 620 may be configured to determine, for each seed of the resource, index information corresponding to the seed according to the resource identifier of the resource and the location identifier of the seed, where the number of seeds of the resource is greater than the number threshold.
The storage module 630 may be configured to store seed data of each seed according to the index information corresponding to each seed.
The present disclosure also provides an electronic device, a readable storage medium, and a computer program product according to embodiments of the present disclosure.
FIG. 7 illustrates a schematic block diagram of an example electronic device 700 that can be used to implement embodiments of the present disclosure. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not intended to limit implementations of the disclosure described and/or claimed herein.
As shown in fig. 7, the device 700 comprises a computing unit 701 which may perform various suitable actions and processes according to a computer program stored in a Read Only Memory (ROM) 702 or a computer program loaded from a storage unit 708 into a Random Access Memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the device 700 can be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other by a bus 704. An input/output (I/O) interface 705 is also connected to bus 704.
A number of components in the device 700 are connected to the I/O interface 705, including: an input unit 706 such as a keyboard, a mouse, or the like; an output unit 707 such as various types of displays, speakers, and the like; a storage unit 708 such as a magnetic disk, optical disk, or the like; and a communication unit 709 such as a network card, modem, wireless communication transceiver, etc. The communication unit 709 allows the device 700 to exchange information/data with other devices via a computer network, such as the internet, and/or various telecommunication networks.
Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuitry, field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), system on a chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, receiving data and instructions from, and transmitting data and instructions to, a storage system, at least one input device, and at least one output device.
Program code for implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the program codes, when executed by the processor or controller, cause the functions/operations specified in the flowchart and/or block diagram to be performed. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package partly on the machine and partly on a remote machine or entirely on the remote machine or server.
In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.
The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Networks (WANs), and the Internet.
The computer system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.
It should be understood that various forms of the flows shown above, reordering, adding or deleting steps, may be used. For example, the steps described in the present disclosure may be executed in parallel, sequentially, or in different orders, as long as the desired results of the technical solutions disclosed in the present disclosure can be achieved, and the present disclosure is not limited herein.
The above detailed description should not be construed as limiting the scope of the disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made in accordance with design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present disclosure should be included in the scope of protection of the present disclosure.
Claims (10)
1. A method of data processing, comprising:
under the condition that the number of the obtained seeds of the resource is less than or equal to a number threshold, determining index information corresponding to the seeds according to the resource identification of the resource aiming at each seed of the resource;
determining, for each seed of the resource, index information corresponding to the seed according to a resource identifier of the resource and a location identifier of the seed when the number of seeds of the resource is greater than a number threshold; and
storing seed data of each seed according to the index information corresponding to each seed;
wherein the seed for the resource indicates a copy of the resource stored by any node in the peer-to-peer network.
2. The method of claim 1, wherein the storing seed data for the each seed according to the index information comprises:
generating key value data according to the index information and the seed data, wherein the key value data comprises a key value and a key name, the key name is determined according to the index information, and the key value is determined according to the seed data; and
storing the key value data to a target location.
3. The method of claim 2, wherein the target location comprises a first local memory; the method further comprises the following steps:
and under the condition that the key value data in the first local memory is changed, storing the changed part of the key value data to a remote memory.
4. The method of claim 3, further comprising:
and under the condition that the access frequency of the resource is less than a preset frequency threshold, transferring the key value data corresponding to the resource in the first local memory to a remote memory, and deleting the key value data in the first local memory.
5. The method of claim 3 or 4, further comprising:
backing up the key value data in the first local memory to a second local memory in a preset time period; and
and in response to a data recovery instruction for the target key value data, writing the backup data of the target key value data in the second local memory into the first local memory.
6. The method of claim 5, wherein the backing up the key-value data to a second local memory comprises:
when the backup data of the key value data are not stored in the second local storage, creating a copy of the key value data as the backup data, and writing the backup data into the second local storage; and
and under the condition that the backup data of the key value data are stored in the second local memory, updating the backup data according to the key value data.
7. The method of claim 6, wherein writing the backup data to a second local memory comprises:
and under the condition that the read-write data volume of the second local memory is larger than a preset data volume threshold value, respectively calling a plurality of threads at preset time intervals to write the backup data into the second local memory, wherein each thread in the plurality of threads is used for writing part of the backup data into the second local memory.
8. A data processing apparatus comprising:
the first determining module is used for determining index information corresponding to each seed of the resource according to the resource identifier of the resource under the condition that the number of the obtained seeds of the resource is less than or equal to a number threshold, wherein the seeds of the resource are copies of the resource stored by any node in a peer-to-peer network;
a second determining module, configured to determine, for each seed of the resource, index information corresponding to the seed according to the resource identifier of the resource and the location identifier of the seed when the number of seeds of the resource is greater than a number threshold; and
and the storage module is used for storing the seed data of each seed according to the index information corresponding to each seed.
9. An electronic device, comprising:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-7.
10. A non-transitory computer readable storage medium having stored thereon computer instructions for causing the computer to perform the method of any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110658310.2A CN113364877B (en) | 2021-06-11 | 2021-06-11 | Data processing method, device, electronic equipment and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110658310.2A CN113364877B (en) | 2021-06-11 | 2021-06-11 | Data processing method, device, electronic equipment and medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113364877A CN113364877A (en) | 2021-09-07 |
| CN113364877B true CN113364877B (en) | 2022-10-11 |
Family
ID=77534005
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110658310.2A Active CN113364877B (en) | 2021-06-11 | 2021-06-11 | Data processing method, device, electronic equipment and medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113364877B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113992760B (en) * | 2021-10-22 | 2024-03-01 | 北京百度网讯科技有限公司 | Method, device, equipment and storage medium for scheduling back source traffic |
| CN114218469B (en) * | 2021-12-15 | 2022-09-02 | 掌阅科技股份有限公司 | Resource policy processing method, computing device, and storage medium |
| CN114430395A (en) * | 2022-03-02 | 2022-05-03 | 阿波罗智联(北京)科技有限公司 | Flow control method, device and equipment and intelligent traffic management equipment |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101136911A (en) * | 2006-08-31 | 2008-03-05 | 腾讯科技（深圳）有限公司 | Method to download files using P2P technique and P2P download system |
| CN101610289A (en) * | 2009-07-21 | 2009-12-23 | 腾讯科技（深圳）有限公司 | The methods, devices and systems of data download and node |
| CN101631141A (en) * | 2009-08-19 | 2010-01-20 | 腾讯科技（深圳）有限公司 | P2P system and network resource management method in same |
| CN105049452A (en) * | 2015-08-25 | 2015-11-11 | 广东欧珀移动通信有限公司 | Resource download mode switching method and device, and intelligent terminal |
| CN106686141A (en) * | 2017-03-06 | 2017-05-17 | 成都谛听科技股份有限公司 | Resource download method and resource download device |
| CN108156257A (en) * | 2018-01-17 | 2018-06-12 | 北京奇艺世纪科技有限公司 | A kind of information-pushing method and device |
| CN109905431A (en) * | 2017-12-08 | 2019-06-18 | 京东方科技集团股份有限公司 | Message treatment method and system, storage medium, electronic equipment |
-
2021
- 2021-06-11 CN CN202110658310.2A patent/CN113364877B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101136911A (en) * | 2006-08-31 | 2008-03-05 | 腾讯科技（深圳）有限公司 | Method to download files using P2P technique and P2P download system |
| CN101610289A (en) * | 2009-07-21 | 2009-12-23 | 腾讯科技（深圳）有限公司 | The methods, devices and systems of data download and node |
| CN101631141A (en) * | 2009-08-19 | 2010-01-20 | 腾讯科技（深圳）有限公司 | P2P system and network resource management method in same |
| CN105049452A (en) * | 2015-08-25 | 2015-11-11 | 广东欧珀移动通信有限公司 | Resource download mode switching method and device, and intelligent terminal |
| CN106686141A (en) * | 2017-03-06 | 2017-05-17 | 成都谛听科技股份有限公司 | Resource download method and resource download device |
| CN109905431A (en) * | 2017-12-08 | 2019-06-18 | 京东方科技集团股份有限公司 | Message treatment method and system, storage medium, electronic equipment |
| CN108156257A (en) * | 2018-01-17 | 2018-06-12 | 北京奇艺世纪科技有限公司 | A kind of information-pushing method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113364877A (en) | 2021-09-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113364877B (en) | Data processing method, device, electronic equipment and medium | |
| US11301154B2 (en) | Distributed storage method and device | |
| CN103019960B (en) | Distributed caching method and system | |
| CN113961510B (en) | File processing method, device, equipment and storage medium | |
| CN113656501B (en) | Data reading method, device, equipment and storage medium | |
| CN113760847A (en) | Log data processing method, device, equipment and storage medium | |
| CN111767169A (en) | Data processing method and device, electronic equipment and storage medium | |
| CN113067860B (en) | Method, apparatus, device, medium and product for synchronizing information | |
| CN113220650A (en) | Data storage method, device, apparatus, storage medium, and program | |
| CN113553216A (en) | Data recovery method and device, electronic equipment and storage medium | |
| CN117082046A (en) | Data uploading method, device, equipment and storage medium | |
| CN115510036A (en) | Data migration method, device, equipment and storage medium | |
| CN115905322A (en) | Service processing method and device, electronic equipment and storage medium | |
| CN114518848B (en) | Method, device, equipment and medium for processing stored data | |
| CN113986134B (en) | Method for storing data, method and device for reading data | |
| CN113051244B (en) | Data access method and device, and data acquisition method and device | |
| CN113722389B (en) | Data management method, device, electronic equipment and computer readable storage medium | |
| CN115587147A (en) | Data processing method and system | |
| CN112631517B (en) | Data storage method and device, electronic equipment and storage medium | |
| CN113726885A (en) | Method and device for adjusting flow quota | |
| CN108196979B (en) | Data backup method and device | |
| CN114327293B (en) | Data reading method, device, equipment and storage medium | |
| CN117056248B (en) | Resource reading method, device, electronic equipment and storage medium | |
| CN118132295B (en) | Parameter updating method and device and related equipment | |
| CN115905121A (en) | File processing method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |