CN111803917A

CN111803917A - Resource processing method and device

Info

Publication number: CN111803917A
Application number: CN202010619437.9A
Authority: CN
Inventors: 汤爱迪
Original assignee: Beijing Kingsoft Cloud Network Technology Co Ltd
Current assignee: Beijing Kingsoft Cloud Network Technology Co Ltd
Priority date: 2020-06-30
Filing date: 2020-06-30
Publication date: 2020-10-23

Abstract

The application relates to a resource processing method and device, wherein the method comprises the following steps: receiving a storage request, wherein the storage request is used for requesting to store a target resource into a database, and the storage request carries a resource identifier and a resource address of the target resource; acquiring a target disk file from a database in response to a storage request, wherein the database is set to allow one or more target disk files to be created, and each target disk file is used for storing a plurality of resources requested to be stored in the database; downloading the target resource from the resource address; and writing the target resources into the target disk file in sequence, and recording the storage information of the target resources according to the resource identification. The method and the device solve the technical problem that the resource processing efficiency is low in the related technology.

Description

Resource processing method and device

Technical Field

The present application relates to the field of computers, and in particular, to a resource processing method and apparatus.

Background

The current resource storage mode is that each resource (image, audio, video, text and the like) is landed as a resource file, the scheme is simple and easy to operate, but random reading and writing are carried out when the resource is written in each time, the hard disk carries out the seek operation, and the seek is the most time-consuming operation in the hard disk. And in the process of reading resources, the resource information is searched for and hard disk reading is needed for many times, the speed is low, and the throughput is low. The above-mentioned currently used resource processing manner results in low efficiency of resource processing.

In view of the above problems, no effective solution has been proposed.

Disclosure of Invention

The application provides a resource processing method and device, which are used for at least solving the technical problem of low resource processing efficiency in the related art.

According to an aspect of an embodiment of the present application, there is provided a resource processing method, including:

receiving a storage request, wherein the storage request is used for requesting to store a target resource into a database, and the storage request carries a resource identifier and a resource address of the target resource;

obtaining a target disk file from the database in response to the storage request, wherein the database is configured to allow creation of one or more target disk files, each target disk file being used to store a plurality of resources requested to be stored in the database;

downloading the target resource from the resource address;

and sequentially writing the target resources into the target disk file, and recording the storage information of the target resources according to the resource identification.

Optionally, the obtaining the target disk file from the database in response to the storage request includes:

determining whether a disk file is established in the database;

under the condition that the disk file is determined to be established in the database, determining whether the established disk file has a writing space;

determining the established disk file as the target disk file under the condition that the write-in space of the established disk file is determined;

and under the condition that the disk file is not established in the database or the established disk file does not have a writing space, allocating a storage space with a target data volume from the database as the target disk file.

Optionally, sequentially writing the target resources to the target disk file includes:

copying the target resource to a first cache page of the database;

and writing the data cached on the first cache page into the target disk file sequentially at a first time interval.

Optionally, copying the target resource to the first cache page of the database comprises:

mapping the target resource to a target virtual address;

performing read-write operation on the target virtual address;

and copying the target resource with the mapping relation with the target virtual address to the first cache page in response to the read-write operation.

Optionally, recording storage information of the target resource according to the resource identifier includes:

generating the storage information, wherein the storage information comprises: the resource identifier, the address of the target disk file and the storage location of the target resource, wherein the storage location is used for indicating the storage starting location and the storage ending location of the target resource in the target disk file;

and inserting the storage information into a target resource information file.

Optionally, inserting the storage information into the target resource information file comprises:

determining the target resource information file corresponding to the resource identifier from a plurality of resource information files;

determining the information position of the storage information in the target resource information file according to the resource identifier and the data volume of the storage information;

and inserting the storage information into the information position of the target resource information file.

writing the storage information into a second cache page;

and inserting the data cached on the second cache page into the target resource information file at intervals of a second time.

Optionally, after the target resources are sequentially written into the target disk file, the method further includes:

mapping the resource identifier through a plurality of mapping functions to obtain a plurality of mapping values;

acquiring a target position corresponding to each mapping value in the plurality of mapping values from a resource filter, wherein the resource filter is used for recording resources existing in the database, a first character of a character at a position included in the resource filter represents that the resource corresponding to the position exists, and a second character of the character at the position included in the resource filter represents that the resource corresponding to the position does not exist;

setting the character of the target position as the first character.

According to another aspect of the embodiments of the present application, there is provided another resource processing method, including:

receiving an acquisition request, wherein the acquisition request is used for requesting to acquire a target resource from a database, and the acquisition request carries a resource identifier of the target resource;

responding to the acquisition request to acquire a target disk file corresponding to the resource identifier from one or more disk files established in the database, wherein the one or more disk files are used for additionally storing resources according to the received storage request;

acquiring the target resource from the target disk file;

and sending the target resource.

Optionally, the obtaining, in response to the obtaining request, a target disk file corresponding to the resource identifier from one or more disk files created in the database includes:

determining a target resource information file corresponding to the resource identifier from a plurality of resource information files;

determining the information position of the storage information corresponding to the resource identifier in the target resource information file according to the resource identifier and the data volume of the storage information;

obtaining the storage information from the information position of the target resource information file, wherein the storage information comprises: the resource identifier, the address of the target disk file and the storage location of the target resource, wherein the storage location is used for indicating the storage starting location and the storage ending location of the target resource in the target disk file;

and acquiring the target disk file from the address of the target disk file.

Optionally, the obtaining the target resource from the target disk file includes:

acquiring the target resource from the storage position of the target disk file;

and copying the target resource to a preset cache.

Optionally, the sending the target resource includes:

acquiring the position and offset of the target resource in the preset cache;

and sending the target resource located at the position and the offset in the preset cache.

Optionally, before acquiring, in response to the acquisition request, a target disk file corresponding to the resource identifier from one or more disk files created in the database, the method further includes:

acquiring characters at a target position corresponding to each mapping value in the plurality of mapping values from a resource filter, wherein the resource filter is used for recording resources existing in the database, the character at the position included in the resource filter is a first character and indicates that the resource corresponding to the position exists, and the character at the position included in the resource filter is a second character and indicates that the resource corresponding to the position does not exist;

and under the condition that the characters on the target position are the first characters, responding to the acquisition request to acquire the target disk file corresponding to the resource identifier from one or more disk files established in the database.

According to another aspect of the embodiments of the present application, there is also provided a resource processing apparatus, including:

a first receiving module, configured to receive a storage request, where the storage request is used to request to store a target resource in a database, and the storage request carries a resource identifier and a resource address of the target resource;

a first obtaining module, configured to obtain a target disk file from the database in response to the storage request, where the database is configured to allow creation of one or more target disk files, and each target disk file is used to store a plurality of resources requested to be stored in the database;

the downloading module is used for downloading the target resource from the resource address;

and the writing module is used for sequentially writing the target resources into the target disk file and recording the storage information of the target resources according to the resource identification.

According to another aspect of the embodiments of the present application, there is provided another resource processing apparatus, including:

a second receiving module, configured to receive an acquisition request, where the acquisition request is used to request to acquire a target resource from a database, and the acquisition request carries a resource identifier of the target resource;

a second obtaining module, configured to respond to the obtaining request to obtain a target disk file corresponding to the resource identifier from one or more disk files created in the database, where the one or more disk files are used to add a storage resource according to the received storage request;

a third obtaining module, configured to obtain the target resource from the target disk file;

and the sending module is used for sending the target resource.

According to another aspect of the embodiments of the present application, there is also provided a storage medium including a stored program which, when executed, performs the above-described method.

According to another aspect of the embodiments of the present application, there is also provided an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the above method through the computer program.

In the embodiment of the application, a storage request is received, wherein the storage request is used for requesting to store a target resource in a database, and the storage request carries a resource identifier and a resource address of the target resource; acquiring a target disk file from a database in response to a storage request, wherein the database is set to allow one or more target disk files to be created, and each target disk file is used for storing a plurality of resources requested to be stored in the database; downloading the target resource from the resource address; the method comprises the steps of sequentially writing target resources into target disk files, setting a database to allow one or more target disk files to be created according to a mode of recording storage information of the target resources by resource identification, wherein each target disk file is used for storing a plurality of resources requested to be stored into the database, writing the plurality of resources into the same target disk file in an additional writing mode, and not generating resource files by taking the resources as a unit, so that the target resources requested to be stored by received storage requests are sequentially written into the database, and therefore, sequential reading and writing are used for replacing random reading and writing, the purpose of improving data processing performance is achieved, the technical effect of improving the processing efficiency of the resources is achieved, and the technical problem that the processing efficiency of the resources in the related technology is low is solved.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and together with the description, serve to explain the principles of the invention.

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious for those skilled in the art that other drawings can be obtained according to the drawings without inventive exercise.

FIG. 1 is a schematic diagram of a hardware environment for a method of processing resources according to an embodiment of the application;

FIG. 2 is a flow chart of an alternative resource handling method according to an embodiment of the present application;

FIG. 3 is a flow diagram of an alternative resource handling method according to an embodiment of the application;

FIG. 4 is a schematic diagram of an alternative resource handling device according to an embodiment of the present application;

FIG. 5 is a schematic diagram of another alternative resource handling device according to an embodiment of the application;

fig. 6 is a block diagram of a terminal according to an embodiment of the present application.

Detailed Description

In order to make the technical solutions better understood by those skilled in the art, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only partial embodiments of the present application, but not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.

It should be noted that the terms "first," "second," and the like in the description and claims of this application and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the application described herein are capable of operation in sequences other than those illustrated or described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.

According to an aspect of embodiments of the present application, an embodiment of a method for processing a resource is provided.

Alternatively, in the present embodiment, the processing method of the resource may be applied to a hardware environment formed by the terminal 101 and the server 103 as shown in fig. 1. As shown in fig. 1, a server 103 is connected to a terminal 101 through a network, which may be used to provide services (such as game services, application services, etc.) for the terminal or a client installed on the terminal, and a database may be provided on the server or separately from the server for providing data storage services for the server 103, and the network includes but is not limited to: the terminal 101 is not limited to a PC, a mobile phone, a tablet computer, and the like. The resource processing method in the embodiment of the present application may be executed by the server 103, the terminal 101, or both the server 103 and the terminal 101. The processing method of the terminal 101 for executing the resource according to the embodiment of the present application may be executed by a client installed thereon.

Fig. 2 is a flowchart of an alternative resource processing method according to an embodiment of the present application, and as shown in fig. 2, the method may include the following steps:

step S202, receiving a storage request, wherein the storage request is used for requesting to store a target resource into a database, and the storage request carries a resource identifier and a resource address of the target resource;

step S204, a target disk file is obtained from the database in response to the storage request, wherein the database is set to allow creation of one or more target disk files, and each target disk file is used for storing a plurality of resources requested to be stored in the database;

step S206, downloading the target resource from the resource address;

and step S208, writing the target resources into the target disk file in sequence, and recording the storage information of the target resources according to the resource identification.

Through the steps S202 to S208, the database is set to allow creation of one or more target disk files, each target disk file is used to store a plurality of resources requested to be stored in the database, and by writing the plurality of resources in the same target disk file in an additional writing manner, resource files are not generated in units of resources, so that the target resources requested to be stored by the received storage request are sequentially written in the database, and thus sequential reading and writing are used instead of random reading and writing, the purpose of improving data processing performance is achieved, the technical effect of improving the processing efficiency of resources is achieved, and the technical problem of low processing efficiency of resources in the related art is solved.

Optionally, in this embodiment, the processing method of the resource may be, but is not limited to being, executed by the server. Such as: if the target resource is an image resource, the processing method of the resource can be, but is not limited to, being executed by an image storage server.

In the technical solution provided in step S202, the target resource may include, but is not limited to, any form of resource, such as: audio assets, video assets, image assets, text assets, and the like.

Optionally, in this embodiment, the storage request may be, but is not limited to, sent by a client.

Optionally, in this embodiment, the resource identifier may include, but is not limited to: resource id (identity), such as: an image ID of the image resource. The resource address may be, but is not limited to, a storage address of the target resource, such as: image URL (Uniform Resource Locator) of image Resource.

In the technical solution provided in step S204, the database is configured to allow creation of one or more target disk files, each target disk file being used for storing a plurality of resources requested to be stored in the database. The target disk files may be created one by one, but not limited to, each target disk file presets a storage space with a certain storage amount. When the storage space of one target disk file is used up, the next target disk file is created.

Optionally, in this embodiment, the size of each target disk file may be, but is not limited to, 10M.

In the technical solution provided in step S206, the resource address is used to indicate a download address of the target resource, and the target resource is downloaded by accessing the resource address.

Optionally, in this embodiment, the target resource may be, but is not limited to, downloaded into the socket buffer.

In the technical solution provided in step S208, the target resource is stored in the target disk file by using a sequential write method, and for the ordinary disk, the performance of sequentially writing the 4kb data block can reach 80MB/S, while the random write performance is only 0.2 MB/S. Thereby improving the read-write performance of the database.

Optionally, in this embodiment, the storage information of the target resource is recorded according to the resource identifier, so that the target resource can be quickly acquired according to the resource identifier when the target resource needs to be read.

As an alternative embodiment, the obtaining the target disk file from the database in response to the storage request includes:

s11, determining whether a disk file is established in the database;

s12, determining whether the established disk file has a write space under the condition that the disk file is determined to be established in the database;

s13, determining the established disk file as the target disk file under the condition that the write space of the established disk file is determined;

and S14, under the condition that the disk file is not established in the database or the established disk file has no writing space, allocating a storage space with a target data size from the database as the target disk file.

Optionally, in this embodiment, when the image storage request is received, it is determined whether the target disk file is currently established. And if the target disk file is established, judging whether the target disk file has writing space. And if the write space exists, continuing to use the target disk file. If the target disk file does not exist or the target disk file does not have enough writing space, a new target disk file of 10M is pre-allocated to be used.

As an alternative embodiment, sequentially writing the target resource to the target disk file includes:

s21, copying the target resource to a first cache page of the database;

and S22, writing the data cached on the first cache page into the target disk file in sequence at intervals of first time.

Downloading resources to a memory in the original resource storage scheme, and then writing the resources into a disk. However, since the operating system is divided into a kernel mode and a user mode, resource downloading in a user process can be downloaded to a user memory, the user memory cannot directly interact with a hard disk, and data needs to be copied from the user memory to the kernel memory and then written into a disk. An unnecessary memory copy exists in the middle, and the cpu and memory resources are consumed.

Optionally, in this embodiment, in order to improve the processing performance of the resource, the target resource may be written in a memory mapping manner, but is not limited to the memory mapping manner.

Optionally, in this embodiment, a file is mapped to an address space of a user process by using a first cache page, so as to implement a one-to-one correspondence between a file disk address and a segment of virtual address in a virtual address space of the user process. After the corresponding relationship is realized, the user process can read and write the section of the memory by adopting a pointer mode, and the system can automatically write back a dirty page (namely, a first cache page caching data) to a corresponding target disk file, namely, the operation on the resource is completed, and the memory copy of the resource between the user space and the kernel space is reduced.

Optionally, in this embodiment, but not limited to, the fsyncdata method is called at a timing of a background task to sequentially write the data cached on the first cache page into the target disk file.

As an alternative embodiment, copying the target resource to the first cache page of the database comprises:

s31, mapping the target resource to a target virtual address;

s32, performing read-write operation on the target virtual address;

s33, copying the target resource with the mapping relation with the target virtual address to the first cache page in response to the read-write operation.

Optionally, in this embodiment, a one-to-one mapping relationship between a file physical address (target resource) and a process virtual address (target virtual address) may be implemented, but is not limited to, by calling a system call function mmap of a kernel space.

In an alternative embodiment, the downloaded image resource may be written into the target disk file by, but not limited to, using a memory mapping method through the following processes:

step 1, calling a system calling function mmap of a kernel space to realize the one-to-one mapping relation between a file physical address and a process virtual address

And step 2, initiating access to the file, judging whether the cache page exists in the system (the cache page is the kernel memory), and copying the file content to the kernel memory. If not, the page fault exception is triggered.

And 3, copying the read socket buffer network data to a cache page by the system, downloading the picture to the server, and not writing the picture into a disk.

And 4, regularly calling the fsyncdata method by the subsequent background task to write the cache page data into the disk file, so as to avoid data loss. In the step, the random writing of the disk for many times is converted into the sequential writing once, so that the writing efficiency of the disk is greatly improved.

As an alternative embodiment, the recording the storage information of the target resource according to the resource identifier includes:

s41, generating the storage information, wherein the storage information includes: the resource identifier, the address of the target disk file and the storage location of the target resource, wherein the storage location is used for indicating the storage starting location and the storage ending location of the target resource in the target disk file;

and S42, inserting the storage information into the target resource information file.

Optionally, in this embodiment, storage information such as a resource identifier, an address of the target disk file, and a storage location of the target resource is recorded in the target resource information file, so as to perform a subsequent reading operation on the stored target resource.

Optionally, in this embodiment, the storage location of the target resource is used to indicate a storage start location and a storage end location of the target resource in the target disk file, so that the target resource can be accurately located.

Optionally, in this embodiment, the storage information may further include, but is not limited to, a storage time, a resource address, and the like.

Such as: taking the image resource as an example, the storage information may include, but is not limited to: image id, image url, target disk file address, storage starting position, storage ending position, storage time and other information.

As an alternative embodiment, inserting the storage information into the target resource information file includes:

s51, determining the target resource information file corresponding to the resource identifier from a plurality of resource information files;

s52, determining the information position of the storage information in the target resource information file according to the resource identification and the data size of the storage information;

s53, inserting the storage information into the information position of the target resource information file.

In the current scheme of inserting storage information of resources into a resource information file, all the storage information is written into the same resource information file, all the storage information in the resource information file is sorted according to resource identifiers, and a binary search method is firstly used for finding the insertion position of the resources for insertion each time. Although the time complexity of each search is reduced to o (logn) by using the dichotomy method, the efficiency of searching the resource insertion position is low because of the existence of multiple random reads of the hard disk. Meanwhile, the resource data and the storage information are stored in different files, and the storage information is written after the resource data is written into the hard disk every time, so that the hard disk performs a random writing process every time, and the performance of the system is influenced.

Optionally, in this embodiment, it is considered that resource identifiers are incremented at a system level, and although the resource identifier sequence is changed during writing due to concurrency, the resource identifiers are still ordered within a period of time, so that the resource information file may be split into a plurality of small files i (i ═ 1,2, 3.. times.n), so as to obtain a plurality of resource information files, each small file stores a certain amount of storage information W, a first file stores storage information of the resource identifiers from 1 to W, a second file stores storage information of the resource identifiers from W +1 to 2W, and so on. The storage information in each file is ordered, and is sorted in an increasing order by the resource identification.

Optionally, in this embodiment, the size occupied by each piece of storage information may be, but is not limited to, defined as a fixed length, such as: the resource identifier occupies A bytes, the address of the target disk file occupies B bytes, the storage starting position occupies C bytes, the storage ending position occupies D bytes, and the storage information occupies M-A + B + C + D bytes. If the storage information further includes a resource address and a storage time, the resource address may occupy B bytes, and the storage time may occupy F bytes, so that the storage information occupies M + B + C + D + E + F bytes.

Alternatively, in this embodiment, the file location to which the storage information should be written may be calculated according to the resource identifier and the data amount (M value described above) of the storage information, for example, the storage information with the resource identifier of 3W +2 should be stored in the M +1 th byte to the 2 nd byte of the 3 rd resource information file.

s61, writing the storage information into a second cache page;

and S62, inserting the data cached on the second cache page into the target resource information file at intervals of a second time.

Alternatively, in this embodiment, the storage information may be written to the second cache page, rather than to the disk in real time. And the background timing task flushes the data on the second cache page into the disk at intervals of a second time. Thus, multiple random writes to the disk in the system are combined into one sequential write, thereby improving the performance of the system.

In an alternative embodiment, a storage process of the image resource is taken as an example. The image information of the image resource may be stored into the image information file by, but is not limited to, the following processes:

step a, splitting the picture information file into a plurality of small files i (i is 1,2, 3.. multidot.n), wherein each small file stores 10000 pieces of image information, the first file stores the image information of which the image id is 1-10000, the second file stores the image information of which the image id is 10001-. The image information in each file is ordered, in order of image id.

Step b, the size of the space occupied by each piece of image information is defined as a fixed length, wherein the image id occupies 8 bytes, the image url occupies 300 bytes, the disk file address occupies 300 bytes, the storage starting position occupies 4 bytes, the storage ending position occupies 4 bytes, the storage time occupies 8 bytes, and the total length of 624 bytes is occupied.

Step c, calculating the position of an image information file in which the image information should be written according to the image id, such as: the image information whose id is 30002 should be stored in the 625 th byte to the 1248 th byte of the 3 rd image information file.

And d, writing the image information into the second cache page, and not writing the image information into the disk in real time.

And e, the background timing task flushes the data on the second cache page into the disk every 30 s.

As an optional embodiment, after sequentially writing the target resource to the target disk file, the method further includes:

s71, mapping the resource identifier through a plurality of mapping functions to obtain a plurality of mapping values;

s72, obtaining a target location corresponding to each of the plurality of mapping values from a resource filter, where the resource filter is configured to record a resource existing in the database, a first character of a character in a location included in the resource filter indicates that the resource corresponding to the location exists, and a second character of the character in the location included in the resource filter indicates that the resource corresponding to the location does not exist;

s73, setting the character of the target position as the first character.

Optionally, in this embodiment, in order to avoid that the resource corresponding to the resource identifier needs to be searched from the resource information file each time whether the resource exists in the database or not is checked, an unnecessary disk read-write is performed. Therefore, the resource filter can be established in the memory, and the resource identifier which does not exist can be directly intercepted by the resource filter.

Optionally, in this embodiment, the resource filter may be updated each time a target resource is newly added, and the resource identifier corresponding to the target resource is set to exist in the resource filter.

Optionally, in this embodiment, because the number of resources is large, even if only all the resource identifiers on the server are stored, a large amount of memory is occupied, so in order to save memory overhead, a manner of mapping and setting for many times may be used instead of storing the resource identifiers.

Optionally, in this embodiment, the plurality of mapping functions may be, but is not limited to, a plurality of different hash functions, such as: 5 different hash functions.

Optionally, in this embodiment, the first character and the second character are different characters, such as: the first character is 1, the second character is 0; alternatively, the first character is 0 and the second character is 1. The first and second characters may be in different forms of numbers, letters, words, and the like.

For example: taking image resources as an example, image ids can be mapped through 5 different hash functions, so as to obtain 5 different hash values. The bitset position corresponding to the 5 hash values in the resource filter is set to 1 to indicate that the image id exists.

According to another aspect of embodiments of the present application, there is provided another method embodiment for processing of a resource.

Fig. 3 is a flowchart of another alternative resource processing method according to an embodiment of the present application, and as shown in fig. 3, the method may include the following steps:

step S302, receiving an acquisition request, wherein the acquisition request is used for requesting to acquire a target resource from a database, and the acquisition request carries a resource identifier of the target resource;

step S304, responding to the obtaining request, obtaining a target disk file corresponding to the resource identifier from one or more disk files created in the database, wherein the one or more disk files are used for additionally storing resources according to the received storage request;

step S306, the target resource is obtained from the target disk file;

step S308, the target resource is sent.

Through the steps S302 to S308, the database is set to allow creation of one or more target disk files, each target disk file is used to store a plurality of resources requested to be stored in the database, and by writing the plurality of resources in the same target disk file in an additional writing manner, resource files are not generated in units of resources, so that the target resources requested to be stored by the received storage request are sequentially written in the database, and thus sequential reading and writing are used to replace random reading and writing, the purpose of improving data processing performance is achieved, the technical effect of improving the processing efficiency of resources is achieved, and the technical problem of low processing efficiency of resources in the related art is solved.

In the technical solution provided in step S302, the target resource may include, but is not limited to, any form of resource, such as: audio assets, video assets, image assets, text assets, and the like.

Optionally, in this embodiment, the obtaining request may be, but is not limited to, sent by a client.

Optionally, in this embodiment, the resource identifier may include, but is not limited to: resource id (identity), such as: an image ID of the image resource.

In the technical solution provided in step S304, one or more disk files are created in the database, and each disk file is used for storing a plurality of resources requested to be stored in the database. The disk files may be created one by one, but not limited to, each disk file presets a storage space with a certain storage amount. When the storage space of one disk file is exhausted, the next disk file is created.

Optionally, in this embodiment, the target disk file where the target resource is located can be found according to the resource identifier of the target resource.

As an optional embodiment, the obtaining, in response to the obtaining request, a target disk file corresponding to the resource identifier from one or more disk files created in the database includes:

s81, determining a target resource information file corresponding to the resource identifier from a plurality of resource information files;

s82, determining the information position of the storage information corresponding to the resource identifier in the target resource information file according to the resource identifier and the data volume of the storage information;

s83, obtaining the storage information from the information location of the target resource information file, wherein the storage information includes: the resource identifier, the address of the target disk file and the storage location of the target resource, wherein the storage location is used for indicating the storage starting location and the storage ending location of the target resource in the target disk file;

and S84, acquiring the target disk file from the address of the target disk file.

Optionally, in this embodiment, a plurality of resource information files are created in the database to record the storage information of the resources. And positioning a target resource information file in which the storage information of the target resource is located according to the resource identifier.

Alternatively, in the present embodiment, the size occupied by each piece of storage information may be, but is not limited to, defined as a fixed length. Therefore, the information position of the storage information corresponding to the resource identifier in the target resource information file can be located according to the resource identifier and the data amount of the storage information.

Optionally, in this embodiment, the address of the target disk file is recorded in the storage information of the resource, so that the target disk file where the target resource is located is found according to the address of the target disk file.

As an optional embodiment, the obtaining the target resource from the target disk file includes:

s91, acquiring the target resource from the storage position of the target disk file;

and S92, copying the target resource to a preset cache.

Optionally, in this embodiment, the storage information of the resource further records a storage location of the target resource, so that the target resource in the target disk file is found according to the storage location of the target resource, and the target resource is copied to the preset cache.

Optionally, in this embodiment, the preset cache may include, but is not limited to, a kernel buffer.

As an alternative embodiment, the sending the target resource includes:

s101, acquiring the position and offset of the target resource in the preset cache;

s102, sending the target resource located on the position and the offset in the preset cache.

Optionally, in this embodiment, the target resource may be transmitted in a zero copy manner, but is not limited to this. After the zero-copy mode is used, data copying between the kernel and the application is avoided twice, and consumption of cpu and memory resources is saved.

For example: the DMA copy copies the disk data into the kernel buffer, and adds the position and offset of the data to be sent currently in the kernel buffer to the socket buffer. And the DMA gather copy directly transmits the data copy in the kernel buffer to the network card according to the position and the offset in the socket buffer.

As an optional embodiment, before acquiring, in response to the acquisition request, a target disk file corresponding to the resource identifier from one or more disk files created in the database, the method further includes:

s111, mapping the resource identifier through a plurality of mapping functions to obtain a plurality of mapping values;

s112, obtaining a character at a target position corresponding to each of the plurality of mapping values from a resource filter, where the resource filter is configured to record a resource existing in the database, a character at a position included in the resource filter is a first character indicating that the resource corresponding to the position exists, and a character at a position included in the resource filter is a second character indicating that the resource corresponding to the position does not exist;

and S113, under the condition that the characters on the target position are the first characters, responding to the acquisition request to acquire the target disk file corresponding to the resource identifier from one or more disk files established in the database.

Optionally, in this embodiment, a resource filter is established in the memory, and the resource filter is used for intercepting a non-existent resource identifier directly.

Optionally, in this embodiment, each time the acquisition request is received, the resource filter may be first searched to determine whether the resource identifier corresponding to the requested target resource exists in the resource filter. And if so, responding to the acquisition request to acquire the target disk file corresponding to the resource identifier from one or more disk files established in the database so as to acquire the target resource.

Optionally, in this embodiment, if all the characters in the target position are the first characters, it indicates that the resource identifier corresponding to the target resource exists in the resource filter.

In an alternative embodiment, the process of retrieving the image resource from the image storage server may include, but is not limited to, the steps of:

step A, receiving a request for obtaining an image, wherein the request comprises an image id.

And B, judging whether the image id exists according to the resource filter.

Step B1, map the image id through 5 different hash functions to obtain 5 different hash values.

And step B2, judging whether the bitset bit corresponding to the 5 hash values is 1.

In step B3, if not all are 1, it indicates that the image does not exist, and a NotFound error code is returned.

And step C, if the image resource exists, acquiring the storage position of the storage information of the image resource according to the image id, and if the image resource exists, storing the image resource in the 625 th byte to the 1248 th byte of the 3 rd resource information file, wherein the id is 30002.

And D, reading the storage position, and acquiring storage information (image id, image url, disk file address, storage starting position, storage ending position and storage time) for the image id.

And E, if the storage information is not read, indicating that the image does not exist, and returning a NotFound error code.

And F, if the storage information can be acquired, taking the storage position of the image resource, copying the image resource from the hard disk to the network card device in a zero-copy mode, and returning the image resource to the requester as response.

The present application further provides an optional embodiment, where the optional embodiment provides a read-write method for an image resource, where the read-write method for an image resource includes a writing process for an image resource and a reading process for an image resource.

Optionally, in this optional embodiment, the writing process of the image resource may include, but is not limited to, the following steps:

and step 11, receiving a storage request of the image, wherein the storage request comprises an image id and an image url.

And step 12, determining a target disk file of the image storage.

And step 13, downloading the image and writing the image into the target disk file by using a memory mapping technology.

Step 14, recording the address of the target disk file stored in the image and the start position and the end position stored in the target disk file, and generating the storage information of the image, including: image id, image url, target disk file address, storage starting position, storage ending position and storage time.

Step 15, inserting the storage information into the resource information file.

Step 16, the resource filter for recording the image id present in the database is updated.

Optionally, in this optional embodiment, the reading process of the image resource may include, but is not limited to, the following steps:

step 21, receiving an image acquisition request, wherein the acquisition request comprises an image id.

And step 22, judging whether the image id exists according to the resource filter.

And step 23, if the image exists, acquiring the storage position of the storage information according to the image id.

Step 24, reading the storage location, and obtaining the storage information for the image id, including: image id, image url, target disk file address, storage starting position, storage ending position and storage time.

And step 25, if the relevant storage information is not read, indicating that the image does not exist, and returning a NotFound error code.

And step 26, if the storage information can be acquired, taking the storage position of the image content, copying the image content from the hard disk to the network card device by using a zero-copy technology, and returning the image content to the requesting party as response.

By adopting the optional embodiment, the memory copy between the user mode and the kernel mode is avoided by adopting various technologies such as memory mapping, zero copy and the like, and the memory occupation is saved. The use of the bitset algorithm results in higher performance and lower memory consumption. Various technical means are adopted to ensure the sequential writing of the hard disk, and the writing performance of the hard disk is improved.

It should be noted that, for simplicity of description, the above-mentioned method embodiments are described as a series of acts or combination of acts, but those skilled in the art will recognize that the present application is not limited by the order of acts described, as some steps may occur in other orders or concurrently depending on the application. Further, those skilled in the art should also appreciate that the embodiments described in the specification are preferred embodiments and that the acts and modules referred to are not necessarily required in this application.

Through the above description of the embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but the former is a better implementation mode in many cases. Based on such understanding, the technical solutions of the present application may be embodied in the form of a software product, which is stored in a storage medium (e.g., ROM/RAM, magnetic disk, optical disk) and includes instructions for enabling a terminal device (e.g., a mobile phone, a computer, a server, or a network device) to execute the method according to the embodiments of the present application.

According to another aspect of the embodiment of the present application, there is also provided a processing apparatus for a resource, which is used for implementing the processing method for a resource. Fig. 4 is a schematic diagram of an alternative resource processing apparatus according to an embodiment of the present application, and as shown in fig. 4, the apparatus may include:

a first receiving module 42, configured to receive a storage request, where the storage request is used to request to store a target resource in a database, and the storage request carries a resource identifier and a resource address of the target resource;

a first obtaining module 44, configured to obtain a target disk file from the database in response to the storage request, where the database is configured to allow creation of one or more target disk files, and each target disk file is used to store a plurality of resources requested to be stored in the database;

a download module 46, configured to download the target resource from the resource address;

and a writing module 48, configured to sequentially write the target resources into the target disk file, and record storage information of the target resources according to the resource identifier.

It should be noted that the first receiving module 42 in this embodiment may be configured to execute step S202 in this embodiment, the first obtaining module 44 in this embodiment may be configured to execute step S204 in this embodiment, the downloading module 46 in this embodiment may be configured to execute step S206 in this embodiment, and the writing module 48 in this embodiment may be configured to execute step S208 in this embodiment.

It should be noted here that the modules described above are the same as the examples and application scenarios implemented by the corresponding steps, but are not limited to the disclosure of the above embodiments. It should be noted that the modules described above as a part of the apparatus may operate in a hardware environment as shown in fig. 1, and may be implemented by software or hardware.

Through the modules, the database is set to allow one or more target disk files to be created, each target disk file is used for storing a plurality of resources which are requested to be stored in the database, the resources are written in the same target disk file in an additional writing mode, the resource file is not generated by taking the resources as a unit, the target resources requested to be stored by the received storage request are sequentially written in the database, sequential reading and writing are used for replacing random reading and writing, the purpose of improving data processing performance is achieved, the technical effect of improving the processing efficiency of the resources is achieved, and the technical problem that the processing efficiency of the resources in the related technology is low is solved.

As an alternative embodiment, the first obtaining module includes:

a first determination unit, configured to determine whether a disk file has been established in the database;

a second determination unit, configured to determine whether there is a write space in the established disk file when it is determined that the disk file has been established in the database;

a third determining unit, configured to determine the established disk file as the target disk file when it is determined that the established disk file has a write space;

and the allocation unit is used for allocating the storage space of the target data volume from the database as the target disk file under the condition that the disk file is not established in the database or the established disk file has no writing space.

As an alternative embodiment, the writing module includes:

a copy unit, configured to copy the target resource to a first cache page of the database;

and the writing unit is used for sequentially writing the data cached on the first cache page into the target disk file at intervals of first time.

As an alternative embodiment, the copy unit is configured to:

mapping the target resource to a target virtual address;

performing read-write operation on the target virtual address;

As an alternative embodiment, the writing module includes:

a generating unit, configured to generate the storage information, where the storage information includes: the resource identifier, the address of the target disk file and the storage location of the target resource, wherein the storage location is used for indicating the storage starting location and the storage ending location of the target resource in the target disk file;

and the inserting unit is used for inserting the storage information into the target resource information file.

As an alternative embodiment, the insertion unit is configured to:

writing the storage information into a second cache page;

As an alternative embodiment, the apparatus further comprises:

the first mapping module is used for mapping the resource identifier through a plurality of mapping functions after the target resource is sequentially written into the target disk file, so as to obtain a plurality of mapping values;

a fourth obtaining module, configured to obtain a target location corresponding to each of the plurality of mapping values from a resource filter, where the resource filter is configured to record a resource existing in the database, a character in a location included in the resource filter is a first character indicating that the resource corresponding to the location exists, and a character in a location included in the resource filter is a second character indicating that the resource corresponding to the location does not exist;

and the setting module is used for setting the character of the target position as the first character.

According to another aspect of the embodiments of the present application, there is also provided another processing apparatus for a resource, which is used for implementing the processing method for a resource. Fig. 5 is a schematic diagram of another alternative resource processing apparatus according to an embodiment of the present application, and as shown in fig. 5, the apparatus may include:

a second receiving module 52, configured to receive an obtaining request, where the obtaining request is used to request to obtain a target resource from a database, and the obtaining request carries a resource identifier of the target resource;

a second obtaining module 54, configured to respond to the obtaining request to obtain a target disk file corresponding to the resource identifier from one or more disk files created in the database, where the one or more disk files are used to add a storage resource according to the received storage request;

a third obtaining module 56, configured to obtain the target resource from the target disk file;

a sending module 58, configured to send the target resource.

It should be noted that the second receiving module 52 in this embodiment may be configured to execute step S302 in this embodiment, the second obtaining module 54 in this embodiment may be configured to execute step S304 in this embodiment, the third obtaining module 56 in this embodiment may be configured to execute step S306 in this embodiment, and the sending module 58 in this embodiment may be configured to execute step S308 in this embodiment.

As an alternative embodiment, the second obtaining module includes:

a fourth determining unit, configured to determine, from multiple resource information files, a target resource information file corresponding to the resource identifier;

a fifth determining unit, configured to determine, according to the resource identifier and the data size of the storage information, an information position of the storage information corresponding to the resource identifier in the target resource information file;

a first obtaining unit, configured to obtain the storage information from the information location of the target resource information file, where the storage information includes: the resource identifier, the address of the target disk file and the storage location of the target resource, wherein the storage location is used for indicating the storage starting location and the storage ending location of the target resource in the target disk file;

and the second acquisition unit is used for acquiring the target disk file from the address of the target disk file.

As an alternative embodiment, the second obtaining unit is configured to:

and copying the target resource to a preset cache.

As an alternative embodiment, the sending module includes:

a third obtaining unit, configured to obtain a position and an offset of the target resource in the preset cache;

a sending unit, configured to send the target resource located in the position and the offset in the preset buffer.

As an alternative embodiment, the apparatus further comprises:

a second mapping module, configured to map the resource identifier through multiple mapping functions to obtain multiple mapping values before acquiring, in response to the acquisition request, a target disk file corresponding to the resource identifier from one or more disk files created in the database;

a fifth obtaining module, configured to obtain, from a resource filter, a character at a target location corresponding to each of the plurality of mapping values, where the resource filter is configured to record a resource existing in the database, a character at a location included in the resource filter is a first character and indicates that the resource corresponding to the location exists, and a character at a location included in the resource filter is a second character and indicates that the resource corresponding to the location does not exist;

and a sixth obtaining module, configured to, in a case that the characters at the target location are the first characters, respond to the obtaining request to obtain a target disk file corresponding to the resource identifier from one or more disk files created in the database.

It should be noted here that the modules described above are the same as the examples and application scenarios implemented by the corresponding steps, but are not limited to the disclosure of the above embodiments. It should be noted that the modules described above as a part of the apparatus may be operated in a hardware environment as shown in fig. 1, and may be implemented by software, or may be implemented by hardware, where the hardware environment includes a network environment.

According to another aspect of the embodiment of the present application, a server or a terminal for implementing the processing method of the resource is also provided.

Fig. 6 is a block diagram of a terminal according to an embodiment of the present application, and as shown in fig. 6, the terminal may include: one or more processors 601 (only one of which is shown), a memory 603, and a transmission device 605. as shown in fig. 6, the terminal may further include an input-output device 607.

The memory 603 may be used to store software programs and modules, such as program instructions/modules corresponding to the resource processing method and apparatus in the embodiment of the present application, and the processor 601 executes various functional applications and data processing by running the software programs and modules stored in the memory 603, that is, implements the above-mentioned resource processing method. The memory 603 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 603 may further include memory located remotely from the processor 601, which may be connected to the terminal through a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.

The above-mentioned transmission device 605 is used for receiving or sending data via a network, and may also be used for data transmission between a processor and a memory. Examples of the network may include a wired network and a wireless network. In one example, the transmission device 605 includes a Network adapter (NIC) that can be connected to a router via a Network cable and other Network devices to communicate with the internet or a local area Network. In one example, the transmission device 605 is a Radio Frequency (RF) module, which is used for communicating with the internet in a wireless manner.

Among them, the memory 603 is used to store an application program, in particular.

The processor 601 may call the application stored in the memory 603 through the transmission device 605 to perform the following steps:

downloading the target resource from the resource address;

By adopting the embodiment of the application, a resource processing scheme is provided. The database is set to allow one or more target disk files to be created, each target disk file is used for storing a plurality of resources requested to be stored in the database, the resources are written in the same target disk file in an additional writing mode, the resource file is not generated by taking the resource as a unit, the target resources requested to be stored by the received storage request are sequentially written in the database, sequential reading and writing are used for replacing random reading and writing, the purpose of improving data processing performance is achieved, the technical effect of improving the processing efficiency of the resources is achieved, and the technical problem that the processing efficiency of the resources in the related technology is low is solved.

Optionally, the processor 601 may also call an application program stored in the memory 603 through the transmission device 605 to execute the following steps:

acquiring the target resource from the target disk file;

and sending the target resource.

Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments, and this embodiment is not described herein again.

It can be understood by those skilled in the art that the structure shown in fig. 6 is only an illustration, and the terminal may be a terminal device such as a smart phone (e.g., an Android phone, an iOS phone, etc.), a tablet computer, a palm computer, and a Mobile Internet Device (MID), a PAD, etc. Fig. 6 is a diagram illustrating a structure of the electronic device. For example, the terminal may also include more or fewer components (e.g., network interfaces, display devices, etc.) than shown in FIG. 6, or have a different configuration than shown in FIG. 6.

Those skilled in the art will appreciate that all or part of the steps in the methods of the above embodiments may be implemented by a program instructing hardware associated with the terminal device, where the program may be stored in a computer-readable storage medium, and the storage medium may include: flash disks, Read-Only memories (ROMs), Random Access Memories (RAMs), magnetic or optical disks, and the like.

Embodiments of the present application also provide a storage medium. Alternatively, in this embodiment, the storage medium may be a program code for executing the processing method of the resource.

Optionally, in this embodiment, the storage medium may be located on at least one of a plurality of network devices in a network shown in the above embodiment.

Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps:

downloading the target resource from the resource address;

Optionally, in this embodiment, the storage medium is further configured to store program code for performing the following steps:

acquiring the target resource from the target disk file;

and sending the target resource.

Optionally, in this embodiment, the storage medium may include, but is not limited to: a U-disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a removable hard disk, a magnetic or optical disk, and other various media capable of storing program codes.

The above-mentioned serial numbers of the embodiments of the present application are merely for description and do not represent the merits of the embodiments.

The integrated unit in the above embodiments, if implemented in the form of a software functional unit and sold or used as a separate product, may be stored in the above computer-readable storage medium. Based on such understanding, the technical solution of the present application may be substantially implemented or a part of or all or part of the technical solution contributing to the prior art may be embodied in the form of a software product stored in a storage medium, and including instructions for causing one or more computer devices (which may be personal computers, servers, network devices, or the like) to execute all or part of the steps of the method described in the embodiments of the present application.

In the above embodiments of the present application, the descriptions of the respective embodiments have respective emphasis, and for parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments.

In the several embodiments provided in the present application, it should be understood that the disclosed client may be implemented in other manners. The above-described embodiments of the apparatus are merely illustrative, and for example, the division of the units is only one type of division of logical functions, and there may be other divisions when actually implemented, for example, a plurality of units or components may be combined or may be integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, units or modules, and may be in an electrical or other form.

The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.

The foregoing is only a preferred embodiment of the present application and it should be noted that those skilled in the art can make several improvements and modifications without departing from the principle of the present application, and these improvements and modifications should also be considered as the protection scope of the present application.

Claims

1. A method for processing resources, comprising:

downloading the target resource from the resource address;

2. The method of claim 1, wherein retrieving a target disk file from the database in response to the storage request comprises:

determining whether a disk file is established in the database;

3. The method of claim 1, wherein writing the target resource sequentially to the target disk file comprises:

copying the target resource to a first cache page of the database;

4. The method of claim 3, wherein copying the target resource to the first cache page of the database comprises:

mapping the target resource to a target virtual address;

performing read-write operation on the target virtual address;

5. The method of claim 1, wherein recording the storage information of the target resource according to the resource identifier comprises:

and inserting the storage information into a target resource information file.

6. The method of claim 5, wherein inserting the storage information into the target resource information file comprises:

7. The method of claim 5, wherein inserting the storage information into the target resource information file comprises:

writing the storage information into a second cache page;

8. The method of claim 1, wherein after sequentially writing the target resource to the target disk file, the method further comprises:

setting the character of the target position as the first character.

9. A method for processing resources, comprising:

acquiring the target resource from the target disk file;

and sending the target resource.

10. The method of claim 9, wherein obtaining the target disk file corresponding to the resource identifier from the one or more disk files created in the database in response to the obtaining request comprises:

and acquiring the target disk file from the address of the target disk file.

11. The method of claim 10, wherein obtaining the target resource from the target disk file comprises:

and copying the target resource to a preset cache.

12. The method of claim 11, wherein sending the target resource comprises:

acquiring the position and offset of the target resource in the preset cache;

13. The method according to claim 9, before acquiring, in response to the acquisition request, a target disk file corresponding to the resource identifier from one or more disk files created in the database, the method further comprising:

14. An apparatus for processing a resource, comprising:

15. An apparatus for processing a resource, comprising:

and the sending module is used for sending the target resource.

16. A storage medium, characterized in that the storage medium comprises a stored program, wherein the program when executed performs the method of any of the preceding claims 1 to 13.

17. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor executes the method of any of claims 1 to 13 by means of the computer program.