CN116974998A - Data file updating method, device, computer equipment and storage medium - Google Patents
Data file updating method, device, computer equipment and storage medium Download PDFInfo
- Publication number
- CN116974998A CN116974998A CN202310804593.6A CN202310804593A CN116974998A CN 116974998 A CN116974998 A CN 116974998A CN 202310804593 A CN202310804593 A CN 202310804593A CN 116974998 A CN116974998 A CN 116974998A
- Authority
- CN
- China
- Prior art keywords
- data file
- file
- information
- target data
- meta
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 77
- 238000003860 storage Methods 0.000 title claims abstract description 27
- 230000004048 modification Effects 0.000 claims abstract description 57
- 238000012986 modification Methods 0.000 claims abstract description 57
- 238000004590 computer program Methods 0.000 claims abstract description 24
- 238000012795 verification Methods 0.000 claims description 145
- 239000012634 fragment Substances 0.000 claims description 50
- 230000004044 response Effects 0.000 claims description 41
- 230000002829 reductive effect Effects 0.000 abstract description 7
- 239000002699 waste material Substances 0.000 abstract description 2
- 238000001514 detection method Methods 0.000 description 17
- 230000008569 process Effects 0.000 description 17
- 238000012545 processing Methods 0.000 description 17
- 238000004891 communication Methods 0.000 description 16
- 238000005516 engineering process Methods 0.000 description 14
- 238000010586 diagram Methods 0.000 description 11
- 239000000284 extract Substances 0.000 description 9
- 230000006870 function Effects 0.000 description 5
- 238000013507 mapping Methods 0.000 description 5
- 230000005540 biological transmission Effects 0.000 description 4
- 230000008859 change Effects 0.000 description 4
- 230000002159 abnormal effect Effects 0.000 description 3
- 238000004458 analytical method Methods 0.000 description 3
- 238000004364 calculation method Methods 0.000 description 3
- 238000013499 data model Methods 0.000 description 2
- 238000013500 data storage Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 230000000670 limiting effect Effects 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- 230000001360 synchronised effect Effects 0.000 description 2
- OKTJSMMVPCPJKN-UHFFFAOYSA-N Carbon Chemical compound [C] OKTJSMMVPCPJKN-UHFFFAOYSA-N 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 230000003203 everyday effect Effects 0.000 description 1
- 238000007667 floating Methods 0.000 description 1
- 229910021389 graphene Inorganic materials 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 230000002427 irreversible effect Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 230000006855 networking Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000036961 partial effect Effects 0.000 description 1
- 238000007781 pre-processing Methods 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/16—File or folder operations, e.g. details of user interfaces specifically adapted to file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/14—Details of searching files based on file metadata
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Library & Information Science (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application relates to a data file updating method, a device, computer equipment, a storage medium and a computer program product, wherein the method can be applied to the technical field of cloud computing, and comprises the following steps: acquiring a network address of a source data file from a metafile for managing the data file; acquiring file meta information of the source data file based on the network address; searching at least one field of a tag field or a modification field in the file meta-information; when the at least one field is found and the information matched with the at least one field does not exist in the database, determining a target data file according to the file meta-information in the source data file; and downloading the target data file and updating the local data file based on the target data file. By adopting the method, the updating efficiency of file updating can be improved, and the waste of resources is reduced.
Description
Technical Field
The present application relates to the field of computer technologies, and in particular, to a method and apparatus for updating a data file, a computer device, and a storage medium.
Background
With the development of internet technology, file updating plays an important role in the fields of software updating, website content updating, data synchronization and the like. The existing data file updating mode needs to download the complete source data file to the local, then compares the downloaded source data file with the local data file to perform updating detection, and when the updating detection result represents that the downloaded source data file is updated relative to the local data file, the local data file is replaced by the downloaded source data file, and the downloaded source data file is effective data; when the update detection result indicates that the downloaded source data file is not updated relative to the local data file, the local file is reserved, and the downloaded source data file is invalid data.
When the data file updating method is applied to a batch local data file updating scene, even if only part of all source data files are effective data, all source data files need to be downloaded locally, and because time and network resources are required for downloading the source data files, the downloaded source data files also need to consume storage space, so that the updating efficiency of the batch local data files is lower and resource waste is caused.
Disclosure of Invention
In view of the foregoing, it is desirable to provide a method, an apparatus, a computer device, and a storage medium for updating a data file, which can save resources and improve file updating efficiency.
In a first aspect, the present application provides a method for updating a data file. The method comprises the following steps:
acquiring a network address of a source data file from a metafile for managing the data file;
acquiring file meta information of the source data file based on the network address;
searching at least one field of a tag field or a modification field in the file meta-information;
when the at least one field is found and the information matched with the at least one field does not exist in the database, determining a target data file according to the file meta-information in the source data file;
and downloading the target data file and updating the local data file based on the target data file.
In a second aspect, the application further provides a device for updating the data file. The device comprises:
a network address acquisition module, configured to acquire a network address of a source data file from a metafile for managing the data file;
The file meta information acquisition module is used for acquiring the file meta information of the source data file based on the network address;
the field searching module searches at least one field of a tag field or a modified field in the file meta-information;
the target data file determining module is used for determining a target data file according to the file meta-information in the source data file when the at least one field is found and the information matched with the at least one field does not exist in the database;
and the updating module is used for downloading the target data file and updating the local data file based on the target data file.
In a third aspect, the present application also provides a computer device. The computer device comprises a memory storing a computer program and a processor which when executing the computer program performs the steps of:
acquiring a network address of a source data file from a metafile for managing the data file;
acquiring file meta information of the source data file based on the network address;
searching at least one field of a tag field or a modification field in the file meta-information;
When the at least one field is found and the information matched with the at least one field does not exist in the database, determining a target data file according to the file meta-information in the source data file;
and downloading the target data file and updating the local data file based on the target data file.
In a fourth aspect, the present application also provides a computer-readable storage medium. The computer readable storage medium having stored thereon a computer program which when executed by a processor performs the steps of:
acquiring a network address of a source data file from a metafile for managing the data file;
acquiring file meta information of the source data file based on the network address;
searching at least one field of a tag field or a modification field in the file meta-information;
when the at least one field is found and the information matched with the at least one field does not exist in the database, determining a target data file according to the file meta-information in the source data file;
and downloading the target data file and updating the local data file based on the target data file.
In a fifth aspect, the present application also provides a computer program product. The computer program product comprises a computer program which, when executed by a processor, implements the steps of:
acquiring a network address of a source data file from a metafile for managing the data file;
acquiring file meta information of the source data file based on the network address;
searching at least one field of a tag field or a modification field in the file meta-information;
when the at least one field is found and the information matched with the at least one field does not exist in the database, determining a target data file according to the file meta-information in the source data file;
and downloading the target data file and updating the local data file based on the target data file.
The above-described data file updating method, apparatus, computer device, storage medium, and computer program product, by acquiring a network address of a source data file from a metafile for managing the data file; when at least one field is found and the information matched with the at least one field does not exist in the database, the target data file updated relative to the local data file is determined in the source data file according to the file meta-information, only the updated target data file is downloaded, and the local data file is updated based on the target data file.
Drawings
FIG. 1 is an application environment diagram of a method of updating data files in one embodiment;
FIG. 2 is a flow chart of a method for updating a data file according to one embodiment;
FIG. 3 is a diagram of file mappings in one embodiment;
FIG. 4 is a flowchart illustrating a file meta-information obtaining step in one embodiment;
FIG. 5 is a flow chart of a content format verification step in one embodiment;
FIG. 6 is a flow chart of a download method determining step in one embodiment;
FIG. 7 is a flow chart of a file update detection step in one embodiment;
FIG. 8 is a diagram of file meta-information storage, in one embodiment;
FIG. 9 is a schematic diagram of an update system for data files in one embodiment;
FIG. 10 is a flowchart of another embodiment of a method for updating a data file;
FIG. 11 is a diagram of a data update page in one embodiment;
FIG. 12 is a diagram of a data update page in another embodiment;
FIG. 13 is a block diagram showing the structure of an updating apparatus for data files in one embodiment;
FIG. 14 is a block diagram showing an update apparatus for data files according to another embodiment;
fig. 15 is an internal structural view of a computer device in one embodiment.
Detailed Description
The present application will be described in further detail with reference to the drawings and examples, in order to make the objects, technical solutions and advantages of the present application more apparent. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the application.
The method for updating the data file provided by the embodiment of the application can be applied to the field of Cloud technology, wherein Cloud technology refers to a hosting technology for integrating hardware, software, network and other series resources in a wide area network or a local area network to realize calculation, storage, processing and sharing of data.
Cloud technology (Cloud technology) is based on the general terms of network technology, information technology, integration technology, management platform technology, application technology and the like applied by Cloud computing business models, and can form a resource pool, so that the Cloud computing business model is flexible and convenient as required. Cloud computing technology will become an important support. Background services of technical networking systems require a large amount of computing, storage resources, such as video websites, picture-like websites, and more portals. Along with the high development and application of the internet industry, each article possibly has an own identification mark in the future, the identification mark needs to be transmitted to a background system for logic processing, data with different levels can be processed separately, and various industry data needs strong system rear shield support and can be realized only through cloud computing.
Cloud computing (clouding) is a computing model that distributes computing tasks across a large pool of computers, enabling various application systems to acquire computing power, storage space, and information services as needed. The network that provides the resources is referred to as the "cloud". Resources in the cloud are infinitely expandable in the sense of users, and can be acquired at any time, used as needed, expanded at any time and paid for use as needed.
As a basic capability provider of cloud computing, a cloud computing resource pool (cloud platform for short, generally referred to as IaaS (Infrastructure as a Service, infrastructure as a service) platform) is established, in which multiple types of virtual resources are deployed for external clients to select for use.
The method for updating the data file provided by the embodiment of the application can be applied to an application environment shown in fig. 1. Wherein computer device 102 communicates with server 104 over a network. The data storage system may store data to be processed by the server 104, specifically may be a source data file, the data storage system may be integrated on the server 104, or may be placed on a cloud or other servers, the computer device 102 stores a local data file, the computer device 102 may be a terminal or a server, an update method of the data file may be executed by the computer device 102, and the computer device 102 obtains a network address of the source data file from a metafile for managing the data file; acquiring file meta information of a source data file based on a network address; searching at least one field of a tag field or a modification field in the file meta-information; when at least one field is found and the information matched with the at least one field does not exist in the database, determining a target data file according to the file meta-information in the source data file; and downloading the target data file and updating the local data file based on the target data file.
The terminal can be, but not limited to, various desktop computers, notebook computers, smart phones, tablet computers, internet of things devices and portable wearable devices, and the internet of things devices can be smart speakers, smart televisions, smart air conditioners, smart vehicle devices and the like. The portable wearable device may be a smart watch, smart bracelet, headset, or the like. The server may be an independent physical server, a server cluster or a distributed system formed by a plurality of physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDNs, basic cloud computing services such as big data and artificial intelligent platforms. The terminal and the server may be directly or indirectly connected through wired or wireless communication, and the present application is not limited herein.
In one embodiment, as shown in fig. 2, a method for updating a data file is provided, and the method is applied to the computer device in fig. 1 for illustration, and includes the following steps:
s202, acquiring the network address of the source data file from the metafile for managing the data file.
The metafile refers to a file or a data structure for managing a data file, which may store attributes, structures, version information and other information related to the data file, and may be a single file, a database table, a configuration file, a JSON file, an XML file, or the like, and the data file managed by the metafile may be a local data file, which is stored in a local system and needs to be updated and detected periodically, and the local data file may be any type of file, such as a text file, an image file, a video file, a program file, or the like.
The source data file refers to a data file provided by a network or other data sources in the file updating process, and is a source of a local data file, and the source data file can be processed and used as the local data file by downloading the source data file locally, and the source data file can be a file on a remote server or a file in other storage media, such as an FTP server, a cloud storage service and the like.
The source data files may be various types of files, such as up-to-date version of software files, content files of a website, remote backup files of data, etc., that are stored on a remote server and accessed and transmitted over a network.
It should be noted that in the conventional file update detection scheme, the local data file is generally compared with the source data file in full content to determine whether the local data file needs to be updated.
The network address refers to an access path or Uniform Resource Locator (URL) on a remote server that stores a file, for specifying the location of the file in the network. For example, one example network address may be: HTTPS:// sample.com/files/myfile.txt, where the protocol is HTTPS, host is sample.com, path is/files/myfile.txt.
Specifically, when the local data file needs to be updated, the computer device reads the locally stored metafile for managing the local data file, reads the source file identifier of the source data file corresponding to each local data file from the metafile, and obtains the network address of the corresponding source data file based on the source file identifier.
For example, a file mapping relationship between a file identifier of a local data file and a source file identifier of a source data file is stored in a metafile, a source file identifier corresponding to each file identifier is determined based on the file mapping relationship, and a network address corresponding to the source file identifier is obtained, where the network address is the network address of the source data file.
As shown in fig. 3, a file mapping table in one embodiment is shown, and the network address of the source data file corresponding to each local data file can be determined through the file mapping table.
S204, acquiring file meta information of the source data file based on the network address.
The file meta information is descriptive information about the target remote file itself, and is used to describe attributes, characteristics and other related information of the file, and provides various metadata about the file, such as size, type, creation date, modification date, owner, rights, file path, file hash value, version identification (ETag), MIME type, etc.
File Size (File Size) refers to the Size of storage space occupied by a File, typically expressed in Bytes (Bytes); the File Type (File Type) refers to the format or extension of a File, such as a text File (.txt), an image File (.jpg), or a video File (.mp 4); creation Date (Creation Date) refers to the Creation time or Date of the file; the modification date (Modification Date) refers to the time or date when the file was last modified; owner (Owner) refers to the Owner or creator of the file; rights (permission) refer to access rights settings of files, such as read, write, and execute rights; file Path (File Path) refers to the Path or location of a File in a File system; file Hash (File Hash) refers to a Hash value of the File content, such as MD5, SHA-1 or SHA-256, for checking the integrity of the File; ETag (Entity Tag) refers to an entity tag of a file for identifying version or content changes of the file; MIME Type (MIME Type) refers to the multipurpose Internet mail extension Type of a file for identifying the content Type of the file, such as text, image, audio, video, etc.
In one embodiment, S204 specifically includes the steps of: generating a meta information acquisition request of a source data file based on the network address; transmitting a meta information acquisition request to a source data server corresponding to the network address; the file meta information of the source data file returned by the source data server in response to the meta information acquisition request is received, so that file update detection can be performed based on the file meta information, complete source data file downloading is avoided, bandwidth and time can be saved, unnecessary resource occupation can be avoided, and update detection efficiency can be improved.
The meta information obtaining request is an HTTP request for obtaining file meta information of a source data file, specifically may be a HEAD request, which is a request method in the HTTP protocol, for obtaining header information of a resource from a server, without obtaining actual resource content, and compared with the GET request, the HEAD request only returns a response header, and does not include a response body.
The request line of the HEAD request is the same as the GET request and contains the method, URL and HTTP protocol version, the request header of the HEAD request contains the same request header as the GET request for delivering additional request information, the request body of the HEAD request is different from the GET request, the HEAD request has no request body because it only pays attention to the response header and does not acquire the actual resource content. Compared with the GET request, the HEAD request only returns a response HEAD because the actual resource content is not returned, so that the HEAD request can reduce the data transmission quantity and save bandwidth resources; the HEAD request can acquire the meta-information of the resource more quickly because the response body is not transmitted, especially for the case of large files or bandwidth limitation; because the HEAD request has no response body, the actual resource content does not need to be saved in the client, and the storage cost is reduced.
The following is an example format of a meta information acquisition request:
HEAD/path/to/file HTTP/1.1
Host:example.com
in the above example, "HEAD" is used to specify that the meta information acquisition request is made using the HEAD method; "/path/to/file" is used to specify a path or URL of a file for which meta information is to be acquired; "HTTP/1.1" is used to designate versions using the HTTP protocol; "Host" is used to specify the hostname or IP address of the target server.
The source data server refers to a remote server storing source data files, and the target server may be various types of servers, such as a Web server, a file server, or a data server, depending on the type and location of the source data files to be accessed.
In one embodiment, when there are multiple local data files, that is, when a batch of local data files need to be updated, the computer device may generate meta-information acquisition requests corresponding to each source data file, send each information acquisition request to each corresponding source data server in a parallel manner, and receive file meta-information of the source data file returned by each source data server in response to the received meta-information acquisition request, so that file update detection may be performed based on the file meta-information, avoiding downloading a complete source data file, saving bandwidth and time, and also avoiding unnecessary resource occupation, and improving efficiency of update detection.
In one embodiment, the process of receiving source file meta-information of a source data file returned by a source data server in response to a meta-information acquisition request by a computer device specifically includes the steps of: receiving a response message returned by the source data server in response to the meta-information acquisition request; extracting header information from the response message; when the status code in the header information indicates that the status of the meta information acquisition request is successful, the file meta information of the source data file is extracted from the header information, only the file meta information in the header information is extracted instead of downloading the complete source data file, so that the data transmission amount can be reduced, the file update detection can be performed based on the file meta information, the downloading of the complete source data file is avoided, the bandwidth and time can be saved, unnecessary resource occupation can be avoided, and the update detection efficiency can be improved.
Describing the above embodiment as an example, after receiving the response message, the computer device uses the response_status_code to obtain the status code from the response message, and if the status code is 200, it indicates that the request is successful, and may continue to extract the file meta information; if the status code is other value, the request is wrong, and the step of extracting the file meta information is not executed any more; the header information of the response contains meta information of the target remote file, and the response header is accessed by using the corresponding HTTP library, and a required meta information field is extracted, for example, response.headers [ 'Content-Length' ] may be used to obtain the file size, response.headers [ 'Last-Modified' ] may be used to obtain the modification date, ETag may be obtained, and response.headers [ 'ETag' ] may be used to obtain the ETag value.
One response message is shown below:
HTTP/2 200
content-type:application/xml
content-length:321799868
vary:Accept-Encoding
date:Mon,10Apr 2023 03:49:50GMT
accept-ranges:bytes
etag:"A79B12EFC512E145C98C24DFF4D88571"
last-modified:Mon,10Apr 2023 03:16:28GMT
wherein, "HTTP/2 200" is the status code of the response message, which indicates that the request of the meta information acquisition request is successful; "content-type" refers to the media type of the file, here "application/XML", indicating that the file is data in XML format; "content-length" refers to the length of the file, here 321799868 bytes, indicating that the file size is 321MB; "source" is used to indicate the policy of the cache, here "Accept-Encoding", meaning that the cache may be changed according to the Accept-Encoding header in different requests; "date" refers to the response date of the file, here "Mon,10Apr 2023 03:49:50GMT", indicating that the response time of the file is 2023, 4, 10, 03:49:50; "accept-ranges" means whether the server supports scope requests, here "bytes", meaning that the server supports per byte scope requests; "etag" refers to the physical tag of a file, which is used to identify the version of the file, here "A79B12EFC512E145C98C24DFF4D88571"; "last-modified" refers to the last modification date of the file, here Mon,10Apr 2023 03:16:28GMT, indicating that the last modification time of the file is 2023, 4, 10, 03:16:28. It will be appreciated that the computer device, upon receipt of the response message, may extract the file meta-information from the response message, such as the modification date "Mon,10Apr 2023 03:16:28GMT" and version identification "a79B12EFC512E145C98C24DFF4D88571" of the file.
As shown in fig. 4, in a file meta-information obtaining flow chart in an embodiment, a computer device may obtain a network address, that is, a file address, of a source data file from a meta-file for managing the data file, generate a HEAD request for obtaining the file meta-information of the source data file based on the file address, send the generated HEAD request to a source data server corresponding to the source data file, receive response information returned by the source data server, extract the file meta-information of the source data file from the response information, and store the obtained file meta-information, for example, write the obtained file meta-information into a local storage space, so as to perform operations such as updating detection of a local data file based on the stored file meta-information of the source data file.
S206, searching at least one field of a tag field or a modified field in the file meta-information.
Wherein the tag field is an entity tag (ETag) of the file for identifying a version or content change of the file; the modification field refers to the time or date when the file was last modified.
Specifically, for the file meta information of any one source data file, the computer device may search the field corresponding to the keyword from the file meta information based on the preset keyword.
For example, the keyword is "etag", the tag field "etag:" A79B12EFC512E145C98C24DFF4D88571 "can be found from the file meta-information, the keyword is" last-modified ", and the modification field" last-modified: mon,10Apr 2023 03:16:28GMT "can be found from the file meta-information.
S208, when at least one field is found and the information matched with the at least one field does not exist in the database, determining a target data file according to the file meta-information in the source data file.
The database refers to a database in a local system for storing local data files and local file meta information of the local data files, and the absence of information matched with the at least one field is that the value of the field corresponding to the at least one field in the database is different from the value of the at least one field.
For example, a modification field is found in the file meta information of the source data file a, the modification date (value of the modification field) of the source data file a is determined according to the modification field, the local file meta information of the local data file a is obtained in the database, the modification field is extracted in the local file meta information, the modification date (value of the modification field) of the local data file is determined based on the modification field, the modification date of the source data file a is compared with the modification date of the local data file a, if the modification date and the modification date are different, it is determined that there is no information matched with the modification field of the source data file a in the database, for example, the obtained modification date information of the source data file a is "2023 month 4 month 10 date 03:16:28", the modification date information of the local data file a is "2023 month 10 date 05:18:30", it is determined that there is no information matched with the modification field of the source data file a in the database, that is the modification date of the source data file a is more than the modification date of the local data file a, and if the modification date of the source data file a is characterized in that the database is updated by "2023 month 10 month 10:4:4".
It may be understood that, for meta file information of any source data file, after searching at least one field of a tag field or a modification field in the meta file information, the computer device may perform update detection on a local data file corresponding to the source data file based on the at least one field searched, where the update detection on the local data file refers to detecting whether the local data file is synchronous with the corresponding source data file, that is, detecting whether the source data file is updated with respect to the local data file.
Specifically, for any one source data file and a local data file corresponding to the source data file, after searching at least one field in file meta-information of the source data file, the computer device obtains local file meta-information of the local data file stored in the database, and when the searched at least one field is not matched with a corresponding field in the corresponding local file meta-information, that is, when information matched with the at least one field does not exist in the database, it is determined that the source data file is updated relative to the corresponding local data file, and the source data file is determined to be the target data file.
For example, a modification field is found in the file meta information of a certain source data file, the modification date of the source data file is determined according to the modification field, the local file meta information of the corresponding local data file is obtained in the database, the modification field is extracted from the local file meta information, the modification date of the local data file is determined based on the modification field, the modification date of the source data file is compared with the modification date of the local data file, if the modification date of the source data file is different from the modification date of the local data file, the source data file is determined to be updated relative to the local data file corresponding to the source data file, and the source data file is determined to be a target data file; for another example, a tag field is found in the file meta information of another source data file, the version identifier of the source data file is determined according to the tag field, the local file meta information of the corresponding local data file is obtained in the database, the tag field is extracted from the local file meta information, the version identifier of the local data file is determined based on the tag field, the version identifier of the source data file is compared with the version identifier of the local data file, if the version identifier of the source data file and the version identifier of the local data file are different, it is determined that the source data file is updated relative to the local data file corresponding to the source data file, and the source data file is determined to be the target data file.
When updating and detecting the batch of local data files, searching for file meta-information of batch of source data files corresponding to the batch of local data files, respectively searching for at least one field of a tag field or a modification field in each file meta-information, after searching for at least one field, acquiring a corresponding field in the corresponding file meta-information in a database, and determining whether the searched at least one field is matched with the corresponding field, if not, determining that the source data file corresponding to the meta-information is a target data file. For example, for 1000 source data files, if 300 source data files have at least one field of tag fields or modification fields and the database does not have information matching with the at least one field, it is determined that 300 source data files corresponding to the 3000 source data files are target data files.
S210, downloading the target data file and updating the local data file based on the target data file.
Specifically, after determining the target data file, the computer device may generate a data acquisition request based on the network address of the target data file, send the data acquisition request to the source data server where the target data file is located, and receive the target data file returned by the source data server in response to the data acquisition request, so as to complete downloading of the target data file, and replace the local data file corresponding to the target data file with the downloaded target data file, so as to implement updating of the local data file.
The data acquisition request may be an HTTP request, specifically, a GET request, which is a request method for acquiring a resource from a server, where the GET request may append parameters and data to a query string of a network address of a target data file, so that a source data server of the target data file may return a corresponding resource according to the requested parameters.
In one embodiment, before updating the local data file based on the target data file, the computer device further needs to perform content format verification on the target data file to obtain a verification result, when the verification result is that the verification is passed, the local data file corresponding to the target data file is replaced by the downloaded target data file, so that the local data file is updated, and the content of the downloaded file can be ensured to meet the expected format requirement by performing content format verification on the target data file, so that the quality and accuracy of the data can be ensured, and the subsequent processing problem or the generation of an error result caused by data format errors can be avoided.
Specifically, the computer device may obtain entity information of at least one entity object in the target number file, obtain content format standard information, and perform format verification on the entity information of at least one entity object based on the content format standard information to obtain a verification result, and when the verification result is that the verification result is passed, that is, the content format of the target data file accords with the content format standard, replace the local data file corresponding to the target data file with the downloaded target data file, so as to implement updating of the local data file, ensure accuracy and quality of the data format, maintain consistency of data, optimize a data updating process, and improve data processing efficiency, thereby providing more reliable and efficient data updating and processing.
The content format standard information refers to related information of the content format standard followed by the target data file, and includes rules such as field definition, data type, length limitation and the like. An entity object refers to a specific, physically meaningful thing or concept. The physical objects may be objects, concepts, roles, entities, etc. in the real world, or may be business objects in a software system, entities in a data model, etc. Entity information of an entity object refers to specific data or attributes associated with the entity object, which describes characteristics, attributes or states of the entity object, and may include various data fields, attribute values, descriptive text, identifiers, etc., for example, the entity object may be a target product, i.e., the entity object is a specific product, and the entity information may include a product name, a product value, a product category, a product inventory number, etc. of the target product.
In one embodiment, the entity object comprises a target product, the entity information comprises a product name and a product value, the content format standard information comprises a name standard information and a value standard information, and the verification result comprises a name verification result and a value verification result; the computer equipment performs format verification on the entity information of at least one entity object based on the content format standard information to obtain a verification result, and the method comprises the following steps: performing format verification on the product name of at least one target product based on the name standard information to obtain a name verification result; and carrying out format verification on the product value of at least one target product based on the value standard information to obtain a value verification result, and checking whether the product name and the product value meet specific specifications, conventions or standards or not by carrying out format verification on the product name and the product value, so that format errors or abnormal data can be found at the early stage of data updating or processing, and the processing of data which does not meet format requirements in subsequent processing is reduced. This can improve the efficiency and accuracy of data processing.
It can be understood that when the name verification result representation verification is passed and the value verification result representation verification is passed, the computer device determines that the content format of the target data file accords with the content format label, and performs the step of replacing the local data file corresponding to the target data file with the downloaded target data file, thereby realizing the update of the local data file.
As shown in fig. 5, after acquiring the file meta information of the source data file, the computer device performs update detection on the local data file based on the file meta information of the source data file, and after determining the target data file in the source data file based on the file meta information, may acquire at least a part of the content of the target data file and acquire content format standard information, and after preprocessing at least a part of the content, extracts entity information of at least one entity object, for example, from at least one step of content, and performs content format verification on the entity information of at least one entity object by using the acquired content format standard information, so as to implement content format verification on the target data file.
In the above-mentioned data file updating method, the network address of the source data file is obtained from the metafile for managing the data file; when at least one field is found and the information matched with the at least one field does not exist in the database, the target data file updated relative to the local data file is determined in the source data file according to the file meta-information, only the updated target data file is downloaded, and the local data file is updated based on the target data file.
In one embodiment, the computer device may also extract a reception range field from file meta information of the target data file; the process of downloading the target data file by the computer equipment specifically comprises the following steps of: and downloading the target data file according to a downloading mode.
The receiving range field is a field for indicating a download mode supported by the source data server, for example, an "Accept-Ranges" field in file meta information. The downloading mode comprises a slice downloading mode and a stream downloading mode, wherein the slice downloading mode is a method for dividing a large file into a plurality of small fragments for downloading, and allows the parallel downloading of a plurality of file fragments, so that the downloading speed can be improved; the streaming download mode is a mode of processing while downloading, also called progressive download, in which received data can be processed and used during the download process.
Specifically, the computer equipment extracts a receiving range field from the file meta information, determines the value of the receiving range field, determines that the downloading mode of the target data file is a slice downloading mode when the value of the receiving range field is a first preset value, and downloads the target data file according to the slice downloading mode; and when the value of the receiving range field is a second preset value, determining that the downloading mode of the target data file is a streaming downloading mode, and downloading the target data file according to the streaming downloading mode.
For example, an "Accept-Ranges" field is extracted from the file meta-information, when the value of the "Accept-Ranges" field is "bytes", the token source server supports the piece-wise downloading according to the byte range, the downloading mode of the target data file is determined to be the piece-wise downloading mode, when the value of the "Accept-Ranges" field is "none", the token source server does not support the piece-wise downloading according to the byte range, and the downloading mode of the target data file is determined to be the streaming downloading mode.
As shown in fig. 6, after determining the target data file, the computer device may further determine whether the source data server corresponding to the target data file supports slice downloading based on the acquired file meta information of the target data file, and if so, may download the target data file of the specified data slice range from the source data server in a slice downloading manner, and store the downloaded content; if not, the target data file can be downloaded from the source data server in a streaming download mode, and a part of the previous content of the downloaded content is acquired and stored, so that the content format verification can be performed on the part of the stored content of the target data file.
In the above embodiment, the computer device extracts the receiving range field from the file meta information of the target data file, determines the downloading mode of the target data file according to the value of the receiving range field, and downloads the target data file according to the downloading mode, so that a proper mode can be selected according to the requirement to download the target data file, the downloading efficiency is improved, and the updating efficiency of the local data file is further improved.
In one embodiment, the process of downloading the target data file by the computer device in a download manner includes the steps of: when the downloading mode is a slice downloading mode, downloading at least one file segment of the target data file in the slice downloading mode; performing content format verification on at least one file fragment to obtain a verification result; and when the verification result is that the verification is passed, downloading the rest file fragments of the target data file.
Specifically, when the computer device determines that the downloading mode of the target data file is a fragmented downloading mode, at least one file fragment of the target data file is downloaded from the source data server of the target data file in the fragmented downloading mode, and content format verification is performed on the downloaded at least one file fragment to obtain a verification result, that is, the verification result of the content format verification of the target data file, and when the verification result is that the verification is passed, that is, the content format of the target data file meets the content format standard, the remaining file fragments of the target data file are continuously downloaded from the source data server of the target data file in the fragmented downloading mode, so that a complete target data file is obtained, and the local data file is updated based on the complete target data file.
It can be understood that when the verification result indicates that the verification is not passed, that is, the content format of the target data file does not meet the content format standard, so that the target data file cannot be used for updating the local data file, the downloading of the target data file is stopped, and the continuous downloading of invalid data is avoided, so that the accuracy of file updating can be ensured, and network resources are saved.
In the above embodiment, when the downloading mode is a piece-wise downloading mode, the computer device downloads at least one file piece of the target data file in the piece-wise downloading mode; performing content format verification on at least one file fragment to obtain a verification result; when the verification result is that the verification is passed, the residual file fragments of the target data file are downloaded, the content format verification is carried out on at least one file fragment, and partial data can be verified before the completion of the downloading, so that fragments with wrong or abnormal content formats can be found out early, unnecessary downloading is stopped in time when the verification is failed, and when the verification is passed, the residual file fragments of the target data file are continuously downloaded, so that the use of network bandwidth can be saved, the repeated downloading of the same file fragments is avoided, and the accuracy and the efficiency of updating the local data file can be improved.
In one embodiment, the process of downloading at least one file segment of the target data file by the computer device in a segment download manner specifically includes the steps of: sending a data acquisition request to a source data server of a target data file; the data acquisition request carries a file fragment range specified by a fragment downloading mode; the source data server receiving the target data file returns at least one file fragment of the target data file according to the file fragment range.
Specifically, after determining that the source data server corresponding to the target data file supports the fragment downloading mode, the computer device may acquire the network address of the target data file, and generate a data acquisition request according to the network address of the target data file, where the data acquisition request carries a range field, where the range field is used to describe a file fragment range specified by the fragment downloading mode, after generating the data acquisition request, send the data acquisition request to the source data server corresponding to the target data file, receive a response message returned by the source data server of the target data file, extract a requested file fragment from the response message, and when more fragment contents of the target data file need to be downloaded, may re-execute the step of generating the data acquisition request.
For example, the request header of the data acquisition request carries a "Content-Range" field, which has a value of "bytes 0-1024", indicating that a file fragment with bytes ranging from 0 to 1024 of the target data file is requested. By carrying a "Content-Range" field in the request header of the data acquisition request and setting an appropriate byte Range, the source data server of the target data file can be informed of the file fragment Range required by the source data server, and the source data server can return corresponding file fragment data according to the Range.
In the above embodiment, the computer device sends the data acquisition request to the source data server of the target data file; the data acquisition request carries a file fragment range specified by a fragment downloading mode; the source data server receiving the target data file returns at least one file fragment of the target data file according to the file fragment range, only the required file fragment but not the whole file can be requested by carrying the data acquisition request of the file fragment range, so that the use of network bandwidth can be saved, the data transmission quantity can be reduced, a plurality of file fragments can be downloaded in parallel by returning the file fragment according to the file fragment range, and the downloading speed can be improved.
In one embodiment, the process of verifying the content format of at least one file segment by the computer device to obtain a verification result specifically includes the following steps: acquiring content format standard information; extracting entity information of at least one entity object from at least one file segment; and carrying out format verification on the entity information of at least one entity object based on the content format standard information to obtain a verification result.
The content format standard information refers to related information of the content format standard followed by the target data file, and includes rules such as field definition, data type, length limitation and the like. An entity object refers to a specific, physically meaningful thing or concept. The physical objects may be objects, concepts, roles, entities, etc. in the real world, or may be business objects in a software system, entities in a data model, etc. Entity information of an entity object refers to specific data or attributes associated with the entity object, which describes characteristics, attributes or states of the entity object, and may include various data fields, attribute values, descriptive text, identifiers, etc., for example, the entity object may be a target product, i.e., the entity object is a specific product, and the entity information may include a product name, a product value, a product category, a product inventory number, etc. of the target product.
Specifically, after obtaining at least a part of file fragments of the target data file, the computer device may extract content format standard information corresponding to the source data file from the locally stored canonical document, analyze the downloaded at least one file fragment to obtain analysis content, extract entity information of at least one entity object from the analysis content, and then perform format verification on the extracted entity information according to the obtained content format standard information to obtain a verification result.
In the above embodiment, the computer device obtains the content format standard information; extracting entity information of at least one entity object from at least one file segment; and carrying out format verification on the entity information of at least one entity object based on the content format standard information to obtain a verification result, and comparing the verification result with the content format standard information to check whether the entity information accords with an expected format specification, so that errors, incompleteness or inconsistency in data can be eliminated, and the accuracy and reliability of the data are ensured.
In one embodiment, the entity object comprises a target product, the entity information comprises a product name and a product value, the content format standard information comprises a name standard information and a value standard information, and the verification result comprises a name verification result and a value verification result; the computer equipment performs format verification on the entity information of at least one entity object based on the content format standard information, and the process of obtaining a verification result specifically comprises the following steps: performing format verification on the product name of at least one target product based on the name standard information to obtain a name verification result; and carrying out format verification on the product value of at least one target product based on the value standard information to obtain a value verification result.
The name standard information includes a name length standard, a name character set standard, and the like, wherein the name length standard is used for specifying a length limit of a product name, for example, the name length is not more than 128 characters, the name character set is used for specifying a character set allowed by the product name, for example, the name character set standard is a UTF-8 character set, which indicates that the product name should contain characters conforming to a UTF-8 coding specification. The value criterion information includes a data type criterion for specifying a data type of the product value, such as a data type criterion that may be defined as an Integer (intelger), a floating point number (Float), or a monetary value (Currency), etc., a value range criterion for specifying a value range allowed by the product value, such as a value range criterion that may specify the product value to be between 0 and 100, etc.
Specifically, after obtaining the name standard information, the computer equipment performs format verification on the product name of at least one target product according to rules and requirements defined by the name standard information to obtain a name verification result; and after the value standard information is obtained, carrying out format verification on the product value of at least one target product according to rules and requirements defined by the value standard information to obtain a value verification result.
It can be understood that when the product names of the target products all meet the rules and requirements defined by the name standard information, the name verification result is verification passing; when the product name of the target product does not accord with the rule and the requirement defined by the name standard information, the name checking result is that the checking is not passed; when the product value of each target product accords with the rule and the requirement defined by the value standard information, the value verification result is verification passing, and when the product value of the target product does not accord with the rule and the requirement defined by the value standard information, the value verification result is not passing. In addition, the format verification of at least one file fragment is determined to pass only when the name verification result and the value verification result of each target product are both verification passes, and the format verification of at least one file fragment is determined to not pass when at least one of the name verification result and the value verification result of the target product is not passed.
In the above embodiment, the computer device performs format verification on the product name of at least one target product based on the name standard information, to obtain a name verification result; and carrying out format verification on the product value of at least one target product based on the value standard information to obtain a value verification result, and checking whether the product name and the product value meet specific specifications, conventions or standards or not by carrying out format verification on the product name and the product value, so that format errors or abnormal data can be found at the early stage of data updating or processing, and the processing of data which does not meet format requirements in subsequent processing is reduced. This can improve the efficiency and accuracy of data processing.
In one embodiment, the process of downloading the target data file by the computer device in a downloading manner specifically includes the following steps: when the downloading mode is a streaming downloading mode, downloading the target data file based on the streaming downloading mode; when downloading of at least part of data content in the target data file is completed, performing content format verification on at least part of data content to obtain a verification result; and when the verification result is that the verification is passed, continuing to download the residual data content of the target data file in a streaming download mode.
Specifically, when the computer device determines that the downloading mode of the target data file is the streaming downloading mode, downloading the data stream of the target data file from the source data server of the target data file in a lost downloading mode to obtain at least a part of data content in the target data file, and performing content format verification on the downloaded at least a part of data content to obtain a verification result, namely, a verification result of the content format of the target data file, wherein the computer device still continuously receives the data stream of the target data file transmitted by the source data server in the process of performing content format verification on the downloaded at least a part of data content, and when the verification result characterizes the verification, the computer device continuously receives the data stream of the target data file transmitted by the source data server until the downloading of the residual data content of the target data file is completed, so as to obtain a complete target data file, so as to update the local data file based on the complete target data file.
It can be understood that, in the process of verifying the content format of at least a part of the downloaded data content, the computer device is still continuously receiving the data stream of the target data file transmitted by the source data server, and when the verification result indicates that the verification is not passed, that is, the content format of the target data file is not in accordance with the content format standard, at this time, the computer device stops receiving the data stream of the target data file transmitted by the source data server, so as to avoid continuously downloading invalid data, thereby ensuring the accuracy of file update and saving network resources.
In the above embodiment, when the downloading mode is a streaming downloading mode, the computer device downloads the target data file based on the streaming downloading mode; when downloading of at least part of data content in the target data file is completed, content format verification is carried out on at least part of data content to obtain a verification result, when the verification result is that the verification is passed, the residual data content of the target data file is continuously downloaded in a streaming downloading mode, and the verification can be carried out in real time in the downloading process through streaming downloading, so that bandwidth can be saved, waiting time can be reduced, resource utilization efficiency can be improved, unnecessary downloading can be stopped in time when the verification is not passed, the residual content of the target data file is continuously downloaded when the verification is passed, thus the use of network bandwidth can be saved, repeated downloading of the same file content can be avoided, and the accuracy and efficiency of updating the local data file can be improved.
In one embodiment, the process of downloading the target data file by the computer device based on the streaming download method comprises the following steps: after establishing a communication connection with a source data server of a target data file, sending a data acquisition request to the source data server of the target data file; the data acquisition request is used for requesting to transmit the target data file in a streaming manner; a data stream of a target data file transmitted by a source data server in response to a data acquisition request is received over the established communication connection.
Specifically, after determining that the source data server corresponding to the target data file supports the streaming download mode, the computer device may acquire a network address of the target data file, establish a communication connection with the source data server by using a preset communication protocol based on the network address, and generate a data acquisition request according to the network address of the target data file, where the data acquisition request carries a specified field, where the specified field is used to instruct the source data server to transmit data in the streaming mode and the data acquisition request, and after generating the data acquisition request, send the data acquisition request to the source data server corresponding to the target data file, and receive, through the established communication connection, the source data server corresponding to the target data file to transmit the data stream of the target data file.
The preset communication protocol may be HTTP, FTP, or the like, and the established communication connection may be a long connection so as to continuously receive the data stream. The specified field may be an "Accept" field for instructing the server to stream data.
In the above embodiment, after establishing a communication connection with a source data server of a target data file, the computer device sends a data acquisition request to the source data server of the target data file; the data acquisition request is used for requesting to transmit the target data file in a streaming manner; the data stream of the target data file transmitted by the source data server in response to the data acquisition request is received through the established communication connection, so that real-time verification can be performed while data is received, and incomplete or erroneous data transmission is avoided.
It should be noted that, the process of performing the content format check on the downloaded at least a portion of the data content by the computer device may be the same as the process of performing the content format check on the at least one file segment in the foregoing embodiment, which is not described herein.
In one embodiment, the method for updating a data file further includes the following steps: downloading the source data file when the tag field and the modification field do not exist in the metafile information; determining the information abstract value of the source data file; when the information abstract value of the local data file matched with the information abstract value does not exist in the database, selecting a target data file from the source data file, and updating the local data file based on the selected target data file.
It should be noted that, some source data servers may not support providing file meta-information, or the provided file meta-information does not include a tag field and a modification field, and in this case, the embodiment of the present application further provides a solution for detecting file update based on the information digest value of the file.
The information abstract value is a fixed-length character string obtained by abstract calculation of the file. The message digest value may be calculated using a hash function including MD5, SHA-1, SHA-256, and the like. It should be noted that the process of calculating the information digest value is irreversible, i.e. the original data cannot be restored from the digest value, and for the same input data, the information digest value obtained by the same hash function calculation should be the same, and even a small change in the input data will result in a completely different digest value. The target data file is at least one of the source data files. For example, there are 1000 source data files, 300 target data files, or 1000 source data files, and 1000 target data files.
Specifically, when determining that a tag field and a modification field do not exist in file meta information of a certain source data file, the computer device downloads the source data file from a source data server corresponding to the source data file, calculates an information abstract value of the source data file based on a preset hash function after the downloading is completed, obtains a local information abstract value of a local file corresponding to the source data file from a database, compares the information abstract value of the source data file with the obtained local information abstract value, if the information abstract value is different from the obtained local information abstract value, the information abstract value of the local data file matched with the information abstract value does not exist in the database, determines that the source data file corresponding to the information abstract value is a target data file, and updates the local data file based on the target data file.
When the local data files in the batch are updated, the information abstract values of the source data files in the batch corresponding to the local data files in the batch are obtained respectively, and when the information abstract value of a certain source data file is different from the information abstract value of the corresponding local data file, the source data file is determined to be the target data file. For example, for 1000 information abstract values of 1000 source data files, if 300 information abstract values do not have the information abstract value of the local data file matched with the information abstract value in the database, determining 300 source data files corresponding to the 300 information abstract values as target data files; if 1000 information abstract values do not exist in the database, the 1000 source data files corresponding to the 1000 information abstract values are determined to be target data files.
In the above embodiment, the computer device downloads the source data file when the tag field and the modification field are not present in the meta file information; determining the information abstract value of the source data file; when the information abstract value of the local data file matched with the information abstract value does not exist in the database, selecting a target data file from the source data file, and updating the local data file based on the selected target data file, namely, quickly determining which files need to be updated based on the information abstract value, avoiding byte-by-byte comparison of file contents to realize update detection, and further improving the update efficiency of the local data file.
In one embodiment, the method for updating a data file further includes the following steps: and after the local data file is updated, updating the file meta information stored in the database based on the file meta information of the target data file.
Specifically, after the local data file is updated, that is, after the corresponding local data file is replaced by the target data file, the computer device obtains the file meta-information of the target data file, and replaces the file meta-information of the corresponding local data file stored in the database by using the target data file meta-information, so as to update the file meta-information of the local data file.
In addition, after the update of the local data file is completed, the computer device may further obtain the information digest value of the target data file, and replace the rest digest value of the corresponding local data file stored in the database with the information digest value of the target data file, so as to update the information digest value of the local data file.
As shown in fig. 7, for file meta information of any one source data file, the computer device obtains local file meta information of a local data file corresponding to the source data file from a file meta information database, searches at least one field of a tag field and a modification field in the file meta information of the source data file, compares the searched field with the obtained corresponding field in the local file meta information when at least one field is searched, determines a target data file based on the file meta information if the two fields are consistent, and updates the corresponding local file meta information in the file meta information database based on the file meta information of the target data file after updating the local data file based on the target data file; and when at least one field is not found, downloading the target data file, calculating the MD5 value of the target data file, acquiring the MD5 value of the local data file corresponding to the source data file from the file meta-information database, comparing the MD5 value of the target data file with the MD5 value of the acquired local data file, if the MD5 value of the target data file is consistent with the MD5 value of the local data file, updating the local data file based on the downloaded target data file, and updating the MD5 value of the corresponding local data file in the file meta-information database based on the MD5 value of the target data file after updating. In one embodiment, the related information corresponding to a local data file in the file meta information includes, in particular, a URL address of a source data file corresponding to the local data file, an MD5 value of the local data file, a latest modification date of the local data file, a version identifier of the local data file, a local storage address of the local data file, and so on, as shown in fig. 8.
In the above embodiment, after the update of the local data file is completed, the computer device updates the file meta information stored in the database based on the file meta information of the target data file, so as to ensure that the file meta information in the database is consistent with the actual state of the local data file, and avoid inconsistent or erroneous file meta information.
The embodiment of the application also provides an application scene, the application scene applies the updating method of the data file, the method can be applied to an updating system of the data file shown in figure 9, the system comprises a file meta information acquisition module, a file updating detection module, a file part content downloading module, a file content format verification module and a file meta information storage unit, wherein the function of each module is shown in a flow chart shown in figure 10, the file meta information acquisition module is used for acquiring url addresses of source data files, acquiring file meta information of the source data files based on url addresses of the source data files, judging whether at least one of etag and last modified is present in the file meta information after the file meta information of the source data files is obtained, if so, determining whether corresponding fields in the file meta information of the local data files corresponding to the database are consistent, if not, updating the file meta information of the local data files in the database based on the file meta information, downloading module is used for acquiring url addresses of the source data files, and writing at least one of the corresponding files in the database in the verification format verification module, and verifying the content of the file after the file meta information is downloaded at least partially, and the content of the file is verified and the content of the file is continuously downloaded based on the verification result is completed; if any one of etag and last modified does not exist in the file meta-information, downloading the source data file, calculating the MD5 value of the source data file, acquiring the MD5 value of the local data file corresponding to the source data file from the file meta-information database, comparing the MD5 value of the source data file with the MD5 value of the acquired local data file, if the MD5 value of the source data file and the MD5 value of the local data file are consistent, updating the local data file based on the downloaded source data file, and updating the MD5 value of the corresponding local data file in the file meta-information database based on the MD5 value of the source data file after updating.
The method for updating the data file provided by the application can be applied to a data file updating scene of intelligent wearable equipment, as shown in fig. 11, a page schematic diagram of an application program on a terminal corresponding to an intelligent bracelet is shown, when the application program of the terminal is started or a synchronous control in the page shown in fig. 11 is triggered, the terminal can acquire a network address of a source data file from a metafile for managing the bracelet data file, acquire file meta information of the source data file based on the network address, search at least one field in a tag field or a modified field in the file meta information, and determine a target data file according to the file meta information in the source data file when at least one field is searched and the information matched with the at least one field does not exist in a database; the target data file is downloaded and the local data file is updated based on the target data file, as shown in fig. 11, and sleep data and movement data may be specifically updated.
The method for updating the data file provided by the application can also be applied to a data file updating scene of enterprise operation data, as shown in fig. 12, at 8 o' clock every day, an operation terminal acquires a network address of a source data file from a metafile for managing a bracelet data file, acquires file meta information of the source data file based on the network address, searches at least one field in a tag field or a modified field in the file meta information, and determines a target data file according to the file meta information in the source data file when at least one field is searched and no information matched with the at least one field exists in a database; the target data file is downloaded, and the local data file is updated based on the target data file, as shown in fig. 12, specifically, the relevant operation data such as payment amount, access-payment conversion rate, payment order number, and the like can be updated.
It should be understood that, although the steps in the flowcharts related to the embodiments described above are sequentially shown as indicated by arrows, these steps are not necessarily sequentially performed in the order indicated by the arrows. The steps are not strictly limited to the order of execution unless explicitly recited herein, and the steps may be executed in other orders. Moreover, at least some of the steps in the flowcharts described in the above embodiments may include a plurality of steps or a plurality of stages, which are not necessarily performed at the same time, but may be performed at different times, and the order of the steps or stages is not necessarily performed sequentially, but may be performed alternately or alternately with at least some of the other steps or stages.
Based on the same inventive concept, the embodiment of the application also provides a data file updating device for realizing the above related data file updating method. The implementation of the solution provided by the device is similar to the implementation described in the above method, so the specific limitation in the embodiments of the update device for one or more data files provided below may refer to the limitation of the update method for the data file hereinabove, and will not be repeated herein.
In one embodiment, as shown in fig. 13, there is provided an updating apparatus of a data file, including: a network address acquisition module 1302, a file meta information acquisition module 1304, a field lookup module 1306, a target data file determination module 1308, and an update module 1310, wherein:
a network address obtaining module 1302 is configured to obtain a network address of a source data file from a metafile for managing the data file.
The file meta information obtaining module 1304 is configured to obtain file meta information of the source data file based on the network address.
The field lookup module 1306 looks up at least one of a tag field or a modified field in the file meta-information.
The target data file determining module 1308 is configured to determine, in the source data file, a target data file according to the file meta information when at least one field is found and there is no information matching the at least one field in the database.
And an updating module 1310, configured to download the target data file and update the local data file based on the target data file.
In the above-described embodiment, the network address of the source data file is acquired by from the metafile for managing the data file; when at least one field is found and the information matched with the at least one field does not exist in the database, the target data file updated relative to the local data file is determined in the source data file according to the file meta-information, only the updated target data file is downloaded, and the local data file is updated based on the target data file.
In one embodiment, the file meta-information acquisition module 1304 is further configured to: generating a meta information acquisition request of a source data file based on the network address; transmitting a meta information acquisition request to a source data server corresponding to the network address; and receiving file meta information of the source data file returned by the source data server in response to the meta information acquisition request.
In one embodiment, the file meta-information acquisition module 1304 is further configured to: receiving a response message returned by the source data server in response to the meta-information acquisition request;
extracting header information from the response message; when the status code in the header information indicates that the status of the meta information acquisition request is successful, the file meta information of the source data file is extracted from the header information.
In one embodiment, as shown in fig. 14, the apparatus further comprises: the download manner determining module 1312 is configured to extract a receiving range field from file meta information of the target data file; determining a downloading mode of the target data file according to the value of the receiving range field; the update module 1310 is further configured to download the target data file according to a downloading manner.
In one embodiment, the update module 1310 is further configured to: when the downloading mode is a slice downloading mode, downloading at least one file segment of the target data file in the slice downloading mode; performing content format verification on at least one file fragment to obtain a verification result; and when the verification result is that the verification is passed, downloading the rest file fragments of the target data file.
In one embodiment, the update module 1310 is further configured to: sending a data acquisition request to a source data server of a target data file; the data acquisition request carries a file fragment range specified by a fragment downloading mode; the source data server receiving the target data file returns at least one file fragment of the target data file according to the file fragment range.
In one embodiment, the update module 1310 is further configured to: acquiring content format standard information; extracting entity information of at least one entity object from at least one file segment; and carrying out format verification on the entity information of at least one entity object based on the content format standard information to obtain a verification result.
In one embodiment, the entity object comprises a target product, the entity information comprises a product name and a product value, the content format standard information comprises a name standard information and a value standard information, and the verification result comprises a name verification result and a value verification result; the update module 1310 is further configured to: performing format verification on the product name of at least one target product based on the name standard information to obtain a name verification result; and carrying out format verification on the product value of at least one target product based on the value standard information to obtain a value verification result.
In one embodiment, the update module 1310 is further configured to: when the downloading mode is a streaming downloading mode, downloading the target data file based on the streaming downloading mode; when downloading of at least part of data content in the target data file is completed, performing content format verification on at least part of data content to obtain a verification result; and when the verification result is that the verification is passed, continuing to download the residual data content of the target data file in a streaming download mode.
In one embodiment, the update module 1310 is further configured to: downloading the source data file when the tag field and the modification field do not exist in the metafile information; determining the information abstract value of the source data file; when the information abstract value of the local data file matched with the information abstract value does not exist in the database, selecting a target data file from the source data file, and updating the local data file based on the selected target data file.
In one embodiment, the update module 1310 is further configured to: and after the local data file is updated, updating the file meta information stored in the database based on the file meta information of the target data file.
The above-described respective modules in the data file updating apparatus may be implemented in whole or in part by software, hardware, and combinations thereof. The above modules may be embedded in hardware or may be independent of a processor in the computer device, or may be stored in software in a memory in the computer device, so that the processor may call and execute operations corresponding to the above modules.
In one embodiment, a computer device is provided, which may be a terminal, and an internal structure diagram thereof may be as shown in fig. 15. The computer device includes a processor, a memory, an input/output interface, a communication interface, a display unit, and an input means. The processor, the memory and the input/output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input/output interface. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The input/output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used for carrying out wired or wireless communication with an external terminal, and the wireless mode can be realized through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. The computer program is executed by a processor to implement a method of updating a data file. The display unit of the computer equipment is used for forming a visual picture, and can be a display screen, a projection device or a virtual reality imaging device, wherein the display screen can be a liquid crystal display screen or an electronic ink display screen, the input device of the computer equipment can be a touch layer covered on the display screen, can also be a key, a track ball or a touch pad arranged on a shell of the computer equipment, and can also be an external keyboard, a touch pad or a mouse and the like.
It will be appreciated by those skilled in the art that the structure shown in fig. 15 is merely a block diagram of a portion of the structure associated with the present inventive arrangements and is not limiting of the computer device to which the present inventive arrangements are applied, and that a particular computer device may include more or fewer components than shown, or may combine certain components, or have a different arrangement of components.
In an embodiment, there is also provided a computer device comprising a memory and a processor, the memory having stored therein a computer program, the processor implementing the steps of the method embodiments described above when the computer program is executed.
In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored which, when executed by a processor, carries out the steps of the method embodiments described above.
In an embodiment, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the steps of the method embodiments described above.
It should be noted that, the user information (including but not limited to user equipment information, user personal information, etc.) and the data (including but not limited to data for analysis, stored data, presented data, etc.) related to the present application are information and data authorized by the user or sufficiently authorized by each party, and the collection, use and processing of the related data need to comply with the related laws and regulations and standards of the related country and region.
Those skilled in the art will appreciate that implementing all or part of the above described methods may be accomplished by way of a computer program stored on a non-transitory computer readable storage medium, which when executed, may comprise the steps of the embodiments of the methods described above. Any reference to memory, database, or other medium used in embodiments provided herein may include at least one of non-volatile and volatile memory. The nonvolatile Memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash Memory, optical Memory, high density embedded nonvolatile Memory, resistive random access Memory (ReRAM), magnetic random access Memory (Magnetoresistive Random Access Memory, MRAM), ferroelectric Memory (Ferroelectric Random Access Memory, FRAM), phase change Memory (Phase Change Memory, PCM), graphene Memory, and the like. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory, and the like. By way of illustration, and not limitation, RAM can be in the form of a variety of forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), and the like. The databases referred to in the embodiments provided herein may include at least one of a relational database and a non-relational database. The non-relational database may include, but is not limited to, a blockchain-based distributed database, and the like. The processor referred to in the embodiments provided in the present application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, or the like, but is not limited thereto.
The technical features of the above embodiments may be arbitrarily combined, and all possible combinations of the technical features in the above embodiments are not described for brevity of description, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the description.
The foregoing examples illustrate only a few embodiments of the application and are described in detail herein without thereby limiting the scope of the application. It should be noted that it will be apparent to those skilled in the art that several variations and modifications can be made without departing from the spirit of the application, which are all within the scope of the application. Accordingly, the scope of the application should be assessed as that of the appended claims.
Claims (15)
1. A method of updating a data file, the method comprising:
acquiring a network address of a source data file from a metafile for managing the data file;
acquiring file meta information of the source data file based on the network address;
searching at least one field of a tag field or a modification field in the file meta-information;
when the at least one field is found and the information matched with the at least one field does not exist in the database, determining a target data file according to the file meta-information in the source data file;
And downloading the target data file and updating the local data file based on the target data file.
2. The method of claim 1, wherein the obtaining file meta-information of the source data file based on the network address comprises:
generating a meta information acquisition request of the source data file based on the network address;
sending the meta information acquisition request to a source data server corresponding to the network address;
and receiving file meta-information of the source data file returned by the source data server in response to the meta-information acquisition request.
3. The method of claim 2, wherein receiving the file meta-information of the source data file returned by the source data server in response to the meta-information retrieval request comprises:
receiving a response message returned by the source data server in response to the meta-information acquisition request;
extracting header information from the response message;
and when the state code in the header information indicates that the state of the meta-information acquisition request is successful, extracting file meta-information of the source data file from the header information.
4. The method according to claim 1, wherein the method further comprises:
extracting a receiving range field from file meta information of the target data file;
determining a downloading mode of the target data file according to the value of the receiving range field;
the downloading the target data file includes:
and downloading the target data file according to the downloading mode.
5. The method of claim 4, wherein said downloading said target data file in said download manner comprises:
when the downloading mode is a slice downloading mode, downloading at least one file fragment of the target data file in the slice downloading mode;
performing content format verification on the at least one file segment to obtain a verification result;
and when the verification result is that the verification is passed, downloading the rest file fragments of the target data file.
6. The method of claim 5, wherein said downloading at least one file segment of said target data file in said fragmented download comprises:
sending a data acquisition request to a source data server of the target data file; the data acquisition request carries a file fragment range specified by the fragment downloading mode;
And the source data server receiving the target data file returns at least one file fragment of the target data file according to the file fragment range.
7. The method of claim 5, wherein the verifying the content format of the at least one file segment to obtain a verification result comprises:
acquiring content format standard information;
extracting entity information of at least one entity object from the at least one file segment;
and carrying out format verification on the entity information of the at least one entity object based on the content format standard information to obtain a verification result.
8. The method of claim 7, wherein the entity object comprises a target product, the entity information comprises a product name and a product value, the content format standard information comprises a name standard information and a value standard information, and the verification result comprises a name verification result and a value verification result; the performing format verification on the entity information of the at least one entity object based on the content format standard information to obtain a verification result, including:
performing format verification on the product name of the at least one target product based on the name standard information to obtain a name verification result;
And carrying out format verification on the product value of the at least one target product based on the value standard information to obtain a value verification result.
9. The method of claim 4, wherein said downloading said target data file in said download manner comprises:
when the downloading mode is a streaming downloading mode, downloading the target data file based on the streaming downloading mode;
when downloading of at least one part of data content in the target data file is completed, performing content format verification on the at least one part of data content to obtain a verification result;
and when the verification result is that the verification is passed, continuing to download the residual data content of the target data file in the streaming download mode.
10. The method according to claim 1, wherein the method further comprises:
downloading the source data file when the tag field and the modification field are not present in the metafile information;
determining the information abstract value of the source data file;
and when the information abstract value of the local data file matched with the information abstract value does not exist in the database, selecting a target data file from the source data file, and updating the local data file based on the selected target data file.
11. The method according to any one of claims 1 to 10, further comprising:
and after the local data file is updated, updating the file meta-information stored in the database based on the file meta-information of the target data file.
12. An apparatus for updating a data file, the apparatus comprising:
a network address acquisition module, configured to acquire a network address of a source data file from a metafile for managing the data file;
the file meta information acquisition module is used for acquiring the file meta information of the source data file based on the network address;
the field searching module searches at least one field of a tag field or a modified field in the file meta-information;
the target data file determining module is used for determining a target data file according to the file meta-information in the source data file when the at least one field is found and the information matched with the at least one field does not exist in the database;
and the updating module is used for downloading the target data file and updating the local data file based on the target data file.
13. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor implements the steps of the method of any one of claims 1 to 11 when the computer program is executed.
14. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the steps of the method of any of claims 1 to 11.
15. A computer program product comprising a computer program, characterized in that the computer program, when being executed by a processor, implements the steps of the method of any one of claims 1 to 11.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310804593.6A CN116974998A (en) | 2023-06-30 | 2023-06-30 | Data file updating method, device, computer equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310804593.6A CN116974998A (en) | 2023-06-30 | 2023-06-30 | Data file updating method, device, computer equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116974998A true CN116974998A (en) | 2023-10-31 |
Family
ID=88478757
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310804593.6A Pending CN116974998A (en) | 2023-06-30 | 2023-06-30 | Data file updating method, device, computer equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116974998A (en) |
-
2023
- 2023-06-30 CN CN202310804593.6A patent/CN116974998A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9910895B2 (en) | Push subscriptions | |
| US7877682B2 (en) | Modular distributed mobile data applications | |
| US9195773B2 (en) | Structure-based adaptive document caching | |
| US11226944B2 (en) | Cache management | |
| US20120072465A1 (en) | Dynamic schema-based api mapping for traversing inter-cloud application boundaries | |
| US9716771B2 (en) | Cache control for web application resources | |
| US20130282797A1 (en) | Dynamic application programming interface | |
| KR20100133380A (en) | Computing environment representation | |
| EP2854048A1 (en) | Method of operating a mobile telecommunications device | |
| WO2021237467A1 (en) | File uploading method, file downloading method and file management apparatus | |
| WO2014089230A2 (en) | Method and system of storing data files and retrieving data in a data file and requesting information from a computer | |
| US9075722B2 (en) | Clustered and highly-available wide-area write-through file system cache | |
| US11138287B1 (en) | System for reducing size of transmitted content | |
| CN112433921A (en) | Method and apparatus for dynamic point burying | |
| WO2022057525A1 (en) | Method and device for data retrieval, electronic device, and storage medium | |
| CN108134811B (en) | Method, device and system for distributing or downloading target file | |
| CN116974998A (en) | Data file updating method, device, computer equipment and storage medium | |
| CN107103001B (en) | Method, device and system for acquiring target front-end resource file based on browser | |
| CN109857719B (en) | Distributed file processing method, device, computer equipment and storage medium | |
| CN115905120B (en) | Archive file management method, archive file management device, archive file management computer device and archive file management storage medium | |
| CN108763425B (en) | Method and apparatus for storing and reading audio files | |
| CN117971297A (en) | Method, apparatus, computer device and storage medium for packet dependent synchronization | |
| CN115080571A (en) | Index updating method and device, electronic equipment and computer readable medium | |
| CN116974984A (en) | Private cloud product document processing method and device and computer equipment | |
| CN114491324A (en) | Information pushing method and device, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication |