CN117082050A - Media asset file downloading method, device, electronic equipment and computer readable medium - Google Patents

Media asset file downloading method, device, electronic equipment and computer readable medium Download PDF

Info

Publication number
CN117082050A
CN117082050A CN202311023277.1A CN202311023277A CN117082050A CN 117082050 A CN117082050 A CN 117082050A CN 202311023277 A CN202311023277 A CN 202311023277A CN 117082050 A CN117082050 A CN 117082050A
Authority
CN
China
Prior art keywords
downloaded
file
media
media resource
distributed lock
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202311023277.1A
Other languages
Chinese (zh)
Inventor
董浩
李强
赵晓
刘金鹏
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Jingdong Technology Information Technology Co Ltd
Original Assignee
Jingdong Technology Information Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Jingdong Technology Information Technology Co Ltd filed Critical Jingdong Technology Information Technology Co Ltd
Priority to CN202311023277.1A priority Critical patent/CN117082050A/en
Publication of CN117082050A publication Critical patent/CN117082050A/en
Pending legal-status Critical Current

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/06—Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/50—Network services
    • H04L67/56—Provisioning of proxy services
    • H04L67/568—Storing data temporarily at an intermediate stage, e.g. caching
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/50—Network services
    • H04L67/60—Scheduling or organising the servicing of application requests, e.g. requests for application data transmissions using the analysis and optimisation of the required network resources

Abstract

The invention discloses a method and a device for downloading a media resource file, electronic equipment and a computer readable medium, and relates to the technical field of big data processing. One embodiment of the method comprises the following steps: pulling session records from a session server in a multithreading parallel mode, wherein each session record comprises a signature value of a media resource file to be downloaded; downloading the media resource file to be downloaded in the session record in a multithreading parallel mode; and for the media resource files to be downloaded in each session record, downloading the media resource files to be downloaded from the session server by a thread which acquires the distributed lock of the media resource files to be downloaded, uploading the media resource files to a file system for storage, obtaining the storage address of the media resource files to be downloaded, storing the storage address in a cache, and submitting the storage address of the media resource files to be downloaded to a quality inspection platform. The implementation mode can solve the technical problems of occupying resources such as application service network bandwidth, CPU calculation, equipment storage and the like.

Description

Media asset file downloading method, device, electronic equipment and computer readable medium
Technical Field
The present invention relates to the field of big data processing technologies, and in particular, to a method and apparatus for downloading a media file, an electronic device, and a computer readable medium.
Background
The chat contents of operators and clients can be stored in a mode of pulling the chat record of the instant messaging session, and media files related to the chat contents are downloaded one by adopting a media file downloading SDK.
In the process of implementing the present invention, the inventor finds that at least the following problems exist in the prior art:
when an operator sends media files such as the same pictures and videos to all clients, the same media files are required to be downloaded and stored again, so that a large amount of repeated requests of the same network exist, and a large amount of physical storage space exists, so that resources such as application service network bandwidth, CPU (Central processing Unit) calculation, equipment storage and the like are occupied. Because a large number of the same media files are downloaded and processed in a time, the integral processing timeliness of the system is affected. In addition, the same media files are repeatedly downloaded one by one, and the repeated submitting of the same files can be caused to carry out quality inspection, so that a large amount of repeated work of a content quality inspection system and quality inspection personnel is caused.
Disclosure of Invention
In view of this, embodiments of the present invention provide a method, an apparatus, an electronic device, and a computer readable medium for downloading a media file, so as to solve the technical problem of occupying resources such as application service network bandwidth, CPU computing, and device storage.
To achieve the above object, according to one aspect of the embodiments of the present invention, there is provided a media asset file downloading method, including:
pulling session records from a session server in a multithreading parallel mode, wherein each session record comprises a signature value of a media resource file to be downloaded;
downloading the media resource file to be downloaded in the session record in a multithreading parallel mode; and for the media resource files to be downloaded in each session record, downloading the media resource files to be downloaded from the session server by a thread which acquires the distributed lock of the media resource files to be downloaded, uploading the media resource files to a file system for storage, obtaining the storage address of the media resource files to be downloaded, storing the storage address in a cache, and submitting the storage address of the media resource files to be downloaded to a quality inspection platform.
Optionally, the method further comprises:
other threads which do not acquire the distributed lock of the media resource file to be downloaded acquire the storage address of the media resource file to be downloaded from the cache;
responding to the fact that the other threads fail to acquire the storage address of the media resource file to be downloaded from the cache, and continuing to wait by the other threads;
And responding to the fact that the other threads can acquire the storage address of the media resource file to be downloaded from the cache, and submitting the storage address of the media resource file to be downloaded to the quality inspection platform or ending the process by the other threads.
Optionally, downloading the media asset file to be downloaded from the session server by a thread that obtains the distributed lock of the media asset file to be downloaded, including:
generating a distributed lock of the media asset file to be downloaded according to the signature value of the media asset file to be downloaded, and storing the distributed lock in a cache, wherein each thread competes for acquiring the distributed lock, so that the distributed lock is acquired by only one thread;
and downloading the media resource file to be downloaded from the session server by the thread which acquires the distributed lock, and waiting by other threads of the distributed lock which do not acquire the media resource file to be downloaded.
Optionally, each session record further includes a file format name suffix of the media resource file to be downloaded;
generating a distributed lock of the media asset file to be downloaded according to the signature value of the media asset file to be downloaded, including:
and generating a distributed lock of the media asset file to be downloaded according to the signature value and the file format name suffix of the media asset file to be downloaded.
Optionally, each session record further includes a sender identifier of the media asset file to be downloaded;
generating a distributed lock of the media asset file to be downloaded according to the signature value and the file format name suffix of the media asset file to be downloaded, including:
and splicing the sender identifier, the signature value and the file format name suffix of the media asset file to be downloaded, thereby generating the distributed lock of the media asset file to be downloaded.
Optionally, storing the distributed lock in a cache includes:
storing the distributed lock of the media resource file to be downloaded into a cache by adopting a key-value;
the key is a distributed lock of the media file to be downloaded, and the value is a preset default value or a service parameter of the media file to be downloaded.
Optionally, uploading the media asset file to be downloaded to a file system for storage, obtaining a storage address of the media asset file to be downloaded, and storing the storage address in a cache, including:
uploading the media resource file to be downloaded to a file system for storage, so as to obtain a storage address of the media resource file to be downloaded;
storing the storage address of the media resource file to be downloaded into the cache by adopting a key-value;
Wherein, the key is the distributed lock of the media resource file to be downloaded, and the value is the storage address of the media resource file to be downloaded.
In addition, according to another aspect of the embodiment of the present invention, there is provided a media asset file downloading apparatus, including:
the system comprises a pulling module, a server and a server, wherein the pulling module is used for pulling session records from a session server in a multithreading parallel mode, and each session record comprises a signature value of a media resource file to be downloaded;
the downloading module is used for downloading the media resource files to be downloaded in the session record in a multithreading parallel mode; and for the media resource files to be downloaded in each session record, downloading the media resource files to be downloaded from the session server by a thread which acquires the distributed lock of the media resource files to be downloaded, uploading the media resource files to a file system for storage, obtaining the storage address of the media resource files to be downloaded, storing the storage address in a cache, and submitting the storage address of the media resource files to be downloaded to a quality inspection platform.
Optionally, the download module is further configured to:
other threads which do not acquire the distributed lock of the media resource file to be downloaded acquire the storage address of the media resource file to be downloaded from the cache;
Responding to the fact that the other threads fail to acquire the storage address of the media resource file to be downloaded from the cache, and continuing to wait by the other threads;
and responding to the fact that the other threads can acquire the storage address of the media resource file to be downloaded from the cache, and submitting the storage address of the media resource file to be downloaded to the quality inspection platform or ending the process by the other threads.
Optionally, the download module is further configured to:
generating a distributed lock of the media asset file to be downloaded according to the signature value of the media asset file to be downloaded, and storing the distributed lock in a cache, wherein each thread competes for acquiring the distributed lock, so that the distributed lock is acquired by only one thread;
and downloading the media resource file to be downloaded from the session server by the thread which acquires the distributed lock, and waiting by other threads of the distributed lock which do not acquire the media resource file to be downloaded.
Optionally, each session record further includes a file format name suffix of the media resource file to be downloaded;
the download module is further configured to:
and generating a distributed lock of the media asset file to be downloaded according to the signature value and the file format name suffix of the media asset file to be downloaded.
Optionally, each session record further includes a sender identifier of the media asset file to be downloaded;
the download module is further configured to:
and splicing the sender identifier, the signature value and the file format name suffix of the media asset file to be downloaded, thereby generating the distributed lock of the media asset file to be downloaded.
Optionally, the download module is further configured to:
storing the distributed lock of the media resource file to be downloaded into a cache by adopting a key-value;
the key is a distributed lock of the media file to be downloaded, and the value is a preset default value or a service parameter of the media file to be downloaded.
Optionally, the download module is further configured to:
uploading the media resource file to be downloaded to a file system for storage, so as to obtain a storage address of the media resource file to be downloaded;
storing the storage address of the media resource file to be downloaded into the cache by adopting a key-value;
wherein, the key is the distributed lock of the media resource file to be downloaded, and the value is the storage address of the media resource file to be downloaded.
According to another aspect of an embodiment of the present invention, there is also provided an electronic device including:
one or more processors;
storage means for storing one or more programs,
The one or more processors implement the method of any of the embodiments described above when the one or more programs are executed by the one or more processors.
According to another aspect of an embodiment of the present invention, there is also provided a computer readable medium having stored thereon a computer program which, when executed by a processor, implements the method according to any of the embodiments described above.
According to another aspect of embodiments of the present invention, there is also provided a computer program product comprising a computer program which, when executed by a processor, implements the method according to any of the embodiments described above.
One embodiment of the above invention has the following advantages or benefits: the method adopts the technical means that the thread of the distributed lock for obtaining the media resource file to be downloaded downloads the media resource file to be downloaded from the session server, the media resource file to be downloaded is uploaded to the file system for storage, the storage address of the media resource file to be downloaded is obtained, the storage address is stored in the cache, and the storage address of the media resource file to be downloaded is submitted to the quality inspection platform, so that the technical problems that resources such as application service network bandwidth, CPU calculation, equipment storage and the like are occupied in the prior art are solved. The embodiment of the invention downloads and stores the same media files only once through the distributed lock, avoids repeated processing of the same media files, reduces occupied network resources such as application service network bandwidth and the like, saves machine resources such as CPU calculation, equipment storage and the like, and improves the downloading speed of the media files. Moreover, as the storage addresses of the same media resource files are submitted for quality inspection only once, the quality inspection manpower resources are saved, and the quality inspection processing speed is improved.
Further effects of the above-described non-conventional alternatives are described below in connection with the embodiments.
Drawings
In order to more clearly illustrate the embodiments of the invention or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described, it being obvious that the drawings in the following description are only some embodiments of the invention, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art. Wherein:
FIG. 1 is a flow chart of a media file downloading method according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of a system architecture of a media file downloading method according to an embodiment of the present invention;
FIG. 3 is a flow chart of a method for downloading a media file according to one exemplary embodiment of the present invention;
FIG. 4 is a flow chart of a method for downloading a media file according to another exemplary embodiment of the present invention;
FIG. 5 is a schematic diagram of a media file downloading apparatus according to an embodiment of the present invention;
FIG. 6 is an exemplary system architecture diagram in which embodiments of the present invention may be applied;
fig. 7 is a schematic diagram of a computer system suitable for use in implementing an embodiment of the invention.
Detailed Description
Exemplary embodiments of the present invention will now be described with reference to the accompanying drawings, in which various details of the embodiments of the present invention are included to facilitate understanding, and are to be considered merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
It should be noted that, in the technical solution of the present disclosure, the related aspects of collecting, updating, analyzing, processing, using, transmitting, storing, etc. of the personal information of the user all conform to the rules of the related laws and regulations, and are used for legal purposes without violating the public order colloquial. Necessary measures are taken for the personal information of the user, illegal access to the personal information data of the user is prevented, and the personal information security, network security and national security of the user are maintained.
Fig. 1 is a flowchart of a media file downloading method according to an embodiment of the present invention. As an embodiment of the present invention, as shown in fig. 1, the media file downloading method is applied to an operation platform, and may include:
Step 101, pulling session records from a session server in a multithreading parallel mode, wherein each session record comprises a signature value of a media asset file to be downloaded.
After an operator of the operation platform sends the session contents such as pictures, videos and the like to a client at the C end through a session server, the operation platform requests the session server to pull the session record, and the session server can generate the session record between the operator and the client group. The operation platform adopts a multithread parallel mode to pull session records from a session server, wherein each session record comprises a field for judging whether the session record is media asset session content or not, the operation platform can analyze whether the session record is media asset session content or not through the field, and if the session record is media asset session content, the session record also comprises a signature value (such as MD5 value) of a file to be downloaded.
Optionally, in order to generate the distributed lock in the subsequent step, each session record further includes a file format name suffix of the to-be-downloaded media asset file, and in this embodiment, each session record includes a field that is whether the content of the media asset session is media asset type, an MD5 value of the to-be-downloaded media asset file, and a file format name suffix of the to-be-downloaded media asset file. Optionally, each session record further includes a sender identifier and a receiver identifier of the media asset file to be downloaded, so as to improve the robustness of the data. Optionally, each session record further includes a service parameter of the media file to be downloaded, such as a timestamp of the sending party sending the media file.
In the embodiment of the invention, the operation platform is a user operation platform integrating social management and rapid conversion capability, is driven by a digital engine, combines AI and automation capability, realizes user operation, and is matched with electric marketing to perform strong conversion, thereby realizing the operation closed loop of the full life cycle of the user. The media resource file refers to a file of media class and data class such as pictures, videos, files (documents, forms, PPT and the like), voice, compression packets and the like transmitted in the session pulling process.
Step 102, downloading the media resource file to be downloaded in the session record in a multithreading parallel mode; and for the media resource files to be downloaded in each session record, downloading the media resource files to be downloaded from the session server by a thread which acquires the distributed lock of the media resource files to be downloaded, uploading the media resource files to a file system for storage, obtaining the storage address of the media resource files to be downloaded, storing the storage address in a cache, and submitting the storage address of the media resource files to be downloaded to a quality inspection platform.
In the step, the operation platform downloads the media resource files to be downloaded in the session record in a multithreading parallel mode. Specifically, for the media asset file to be downloaded in each session record, a thread acquiring the distributed lock of the media asset file to be downloaded downloads the media asset file to be downloaded from the session server, then the thread uploads the media asset file to be downloaded to a file system for storage to obtain a storage address of the media asset file to be downloaded, then the thread stores the storage address in a cache (such as a Redis) and submits the storage address of the media asset file to be downloaded to a quality inspection platform. Quality inspection refers to inspection and inspection of quality, compliance with marketing specifications, safety and compliance of contents, text, voice and the like, and is one of measures for ensuring marketing safety.
Optionally, the method further comprises: other threads which do not acquire the distributed lock of the media resource file to be downloaded acquire the storage address of the media resource file to be downloaded from the cache; responding to the fact that the other threads fail to acquire the storage address of the media resource file to be downloaded from the cache, and continuing to wait by the other threads; and responding to the fact that the other threads can acquire the storage address of the media resource file to be downloaded from the cache, and submitting the storage address of the media resource file to be downloaded to the quality inspection platform or ending the process by the other threads. Because the distributed lock can only be acquired by one thread, when the thread acquiring the distributed lock downloads the media resource file, other threads which need to download the media resource file cannot acquire the distributed lock of the media resource file, then the other threads continuously try to acquire the storage address of the media resource file to be downloaded from the cache, if the other threads can acquire the storage address of the media resource file to be downloaded from the cache, the other threads submit the storage address of the media resource file to be downloaded to the quality inspection platform (the quality inspection platform can perform deduplication processing) or end processing (only submit once), and if the other threads cannot acquire the storage address of the media resource file to be downloaded from the cache, the other threads continue waiting until the storage address of the media resource file to be downloaded can be acquired from the cache.
Optionally, downloading the media asset file to be downloaded from the session server by a thread that obtains the distributed lock of the media asset file to be downloaded, including: generating a distributed lock of the media asset file to be downloaded according to the signature value of the media asset file to be downloaded, and storing the distributed lock in a cache, wherein each thread competes for acquiring the distributed lock, so that the distributed lock is acquired by only one thread; and downloading the media resource file to be downloaded from the session server by the thread which acquires the distributed lock, and waiting by other threads of the distributed lock which do not acquire the media resource file to be downloaded.
In the session pulling process, the embodiment of the invention ensures that only one downloading request is needed to execute one downloading operation under the condition of concurrency of media files with the same signature value by reasonably setting a distributed lock (key-1 of the lock) of Redis. After the downloading is completed and the storage is successful, the storage address (URL) of the downloaded media file is written into Redis (cache key-2), and the storage address of the media file is submitted to the quality inspection platform. And for media files with the same signature value, the storage address URL is preferentially acquired from the Redis, and if the media files are available, the re-downloading processing is not needed, and the repeated submitting of the quality inspection platform is not needed.
It should be noted that, in the embodiment of the present invention, a distributed lock is generated by the dis, each media file generates a distributed lock, the distributed lock can only be acquired by one thread, the thread downloads the media file, and other threads wait, so that each media file has only one download request, and a download operation is performed, thereby avoiding repeated downloads.
Optionally, generating the distributed lock of the media asset file to be downloaded according to the signature value of the media asset file to be downloaded, including: and generating a distributed lock of the media asset file to be downloaded according to the signature value and the file format name suffix of the media asset file to be downloaded. If the signature values of the media files of the same type are consistent, the media files with the same content can be ensured to be the same, so that the embodiment of the invention generates the distributed lock according to the signature values of the media files to be downloaded and the file format name suffix, and ensures that if the distributed lock of the media files with the same content is only acquired by one thread.
Optionally, generating the distributed lock of the to-be-downloaded media asset file according to the signature value and the file format name suffix of the to-be-downloaded media asset file, including: and splicing the sender identifier, the signature value and the file format name suffix of the media asset file to be downloaded, thereby generating the distributed lock of the media asset file to be downloaded.
Because session pulling is parallel pulling processing, in order to ensure that the downloaded content of parallel processing is not repeated, the embodiment of the invention is realized by using the distributed lock of redis, and the distributed lock comprises the following components: the sender identification of the media asset file to be downloaded, the file format name suffix of the media asset file to be downloaded, and the MD5 value of the media asset file to be downloaded. If the sender identification of the media file to be downloaded is empty, it means that only one thread can execute the downloading operation when the same media file under all senders is pulled in parallel; if the sender identification of the media file to be downloaded is not null, the method can realize that only one thread can execute the downloading operation when the same media file under the same sender is pulled in parallel. Thus, embodiments of the present invention ensure unique downloads across senders or under designated senders.
Optionally, storing the distributed lock in a cache includes: storing the distributed lock of the media resource file to be downloaded into a cache by adopting a key-value; the key is a distributed lock of the media file to be downloaded, and the value is a preset default value or a service parameter of the media file to be downloaded. In order to reduce the storage space of the Redis as much as possible, the distributed lock of the media asset file to be downloaded is stored in the cache by adopting a key-value, wherein the key is the distributed lock of the media asset file to be downloaded, and the value is a preset default value (such as value=1) or a service parameter of the media asset file to be downloaded (such as a timestamp of the media asset file sent by the sender).
Optionally, uploading the media asset file to be downloaded to a file system for storage, obtaining a storage address of the media asset file to be downloaded, and storing the storage address in a cache, including: uploading the media resource file to be downloaded to a file system for storage, so as to obtain a storage address of the media resource file to be downloaded; storing the storage address of the media resource file to be downloaded into the cache by adopting a key-value; wherein, the key is the distributed lock of the media resource file to be downloaded, and the value is the storage address of the media resource file to be downloaded. The thread which obtains the distributed lock uploads the downloaded media resource file to a file system (such as an HDFS distributed file storage system) so as to obtain the storage address of the media resource file, the storage address of the media resource file is stored in a cache by adopting a key-value, the key is the distributed lock of the media resource file, and the value is the storage address of the media resource file. And acquiring the storage address of the media file from the cache in other waiting threads, and submitting the storage address to a quality inspection platform (the quality inspection platform can perform deduplication processing) or not performing processing, so that repeated downloading and repeated submitting of quality inspection are avoided.
According to the above various embodiments, it can be seen that in the embodiments of the present invention, the thread that obtains the distributed lock of the to-be-downloaded media asset file downloads the to-be-downloaded media asset file from the session server, uploads the to-be-downloaded media asset file to the file system for storage, obtains the storage address of the to-be-downloaded media asset file, stores the storage address in the cache, and submits the storage address of the to-be-downloaded media asset file to the quality inspection platform, thereby solving the technical problems of occupying resources such as application service network bandwidth, CPU computing, and device storage in the prior art. The embodiment of the invention downloads and stores the same media files only once through the distributed lock, avoids repeated processing of the same media files, reduces occupied network resources such as application service network bandwidth and the like, saves machine resources such as CPU calculation, equipment storage and the like, and improves the downloading speed of the media files. Moreover, as the storage addresses of the same media resource files are submitted for quality inspection only once, the quality inspection manpower resources are saved, and the quality inspection processing speed is improved.
Fig. 3 is a flowchart of a media file downloading method according to a reference embodiment of the present invention. As still another embodiment of the present invention, as shown in fig. 3, the media file downloading method may include:
After an operator of the operation platform sends the same picture, video and other session contents to the C-terminal client group through the session server, the operation platform requests the session server to pull the session record, and the session server can generate the session record between the operator and the client group.
The operation platform adopts a multithread parallel mode to pull session records from a session server, each session record comprises a field of whether the session record is media asset session content, and if the session record is media asset session content, the session record also comprises a signature value (such as MD5 value) of a file to be downloaded, a file format name suffix, a sender identifier, a receiver identifier, a timestamp for sending the media asset file and the like.
And the media resource file downloading tool of the operation platform analyzes each session record in a multithreading mode, if the content of the session record is media resource session content, the Redis generates a distributed lock according to the signature value, the file format name suffix, the sender identifier and the like of the file to be downloaded, the distributed lock is stored in the Redis, and each thread downloading the same media resource file competes for acquiring the distributed lock, so that the distributed lock is acquired by only one thread. And downloading the media resource file from the session server by the thread which acquires the distributed lock, and waiting by other threads which do not acquire the distributed lock of the media resource file.
The thread which acquires the distributed lock uploads the successfully downloaded media file to a file system for storage, a storage address of the media file is obtained, the storage address is stored in a Redis, and the storage address of the media file is submitted to a quality inspection platform.
Other threads which do not acquire the distributed lock of the media file acquire the storage address of the media file from the Redis, and if the storage address of the media file can be acquired from the Redis, the storage address is submitted to a quality inspection platform or is not processed; if the storage address of the media file cannot be acquired from the Redis, continuing to wait until the storage address of the media file can be acquired from the Redis.
The embodiment of the invention solves the technical problems of repeated downloading, repeated uploading and repeated quality inspection in the process of downloading the media files by utilizing the distributed caching technology and the distributed locking technology of Redis, obviously improves the processing timeliness of the system and saves the storage cost, the hardware cost and the whole network cost of the system.
In addition, in the embodiment of the present invention, the implementation of the method for downloading the media file is described in detail in the above description, so that the description is not repeated here.
Fig. 4 is a flowchart of a media file downloading method according to another exemplary embodiment of the present invention. As another embodiment of the present invention, as shown in fig. 4, the media asset file downloading method may include:
the operation platform requests to pull the session record from the session server, which generates the session record between the operator and the customer base.
The operation platform adopts a multithread parallel mode to pull session records from a session server, each session record comprises a field of whether the session record is media asset session content, and if the session record is media asset session content, the session record also comprises an MD5 value of a file to be downloaded, a file format name suffix, a sender identifier (corpID) and a receiver identifier, a timestamp for sending the media asset file and the like.
The media resource file downloading tool of the operation platform analyzes each session record in a multithreading mode, and if the content of the session record is media resource session content, the thread generates a key according to the signature value, the file format name suffix, the sender identifier and the like of the file to be downloaded: C+morphID+file format name suffix+MD 5 value, traversing each key (value is URL of media file) in Redis, judging whether the key exists in Redis, if so, ending; if not, the Redis generates a distributed lock key according to the signature value, the file format name suffix, the sender identifier and the like of the file to be downloaded: S+corpID+File Format name suffix+MD 5 value, and store the distributed lock in Redis (value is a preset default value or service parameter of the media resource file), and each thread downloading the same media resource file competes for obtaining the distributed lock, so that the distributed lock is obtained by only one thread. And downloading the media resource file from the session server by the thread which acquires the distributed lock, and waiting by other threads which do not acquire the distributed lock of the media resource file.
The thread which acquires the distributed lock uploads the successfully downloaded media resource file to a file system for storage to obtain a storage address URL of the media resource file, then the URL of the media resource file is stored in a Redis, and the URL of the media resource file is submitted to a quality inspection platform.
Other threads which do not acquire the distributed lock of the media file acquire the URL of the media file from the cache, and if the URL of the media file can be acquired from the cache, the URL of the media file is submitted to a quality inspection platform or is not processed; if the URL of the media asset file is not obtained from the Redis, waiting is continued until the URL of the media asset file can be obtained from the Redis.
In addition, in another embodiment of the present invention, the implementation of the method for downloading the media file is described in detail in the above description, so that the description is not repeated here.
Fig. 5 is a schematic diagram of a media file downloading apparatus according to an embodiment of the present invention. As shown in fig. 5, the media asset file downloading device 500 includes a pull module 501 and a download module 502; the pulling module 501 is configured to pull session records from a session server in a multithreading parallel manner, where each session record includes a signature value of a media asset file to be downloaded; the downloading module 502 is configured to download the media asset file to be downloaded in the session record in a multithreading parallel manner; and for the media resource files to be downloaded in each session record, downloading the media resource files to be downloaded from the session server by a thread which acquires the distributed lock of the media resource files to be downloaded, uploading the media resource files to a file system for storage, obtaining the storage address of the media resource files to be downloaded, storing the storage address in a cache, and submitting the storage address of the media resource files to be downloaded to a quality inspection platform.
Optionally, the download module 502 is further configured to:
other threads which do not acquire the distributed lock of the media resource file to be downloaded acquire the storage address of the media resource file to be downloaded from the cache;
responding to the fact that the other threads fail to acquire the storage address of the media resource file to be downloaded from the cache, and continuing to wait by the other threads;
and responding to the fact that the other threads can acquire the storage address of the media resource file to be downloaded from the cache, and submitting the storage address of the media resource file to be downloaded to the quality inspection platform or ending the process by the other threads.
Optionally, the download module 502 is further configured to:
generating a distributed lock of the media asset file to be downloaded according to the signature value of the media asset file to be downloaded, and storing the distributed lock in a cache, wherein each thread competes for acquiring the distributed lock, so that the distributed lock is acquired by only one thread;
and downloading the media resource file to be downloaded from the session server by the thread which acquires the distributed lock, and waiting by other threads of the distributed lock which do not acquire the media resource file to be downloaded.
Optionally, each session record further includes a file format name suffix of the media resource file to be downloaded;
The download module 502 is further configured to:
and generating a distributed lock of the media asset file to be downloaded according to the signature value and the file format name suffix of the media asset file to be downloaded.
Optionally, each session record further includes a sender identifier of the media asset file to be downloaded;
the download module 502 is further configured to:
and splicing the sender identifier, the signature value and the file format name suffix of the media asset file to be downloaded, thereby generating the distributed lock of the media asset file to be downloaded.
Optionally, the download module 502 is further configured to:
storing the distributed lock of the media resource file to be downloaded into a cache by adopting a key-value;
the key is a distributed lock of the media file to be downloaded, and the value is a preset default value or a service parameter of the media file to be downloaded.
Optionally, the download module 502 is further configured to:
uploading the media resource file to be downloaded to a file system for storage, so as to obtain a storage address of the media resource file to be downloaded;
storing the storage address of the media resource file to be downloaded into the cache by adopting a key-value;
wherein, the key is the distributed lock of the media resource file to be downloaded, and the value is the storage address of the media resource file to be downloaded.
The implementation of the media file downloading device according to the present invention has been described in detail in the foregoing method for downloading media files, and thus the description thereof will not be repeated here.
Fig. 6 illustrates an exemplary system architecture 600 to which the media file downloading method or device of the present invention may be applied.
As shown in fig. 6, the system architecture 600 may include terminal devices 601, 602, 603, a network 604, and a server 605. The network 604 is used as a medium to provide communication links between the terminal devices 601, 602, 603 and the server 605. The network 604 may include various connection types, such as wired, wireless communication links, or fiber optic cables, among others.
A user may interact with the server 605 via the network 604 using the terminal devices 601, 602, 603 to receive or send messages, etc. Various communication client applications such as shopping class applications, web browser applications, search class applications, instant messaging tools, mailbox clients, social platform software, etc. (by way of example only) may be installed on the terminal devices 601, 602, 603.
The terminal devices 601, 602, 603 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smartphones, tablets, laptop and desktop computers, and the like.
The server 605 may be a server providing various services, such as a background management server (by way of example only) providing support for shopping-type websites browsed by users using terminal devices 601, 602, 603. The background management server can analyze and other data such as the received article information inquiry request and feed back the processing result to the terminal equipment.
It should be noted that, the method for downloading the media file according to the embodiment of the present invention is generally executed by the server 605, and accordingly, the device for downloading the media file is generally disposed in the server 605. The media file downloading method provided by the embodiment of the invention can also be executed by the terminal devices 601, 602 and 603, and correspondingly, the media file downloading device can be arranged in the terminal devices 601, 602 and 603.
It should be understood that the number of terminal devices, networks and servers in fig. 6 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.
Referring now to FIG. 7, there is illustrated a schematic diagram of a computer system 700 suitable for use in implementing an embodiment of the present invention. The terminal device shown in fig. 7 is only an example, and should not impose any limitation on the functions and the scope of use of the embodiment of the present invention.
As shown in fig. 7, the computer system 700 includes a Central Processing Unit (CPU) 701, which can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM) 702 or a program loaded from a storage section 708 into a Random Access Memory (RAM) 703. In the RAM703, various programs and data required for the operation of the system 700 are also stored. The CPU 701, ROM 702, and RAM703 are connected to each other through a bus 704. An input/output (I/O) interface 705 is also connected to bus 704.
The following components are connected to the I/O interface 705: an input section 706 including a keyboard, a mouse, and the like; an output portion 707 including a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, a speaker, and the like; a storage section 708 including a hard disk or the like; and a communication section 709 including a network interface card such as a LAN card, a modem, or the like. The communication section 709 performs communication processing via a network such as the internet. The drive 710 is also connected to the I/O interface 705 as needed. A removable medium 711 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 710 as necessary, so that a computer program read therefrom is mounted into the storage section 708 as necessary.
In particular, according to embodiments of the present disclosure, the processes described above with reference to flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method shown in the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network via the communication portion 709, and/or installed from the removable medium 711. The above-described functions defined in the system of the present invention are performed when the computer program is executed by a Central Processing Unit (CPU) 701.
The computer readable medium shown in the present invention may be a computer readable signal medium or a computer readable storage medium, or any combination of the two. The computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples of the computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present invention, however, the computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, with the computer-readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.
The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer programs according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The modules involved in the embodiments of the present invention may be implemented in software or in hardware. The described modules may also be provided in a processor, for example, as: a processor includes a pull module and a download module, where the names of the modules do not constitute a limitation on the module itself in some cases.
As another aspect, the present invention also provides a computer-readable medium that may be contained in the apparatus described in the above embodiments; or may be present alone without being fitted into the device. The computer readable medium carries one or more programs which, when executed by a device, implement the method of: pulling session records from a session server in a multithreading parallel mode, wherein each session record comprises a signature value of a media resource file to be downloaded; downloading the media resource file to be downloaded in the session record in a multithreading parallel mode; and for the media resource files to be downloaded in each session record, downloading the media resource files to be downloaded from the session server by a thread which acquires the distributed lock of the media resource files to be downloaded, uploading the media resource files to a file system for storage, obtaining the storage address of the media resource files to be downloaded, storing the storage address in a cache, and submitting the storage address of the media resource files to be downloaded to a quality inspection platform.
As a further aspect, embodiments of the present invention also provide a computer program product comprising a computer program which, when executed by a processor, implements the method according to any of the above embodiments.
According to the technical scheme of the embodiment of the invention, the technical means that the thread of the distributed lock for obtaining the media resource file to be downloaded downloads the media resource file from the session server, uploads the media resource file to be downloaded to the file system for storage, obtains the storage address of the media resource file to be downloaded, stores the storage address into the cache and submits the storage address of the media resource file to be downloaded to the quality inspection platform are adopted, so that the technical problems that resources such as application service network bandwidth, CPU calculation, equipment storage and the like are occupied in the prior art are overcome. The embodiment of the invention downloads and stores the same media files only once through the distributed lock, avoids repeated processing of the same media files, reduces occupied network resources such as application service network bandwidth and the like, saves machine resources such as CPU calculation, equipment storage and the like, and improves the downloading speed of the media files. Moreover, as the storage addresses of the same media resource files are submitted for quality inspection only once, the quality inspection manpower resources are saved, and the quality inspection processing speed is improved.
The above embodiments do not limit the scope of the present invention. It will be apparent to those skilled in the art that various modifications, combinations, sub-combinations and alternatives can occur depending upon design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of the present invention.

Claims (11)

1. A method for downloading a media asset file, comprising:
pulling session records from a session server in a multithreading parallel mode, wherein each session record comprises a signature value of a media resource file to be downloaded;
downloading the media resource file to be downloaded in the session record in a multithreading parallel mode; and for the media resource files to be downloaded in each session record, downloading the media resource files to be downloaded from the session server by a thread which acquires the distributed lock of the media resource files to be downloaded, uploading the media resource files to a file system for storage, obtaining the storage address of the media resource files to be downloaded, storing the storage address in a cache, and submitting the storage address of the media resource files to be downloaded to a quality inspection platform.
2. The method as recited in claim 1, further comprising:
other threads which do not acquire the distributed lock of the media resource file to be downloaded acquire the storage address of the media resource file to be downloaded from the cache;
responding to the fact that the other threads fail to acquire the storage address of the media resource file to be downloaded from the cache, and continuing to wait by the other threads;
And responding to the fact that the other threads can acquire the storage address of the media resource file to be downloaded from the cache, and submitting the storage address of the media resource file to be downloaded to the quality inspection platform or ending the process by the other threads.
3. The method of claim 1, wherein downloading the media asset file to be downloaded from the session server by a thread that obtains the distributed lock for the media asset file to be downloaded comprises:
generating a distributed lock of the media asset file to be downloaded according to the signature value of the media asset file to be downloaded, and storing the distributed lock in a cache, wherein each thread competes for acquiring the distributed lock, so that the distributed lock is acquired by only one thread;
and downloading the media resource file to be downloaded from the session server by the thread which acquires the distributed lock, and waiting by other threads of the distributed lock which do not acquire the media resource file to be downloaded.
4. A method according to claim 3, wherein each session record further comprises a file format name suffix for the media asset file to be downloaded;
generating a distributed lock of the media asset file to be downloaded according to the signature value of the media asset file to be downloaded, including:
And generating a distributed lock of the media asset file to be downloaded according to the signature value and the file format name suffix of the media asset file to be downloaded.
5. The method of claim 4, wherein each session record further includes a sender identification of the media asset file to be downloaded;
generating a distributed lock of the media asset file to be downloaded according to the signature value and the file format name suffix of the media asset file to be downloaded, including:
and splicing the sender identifier, the signature value and the file format name suffix of the media asset file to be downloaded, thereby generating the distributed lock of the media asset file to be downloaded.
6. The method of claim 3, wherein storing the distributed lock in a cache comprises:
storing the distributed lock of the media resource file to be downloaded into a cache by adopting a key-value;
the key is a distributed lock of the media file to be downloaded, and the value is a preset default value or a service parameter of the media file to be downloaded.
7. The method of claim 1, wherein uploading the media asset file to be downloaded to a file system for storage, obtaining a storage address of the media asset file to be downloaded, and storing the storage address in a cache, comprises:
Uploading the media resource file to be downloaded to a file system for storage, so as to obtain a storage address of the media resource file to be downloaded;
storing the storage address of the media resource file to be downloaded into the cache by adopting a key-value;
wherein, the key is the distributed lock of the media resource file to be downloaded, and the value is the storage address of the media resource file to be downloaded.
8. A media asset file downloading device, comprising:
the system comprises a pulling module, a server and a server, wherein the pulling module is used for pulling session records from a session server in a multithreading parallel mode, and each session record comprises a signature value of a media resource file to be downloaded;
the downloading module is used for downloading the media resource files to be downloaded in the session record in a multithreading parallel mode; and for the media resource files to be downloaded in each session record, downloading the media resource files to be downloaded from the session server by a thread which acquires the distributed lock of the media resource files to be downloaded, uploading the media resource files to a file system for storage, obtaining the storage address of the media resource files to be downloaded, storing the storage address in a cache, and submitting the storage address of the media resource files to be downloaded to a quality inspection platform.
9. An electronic device, comprising:
one or more processors;
storage means for storing one or more programs,
the one or more processors implement the method of any of claims 1-7 when the one or more programs are executed by the one or more processors.
10. A computer readable medium, on which a computer program is stored, characterized in that the program, when being executed by a processor, implements the method according to any of claims 1-7.
11. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any of claims 1-7.
CN202311023277.1A 2023-08-14 2023-08-14 Media asset file downloading method, device, electronic equipment and computer readable medium Pending CN117082050A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202311023277.1A CN117082050A (en) 2023-08-14 2023-08-14 Media asset file downloading method, device, electronic equipment and computer readable medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202311023277.1A CN117082050A (en) 2023-08-14 2023-08-14 Media asset file downloading method, device, electronic equipment and computer readable medium

Publications (1)

Publication Number Publication Date
CN117082050A true CN117082050A (en) 2023-11-17

Family

ID=88709040

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202311023277.1A Pending CN117082050A (en) 2023-08-14 2023-08-14 Media asset file downloading method, device, electronic equipment and computer readable medium

Country Status (1)

Country Link
CN (1) CN117082050A (en)

Similar Documents

Publication Publication Date Title
CN110601952A (en) Multi-channel message notification sending method and device
WO2020199751A1 (en) Method and apparatus for loading page picture, and electronic device
CN109510754B (en) Online document generation method, device and system and electronic equipment
CN111478781B (en) Message broadcasting method and device
CN113517985B (en) File data processing method and device, electronic equipment and computer readable medium
CN110572422A (en) Data downloading method and device
CN109218041B (en) Request processing method and device for server system
CN111427899A (en) Method, device, equipment and computer readable medium for storing file
US20180343216A1 (en) Context driven modification of attachments in a messaging session
CN111831503B (en) Monitoring method based on monitoring agent and monitoring agent device
CN113452733A (en) File downloading method and device
CN112149392A (en) Rich text editing method and device
CN112948138A (en) Method and device for processing message
CN117082050A (en) Media asset file downloading method, device, electronic equipment and computer readable medium
CN114861106A (en) Information processing method, device, electronic equipment and storage medium
CN112688982B (en) User request processing method and device
CN111786801B (en) Method and device for charging based on data flow
CN114417318A (en) Third-party page jumping method and device and electronic equipment
CN112532734A (en) Message sensitive information detection method and device
CN111291290A (en) Data processing method and device
CN113722115A (en) Method, device, equipment and computer readable medium for calling interface
CN111163156A (en) Data processing method, device and storage medium based on block chain
CN113326060A (en) Service request processing method, device and system and service configuration method and device
CN114979295B (en) Gateway management method and device
CN110909269B (en) Log reporting method and device

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination