CN116980641B - Asynchronous processing method, system, computer and storage medium for video migration - Google Patents
Asynchronous processing method, system, computer and storage medium for video migration Download PDFInfo
- Publication number
- CN116980641B CN116980641B CN202311228101.XA CN202311228101A CN116980641B CN 116980641 B CN116980641 B CN 116980641B CN 202311228101 A CN202311228101 A CN 202311228101A CN 116980641 B CN116980641 B CN 116980641B
- Authority
- CN
- China
- Prior art keywords
- migration
- preset
- objects
- disk
- metadata
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 230000005012 migration Effects 0.000 title claims abstract description 157
- 238000013508 migration Methods 0.000 title claims abstract description 141
- 238000003672 processing method Methods 0.000 title claims abstract description 27
- 238000012545 processing Methods 0.000 claims abstract description 64
- 238000000034 method Methods 0.000 claims abstract description 23
- 238000012546 transfer Methods 0.000 claims abstract description 4
- 230000036541 health Effects 0.000 claims description 41
- 238000004590 computer program Methods 0.000 claims description 13
- 238000005516 engineering process Methods 0.000 abstract description 6
- 230000008569 process Effects 0.000 description 13
- 230000005540 biological transmission Effects 0.000 description 11
- 238000004891 communication Methods 0.000 description 8
- 238000012937 correction Methods 0.000 description 5
- 230000001360 synchronised effect Effects 0.000 description 4
- 238000010586 diagram Methods 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 230000009471 action Effects 0.000 description 2
- 238000003491 array Methods 0.000 description 2
- 238000007730 finishing process Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000005457 optimization Methods 0.000 description 2
- 230000002093 peripheral effect Effects 0.000 description 2
- 239000007787 solid Substances 0.000 description 2
- FGUUSXIOTUKUDN-IBGZPJMESA-N C1(=CC=CC=C1)N1C2=C(NC([C@H](C1)NC=1OC(=NN=1)C1=CC=CC=C1)=O)C=CC=C2 Chemical compound C1(=CC=CC=C1)N1C2=C(NC([C@H](C1)NC=1OC(=NN=1)C1=CC=CC=C1)=O)C=CC=C2 FGUUSXIOTUKUDN-IBGZPJMESA-N 0.000 description 1
- 230000001133 acceleration Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008878 coupling Effects 0.000 description 1
- 238000010168 coupling process Methods 0.000 description 1
- 238000005859 coupling reaction Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000000802 evaporation-induced self-assembly Methods 0.000 description 1
- 230000003203 everyday effect Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The application provides an asynchronous processing method, a system, a computer and a storage medium for video migration, wherein the method comprises the steps of obtaining video object metadata of metadata management service and performing erasure coding processing on the video object metadata; the thread pool obtains the current capacity of a buffer area of each disk in the disk group, and counts the number of disks of which the current capacity exceeds the preset capacity in the buffer area in the disk group; judging whether the number of the magnetic disks exceeding the preset capacity is smaller than a threshold value, if so, delaying a first preset time to transfer the metadata of the video object subjected to erasure coding to the cache region; and migrating the video object metadata in the buffer memory area to the disk according to a migration flow, and enabling the whole erasure coding operation to be more efficient through a thread pool technology.
Description
Technical Field
The application belongs to the technical field of distributed data processing, and particularly relates to an asynchronous processing method, an asynchronous processing system, a computer and a storage medium for video migration.
Background
With the development of technologies such as AI, big data, cloud computing and the like, the use of intelligent video cloud is mature gradually, the intelligent video cloud needs to process massive video data every day, the video data is generated by a streaming media forwarding server, and a video slice object is generated every 30 seconds.
And the streaming media forwarding server reports the metadata of the video slice object to the metadata management server, the migration program reads the video object from the streaming media forwarding server after taking the metadata of the video object from the metadata management server, performs erasure coding and slicing on the video object to generate m+n data, and finally sends the m+n data to m+n discs of the disc storage server to finish video object data disc dropping. Erasure Coding (EC): definition of EC: eraseR Code is a coding technology, which can add n parts of original data, and can restore the original data by any m parts of m+n parts. That is, if any data of n parts or less fail, the data can still be restored by the rest data. Health degree: the health degree is closely related to erasure codes, for example, erasure codes are 8+4, then 12 is called full health degree, less than 8 is unable to restore data, and 8 to 12 is called insufficient health degree.
In the process of migrating the existing video object data to a disk, the following challenges are faced:
1. how is the process of ensuring erasure coding very efficient? Erasure coding is a very cpu-consuming operation, how does the migration of many video objects maximize the performance of the hardware?
2. In the process of transmitting m+n data to m+n disks after erasure coding is completed on a video object, because each disk is processed asynchronously, how is the health of the video object data balanced with the migration time? For example, a video object of 8MB is subjected to erasure coding (8+4) to generate 12 pieces of data, each piece of data is 1MB, assuming that only 10 discs return a dropped result, and another 2 hard discs return slowly due to slow io processing, then how is the processing slow io processing? If the results are reported from 12 discs, the whole migration process is slow, the whole migration speed is affected, if the results are reported from 10 discs, the health of the video object data is insufficient, the full health is 8+4, and the health of the 10 discs is 8+2.
3. How to ensure that the network IO is smoother in the data migration process, the situation of large rise and fall can not occur, and otherwise, the service can be influenced.
Disclosure of Invention
In order to solve the technical problems, the application provides an asynchronous processing method and an asynchronous processing system for video migration, which are used for solving the technical problem that the processing process of erasure coding is low in the process of migrating the existing video object data to a disk.
In one aspect, the present application provides the following technical solutions, and an asynchronous processing method for video migration, where the asynchronous processing method is implemented based on a distributed processing model, and the distributed processing model includes: metadata management services and thread pools; the asynchronous processing method comprises the following steps:
acquiring video object metadata of the metadata management service, and performing erasure coding processing on the video object metadata;
the thread pool obtains the current capacity of a buffer area of each disk in the disk group, and counts the number of disks of which the current capacity exceeds the preset capacity in the buffer area in the disk group;
judging whether the number of the magnetic disks exceeding the preset capacity is smaller than a threshold value, if so, delaying a first preset time to transfer the metadata of the video object subjected to erasure coding to the cache region;
and migrating the video object metadata in the buffer memory area to the disk according to a migration flow, sorting the migration result of the disk according to a sorting flow, and uploading the migration result to the metadata management service.
Compared with the prior art, the application has the beneficial effects that: the multiple threads in the thread pool can independently run, so that a full asynchronous processing architecture is formed, and the overall processing is more efficient, because the asynchronous processing means that each step of processing is loosely coupled, the performance optimization is easier to perform for the step of processing, and the overall erasure coding operation can be more efficient through the thread pool;
in addition, the architecture of the fully asynchronous processing means that video data is accessed from the end of the video data to the end of the video data, and the latest video data can be seen more quickly than the existing synchronous processing.
Further, the step of acquiring the video object metadata of the metadata management service includes:
acquiring the preset number of video object metadata of the metadata management service, wherein the preset number is the video object metadata which can be processed by the erasure coding single time;
creating a video object metadata cache queue, and migrating the video object metadata of the preset object number to the video object metadata cache queue.
Further, the migration process includes:
acquiring the number of objects of the video object metadata in the cache region at intervals of a second preset time, and judging whether the number of objects is equal to the number of first preset objects or not;
if the number of the objects is not equal to the first preset number of objects, judging whether the number of the objects exceeds the second preset number of objects or not;
and if the number of the objects exceeds a second preset number of objects, sending the number of the objects to the disk.
Further, after the step of determining whether the number of objects exceeds the second preset number of objects, the asynchronous processing method further includes:
if the number of the objects does not exceed the second preset number of objects, repeating the step of obtaining the number of the objects of the video object metadata in the cache region at intervals of second preset time;
and judging the time spent by repeatedly executing the scanning of the object number of the video object metadata in the buffer area at intervals of the second preset time, and if the spent time exceeds the preset third preset time, sending the object number to the magnetic disk.
Further, the finishing process includes:
creating a migration result queue, and sending the migration result of the disk group to the migration result queue;
acquiring information of the migration result in the migration result queue at intervals of fourth preset time, and counting the health degree of the disk group;
if the health degree of the disk group is insufficient, calculating time required for counting the health degree of the disk group, and if the required time is larger than a preset value;
and transmitting the information of the migration result in the migration result queue to a result report queue.
Further, the step of uploading the migration result to the metadata management service includes:
and acquiring the information of the migration result in the result report queue, and sending the information of the migration result to a metadata management service in batches until the information of the migration result is sent.
Further, before the step of acquiring the video object metadata of the metadata management service, the method further includes:
and applying for obtaining the disk group, and initializing a buffer zone with a preset size for each disk in the disk group.
In a second aspect, the present application provides a video migration asynchronous processing system, where the asynchronous processing system is implemented based on a distributed processing model, and the distributed processing model includes a metadata management service and a thread pool; the asynchronous processing system includes:
the acquisition module is used for acquiring the video object metadata of the metadata management service and performing erasure coding processing on the video object metadata;
the counting module is used for acquiring the current capacity of the buffer area of each disk in the disk group by the thread pool and counting the number of disks of which the current capacity exceeds the preset capacity in the disk group;
the judging module is used for judging whether the number of the magnetic disks exceeding the preset capacity is smaller than a threshold value or not;
the migration module is used for delaying a first preset time to migrate the metadata of the video object subjected to erasure coding to the cache region if the number of the magnetic disks exceeding the preset capacity is smaller than a threshold value;
and the arrangement module is used for migrating the video object metadata in the cache region to the disk according to the migration flow, arranging the migration result of the disk according to the arrangement flow, and uploading the migration result to the metadata management service.
In a third aspect, the present application provides a computer, including a memory, a processor, and a computer program stored in the memory and executable on the processor, where the processor implements an asynchronous processing method for video migration as described above when the processor executes the computer program.
In a fourth aspect, the present application provides a storage medium having a computer program stored thereon, the computer program implementing an asynchronous processing method for video migration as described above when executed by a processor.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the embodiments or the description of the prior art will be briefly described below, it being obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flowchart of an asynchronous processing method for video migration according to a first embodiment of the present application;
FIG. 2 is a schematic diagram of a data flow of an asynchronous processing method for video migration according to a first embodiment of the present application;
FIG. 3 is a block diagram of an asynchronous processing system for video migration according to a second embodiment of the present application;
fig. 4 is a schematic structural diagram of a computer according to a third embodiment of the present application.
Embodiments of the present application will be further described below with reference to the accompanying drawings.
Detailed Description
Embodiments of the present application are described in detail below, examples of which are illustrated in the accompanying drawings, wherein like or similar reference numerals refer to like or similar elements or elements having like or similar functions throughout. The embodiments described below by referring to the drawings are exemplary and intended to illustrate embodiments of the application and should not be construed as limiting the application.
In the description of the embodiments of the present application, it should be understood that the terms "length," "width," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," and the like indicate orientations or positional relationships based on the orientation or positional relationships shown in the drawings, merely to facilitate description of the embodiments of the present application and simplify description, and do not indicate or imply that the devices or elements referred to must have a specific orientation, be configured and operated in a specific orientation, and thus should not be construed as limiting the present application.
Furthermore, the terms "first," "second," and the like, are used for descriptive purposes only and are not to be construed as indicating or implying a relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defining "a first" or "a second" may explicitly or implicitly include one or more such feature. In the description of the embodiments of the present application, the meaning of "plurality" is two or more, unless explicitly defined otherwise.
Example 1
In a first embodiment of the present application, as shown in fig. 1 and fig. 2, an asynchronous processing method for video migration, where the whole process adopts asynchronous processing, can effectively improve the overall processing performance, the asynchronous processing method is implemented based on a distributed processing model, and the distributed processing model includes: the system comprises a disk service program, a migration program, disk scheduling, metadata management service and a thread pool, wherein the thread pool comprises a data acquisition thread, an erasure coding thread, a data transmission thread, a sorting thread and a reporting thread; the other disk service program corresponds to one disk, the disk service program is mainly responsible for writing video data into the disk, and when a service looks at video, the disk service program reads data from the disk;
the migration program obtains disk group information from disk scheduling, then the migration program establishes connection with each disk in the disk group, the migration program obtains the metadata of the video object from the metadata management service, sends the data to the disk to complete migration action after erasure coding, and finally reports the migration result to the metadata management service.
The asynchronous processing method comprises the following steps S01 to S05:
s01, applying for obtaining a disk group, and initializing a buffer zone with a preset size for each disk in the disk group.
The specific implementation steps are as follows;
step one, a migration program has a migration ID to uniquely identify the current migration program, the migration program applies for a disk group to disk scheduling periodically, input data are a migration ID, an erasure correction type (m+n) and a storage period, the disk scheduling distributes a disk group to the migration program according to the migration ID, the erasure correction type and the storage period, the disk group comprises m+n disks, and each disk information comprises key information such as a disk ID, a disk IP, a disk port and the like;
step two, the migration program initializes a transmission buffer zone with the size of 40MB for each disk in the disk group, and the purpose of the initialization buffer zone is to transmit object data in batches, so that the network transmission efficiency is improved;
step three, the migration program establishes connection with each disk in the disk group respectively;
step four, the migration program creates a migration result buffer queue, and the video object is sent to the disk and then the result is put in the queue;
and fifthly, the migration program creates a migration result reporting queue, and the migration result reporting thread fetches data from the queue and sends the data to the metadata management service.
S02, obtaining video object metadata of metadata management service, and performing erasure coding processing on the video object metadata;
wherein the step of acquiring the video object metadata of the metadata management service includes:
acquiring the preset number of video object metadata of the metadata management service, wherein the preset number is the video object metadata which can be processed by the erasure coding single time;
creating a video object metadata cache queue, and migrating the video object metadata of the preset object number to the video object metadata cache queue.
Specifically, the migration program creates a video object metadata cache queue; the data acquisition thread requests the metadata of the video object from the metadata management service, and each request carries the maximum number of objects maxNum processed once; the metadata management service returns the number of objects to be maxNum as much as possible in each response; and the migration program receives the video object metadata and then puts the video object metadata into a video object metadata cache queue.
It is worth to say that the migration program creates a thread pool, the thread pool can exert the hardware performance of the server greatly, each thread operates independently, and the main work is to put the data subjected to erasure coding of video object data into a disk sending buffer area. Because the whole is fully asynchronous, each step of processing is loosely coupled, and the whole erasure coding operation can be more efficient through the thread pool technology.
S03, the thread pool acquires the current capacity of a buffer area of each disk in the disk group, and counts the number of disks, of which the current capacity exceeds the preset capacity, of the buffer area in the disk group;
specifically, the erasure coding thread acquires video object metadata from a video object metadata cache queue; the erasure coding thread counts the size of a sending buffer zone of each disk in the disk group, and counts the number of disks with the current capacity of the sending buffer zone of the disk exceeding half of the total capacity as count. For example, the initialization size of the sending buffer area of the disk is 40MB, and the current sending buffer area is 30MB, so that the disk needs to be counted, and the occurrence of the situation indicates that the disk sending is slower, which may be a network problem or a disk IO problem. The preset capacity is one half of the total capacity of the buffer area.
S04, judging whether the number of the magnetic disks exceeding the preset capacity is smaller than a threshold value, if so, delaying a first preset time to transfer the metadata of the video object subjected to erasure coding to the cache region;
wherein the minimum health of the migration object is k, and the full health is m+n, wherein k is greater than m and less than m+n;
in specific implementation, judging whether the number of the magnetic disks exceeding the preset capacity is smaller than a threshold value (k), if the count is smaller than the threshold value k, waiting for 3 seconds, and aiming at slowing down the migration speed, wherein the occurrence of the situation indicates that the process of migrating data to the magnetic disks is slow, the network problem is possibly caused, and some of the disk IO in the magnetic disk group is possibly caused;
if the count is greater than k, the data after erasure coding is put into the sending buffer area of each disk in the disk group. For example, a video object 8MB is provided, and the erasure coding mode is 8+4, so that erasure coding generates 12MB of data, and then 12 pieces of 1MB of data of the object are put into a transmission buffer of 12 discs.
It should be noted that, the current network transmission condition is determined by detecting the size of the transmission buffer, and the network transmission speed is controlled by the waiting time, so that the transmission is smoother.
S05, migrating the video object metadata in the cache region to the disk according to a migration flow, sorting migration results of the disk according to a sorting flow, and uploading the migration results to the metadata management service;
the migration process includes:
acquiring the number of objects of the video object metadata in the cache region at intervals of a second preset time, and judging whether the number of objects is equal to the number of first preset objects or not;
if the number of the objects is not equal to the first preset number of objects, judging whether the number of the objects exceeds the second preset number of objects or not;
and if the number of the objects exceeds a second preset number of objects, sending the number of the objects to the disk.
After the step of determining whether the number of objects exceeds the second preset number of objects, the asynchronous processing method further includes:
if the number of the objects does not exceed the second preset number of objects, repeating the step of obtaining the number of the objects of the video object metadata in the cache region at intervals of second preset time;
and judging the time spent by repeatedly executing the scanning of the object number of the video object metadata in the buffer area at intervals of the second preset time, and if the spent time exceeds the preset third preset time, sending the object number to the magnetic disk.
When the migration flow is implemented in detail:
step 1, a disk group comprises m+n disks, and each disk corresponds to a sending thread;
step 2, the sending thread scans the number of objects in the sending buffer area once every 10 milliseconds, and sends the objects to the disk immediately if the number is more than 50 objects;
and 3, if the number of the scanned objects is less than 50, continuing to wait for 10 milliseconds to scan, and triggering transmission when waiting for 10 times at most, namely 100 milliseconds. The purpose of waiting is to provide network transmission efficiency for bulk data;
step 4, if the number of the scanned objects is 0, waiting for 1 second, and if the objects exist, entering step 2;
and 5, after the data is successfully sent, the disk returns a migration success result, wherein the migration result consists of an object ID, an uploading ID, uploading time, a disk ID and an RsIndex, wherein the RsIndex indicates which disk can generate 12 results after 8+4 erasure coding of an object, the RsIndex corresponds to 1-12, and finally the migration result is sent to a migration result queue.
The arrangement flow comprises the following steps:
creating a migration result queue, and sending the migration result of the disk group to the migration result queue;
acquiring information of the migration result in the migration result queue at intervals of fourth preset time, and counting the health degree of the disk group;
if the health degree of the disk group is insufficient, calculating time required for counting the health degree of the disk group, and if the required time is larger than a preset value;
and transmitting the information of the migration result in the migration result queue to a result report queue.
When the finishing process is concretely implemented:
step a, generating m+n data after erasure coding of an object, and collecting m+n results for arrangement after m+n data are sent to m+n magnetic disks;
step b, the sorting thread performs sorting action every 1 second;
step c, the finishing thread scans all result information from the migration result queue;
step d, sorting threads classify according to object IDs, and counting the health degree of each object ID, for example, an object is subjected to 8+4 erasure coding, 12 pieces of result information are received after data transmission is successful, if the migration result is 12 pieces of result information, the result information is full health degree, and if the migration result is less than 8 pieces of result information, the data cannot be restored, and waiting is needed to be continued;
and e, if the object ID migration result is full of health, putting the result information into a result report queue, and if the object ID migration result is insufficient of health, not performing processing and entering the step b. If the health degree is insufficient, step f is entered, wherein the health degree is insufficient, which means that the original data after erasure correction coding cannot be restored, and the health degree is insufficient, which means that the original data after erasure correction coding can be restored, but the health degree is not full, it can be understood that the erasure correction coding can add n pieces of data to m pieces of original data, and the original data can be restored through any m pieces of data in m+n pieces, namely, if any data less than or equal to n pieces of data fail, the original data can still be restored through the rest data. For example, 8 parts of original data and 4 parts of data are added, the data after erasure coding is 12 parts, if the migration result is 12 parts, the data is full of health, if the migration result is 8 to 11 parts, the data is insufficient in health, and if the migration result is less than 8 parts, the data is insufficient in health.
And f, judging whether the statistical time t is overtime when the health degree is insufficient, and assuming that the overtime time is t1, and t=the current time-the uploading success time in the migration result. If t is greater than t1, the result information is put into a result report queue, and if t is less than t1, the process does not enter step b.
Wherein the step of uploading the migration result to the metadata management service comprises:
and acquiring the information of the migration result in the result report queue, and sending the information of the migration result to a metadata management service in batches until the information of the migration result is sent.
When the step of uploading the migration result to the metadata management service is specifically implemented:
and (1) the reporting thread acquires a plurality of pieces of migration result information from the result reporting queue and then sends the migration result information to the metadata management service in batches, so that the sending efficiency can be improved. The migration result message consists of an object ID, a migration result, a disk group ID, a disk ID list, an uploading ID list, migration time and erasure types;
step (2), reporting the thread judging result and reporting the length of the queue, if the length is 0, waiting for 1 second to enter the step (2), otherwise, entering the step (3);
and (3) the reporting thread transmits 500 migration results to the metadata management service in batches each time until the migration results are transmitted to completion.
It should be noted that, by setting the timeout to balance between full health and processing time, full health is achieved in most cases during the timeout, and if extreme conditions, such as when the network jitter is severe, may occur, some objects are not full health, then there is no need to delay too much time, and a certain condition is met for reporting. If the full health is reached, the overall processing speed may be severely blocked.
In summary, multiple threads in the thread pool can independently run to form a fully asynchronous processing architecture, so that the overall processing efficiency is higher, because asynchronous processing means that each step of processing is loosely coupled, performance optimization is easier to perform for the step of processing, and the overall erasure coding operation can be more efficient through the thread pool; in addition, the architecture of the fully asynchronous processing means that video data is accessed from the end of the video data to the end of the video data, and the latest video data can be seen more quickly than the existing synchronous processing.
Example two
As shown in FIG. 3, in a second embodiment of the present application, an asynchronous processing system for video migration is provided, the asynchronous processing system being implemented based on a distributed processing model, the distributed processing model including metadata management services and thread pools; the asynchronous processing system includes:
an obtaining module 10, configured to obtain metadata of a video object of the metadata management service, and perform erasure coding processing on the metadata of the video object;
the statistics module 20 is configured to obtain a current capacity of a buffer area of each disk in the disk group by using the thread pool, and count the number of disks in which the current capacity of the buffer area exceeds a preset capacity in the disk group;
a judging module 30, configured to judge whether the number of discs exceeding a preset capacity is smaller than a threshold;
a migration module 40, configured to delay a first preset time to migrate the metadata of the video object after erasure coding processing to the buffer area if the number of disks exceeding a preset capacity is smaller than a threshold;
and the arrangement module 50 is configured to migrate the metadata of the video object in the buffer area to the disk according to a migration flow, arrange the migration result of the disk according to the arrangement flow, and upload the migration result to the metadata management service.
In some alternative embodiments, the acquisition module 10 includes:
an obtaining unit, configured to obtain a preset number of video object metadata of the metadata management service, where the preset number is the video object metadata that can be processed by the erasure coding single time; creating a video object metadata cache queue, and migrating the video object metadata of the preset object number to the video object metadata cache queue.
In some alternative embodiments, the migration module 40 includes:
the judging unit is used for acquiring the number of objects of the video object metadata in the buffer area at intervals of a second preset time and judging whether the number of objects is equal to the number of first preset objects or not; if the number of the objects is not equal to the first preset number of objects, judging whether the number of the objects exceeds the second preset number of objects or not; and if the number of the objects exceeds a second preset number of objects, sending the number of the objects to the disk.
In some alternative embodiments, the migration module 40 further includes:
the sending unit is used for repeatedly executing the step of obtaining the object number of the video object metadata in the buffer area at the interval of second preset time if the object number does not exceed the second preset object number; and judging the time spent by repeatedly executing the scanning of the object number of the video object metadata in the buffer area at intervals of the second preset time, and if the spent time exceeds the preset third preset time, sending the object number to the magnetic disk.
In some alternative embodiments, the collation module 50 comprises:
the creation unit is used for creating a migration result queue and sending the migration result of the disk group to the migration result queue; acquiring information of the migration result in the migration result queue at intervals of fourth preset time, and counting the health degree of the disk group; if the health degree of the disk group is insufficient, calculating time required for counting the health degree of the disk group, and if the required time is larger than a preset value; and transmitting the information of the migration result in the migration result queue to a result report queue.
In some alternative embodiments, the collation module 50 further comprises:
and the reporting unit is used for acquiring the information of the migration result in the result reporting queue and sending the information of the migration result to the metadata management service in batches until the information of the migration result is sent completely.
In some alternative embodiments, the system further comprises:
the application module is used for applying for obtaining the disk group and initializing a buffer zone with a preset size for each disk in the disk group.
The implementation principle and the generated technical effects of the asynchronous processing system for video migration provided by the embodiment of the application are the same as those of the embodiment of the method, and for the sake of brevity, reference is made to the corresponding contents of the embodiment of the method for the part of the system embodiment not mentioned.
Example III
As shown in fig. 4, in a third embodiment of the present application, a computer according to the present application includes a memory 202, a processor 201, and a computer program stored in the memory 202 and executable on the processor 201, where the processor 201 implements the asynchronous processing method for video migration as described above when executing the computer program.
In particular, the processor 201 may include a Central Processing Unit (CPU), or an application specific integrated circuit (Application Specific Integrated Circuit, abbreviated as ASIC), or may be configured as one or more integrated circuits that implement embodiments of the present application.
Memory 202 may include, among other things, mass storage for data or instructions. By way of example, and not limitation, memory 202 may comprise a Hard Disk Drive (HDD), floppy Disk Drive, solid state Drive (Solid State Drive, SSD), flash memory, optical Disk, magneto-optical Disk, tape, or universal serial bus (Universal Serial Bus, USB) Drive, or a combination of two or more of the foregoing. Memory 202 may include removable or non-removable (or fixed) media, where appropriate. The memory 202 may be internal or external to the data processing apparatus, where appropriate. In a particular embodiment, the memory 202 is a Non-Volatile (Non-Volatile) memory. In a particular embodiment, the Memory 202 includes Read-Only Memory (ROM) and random access Memory (Random Access Memory, RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (Programmable Read-Only Memory, abbreviated PROM), an erasable PROM (Erasable Programmable Read-Only Memory, abbreviated EPROM), an electrically erasable PROM (Electrically Erasable Programmable Read-Only Memory, abbreviated EEPROM), an electrically rewritable ROM (Electrically Alterable Read-Only Memory, abbreviated EAROM), or a FLASH Memory (FLASH), or a combination of two or more of these. The RAM may be Static Random-Access Memory (SRAM) or dynamic Random-Access Memory (Dynamic Random Access Memory DRAM), where the DRAM may be a fast page mode dynamic Random-Access Memory (Fast Page Mode Dynamic Random Access Memory FPMDRAM), extended data output dynamic Random-Access Memory (Extended Date Out Dynamic Random Access Memory EDODRAM), synchronous dynamic Random-Access Memory (Synchronous Dynamic Random-Access Memory SDRAM), or the like, as appropriate.
Memory 202 may be used to store or cache various data files that need to be processed and/or communicated, as well as possible computer program instructions for execution by processor 201.
The processor 201 implements the asynchronous processing method of video migration described above by reading and executing computer program instructions stored in the memory 202.
In some of these embodiments, the computer may also include a communication interface 203 and a bus 200. As shown in fig. 4, the processor 201, the memory 202, and the communication interface 203 are connected to each other through the bus 200 and perform communication with each other.
The communication interface 203 is used to implement communications between modules, devices, units, and/or units in embodiments of the application. Communication interface 203 may also enable communication with other components such as: and the external equipment, the image/data acquisition equipment, the database, the external storage, the image/data processing workstation and the like are used for data communication.
Bus 200 includes hardware, software, or both, coupling components of a computer to each other. Bus 200 includes, but is not limited to, at least one of: data Bus (Data Bus), address Bus (Address Bus), control Bus (Control Bus), expansion Bus (Expansion Bus), local Bus (Local Bus). By way of example, and not limitation, bus 200 may include a graphics acceleration interface (Accelerated Graphics Port), AGP, or other graphics Bus, an enhanced industry standard architecture (Extended Industry Standard Architecture, EISA) Bus, front Side Bus (FSB), hyperTransport (HT) interconnect, industry standard architecture (Industry Standard Architecture, ISA) Bus, radio bandwidth (InfiniBand) interconnect, low Pin Count (LPC) Bus, memory Bus, micro channel architecture (Micro Channel Architecture, MCa) Bus, peripheral component interconnect (Peripheral Component Interconnect, PCI) Bus, PCI-Express (PCI-X) Bus, serial advanced technology attachment (Serial Advanced Technology Attachment, SATA) Bus, video electronics standards association local (Video Electronics Standards Association Local Bus, VLB) Bus, or other suitable Bus, or a combination of two or more of these. Bus 200 may include one or more buses, where appropriate. Although embodiments of the application have been described and illustrated with respect to a particular bus, the application contemplates any suitable bus or interconnect.
Example IV
In a fourth embodiment of the present application, in combination with the above-mentioned asynchronous processing method for video migration, the embodiment of the present application provides a technical solution, a storage medium, where a computer program is stored, and the computer program implements the above-mentioned asynchronous processing method for video migration when executed by a processor.
Those of skill in the art will appreciate that the logic and/or steps represented in the flowcharts or otherwise described herein, e.g., a sequence data table of executable instructions that may be considered to implement the logic functions, may be embodied in any computer readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. For the purposes of this description, a "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device.
More specific examples (a non-exhaustive list of data) of the readable medium include the following: an electrical connection (electronic device) having one or more wires, a portable computer diskette (magnetic device), a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). In addition, the computer readable medium may even be paper or other suitable medium on which the program is printed, as the program may be electronically captured, via, for instance, optical scanning of the paper or other medium, then compiled, interpreted or otherwise processed in a suitable manner, if necessary, and then stored in a computer memory.
It is to be understood that portions of the present application may be implemented in hardware, software, firmware, or a combination thereof. In the above-described embodiments, the various steps or methods may be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, may be implemented using any one or combination of the following techniques, as is well known in the art: discrete logic circuits having logic gates for implementing logic functions on data signals, application specific integrated circuits having suitable combinational logic gates, programmable Gate Arrays (PGAs), field Programmable Gate Arrays (FPGAs), and the like.
The technical features of the above-described embodiments may be arbitrarily combined, and all possible combinations of the technical features in the above-described embodiments are not described for brevity of description, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the description.
The above examples illustrate only a few embodiments of the application, which are described in detail and are not to be construed as limiting the scope of the application. It should be noted that it will be apparent to those skilled in the art that several variations and modifications can be made without departing from the spirit of the application, which are all within the scope of the application. Accordingly, the scope of protection of the present application is to be determined by the appended claims.
Claims (8)
1. An asynchronous processing method for video migration, which is realized based on a distributed processing model, is characterized in that the distributed processing model comprises a metadata management service and a thread pool; the asynchronous processing method comprises the following steps:
acquiring video object metadata of the metadata management service, and performing erasure coding processing on the video object metadata;
the thread pool obtains the current capacity of a buffer area of each disk in the disk group, and counts the number of disks of which the current capacity exceeds the preset capacity in the buffer area in the disk group;
judging whether the number of the magnetic disks exceeding the preset capacity is smaller than a threshold value, if so, delaying a first preset time to transfer the metadata of the video object subjected to erasure coding to the cache region;
migrating the video object metadata in the cache region to the disk according to a migration flow, sorting migration results of the disk according to a sorting flow, and uploading the migration results to the metadata management service;
the migration flow comprises the following steps:
acquiring the number of objects of the video object metadata in the cache region at intervals of a second preset time, and judging whether the number of objects is equal to the number of first preset objects or not;
if the number of the objects is not equal to the first preset number of objects, judging whether the number of the objects exceeds the second preset number of objects or not;
if the number of the objects exceeds a second preset number of objects, the number of the objects is sent to the disk;
the arrangement flow comprises the following steps:
creating a migration result queue, and sending the migration result of the disk group to the migration result queue;
acquiring information of the migration result in the migration result queue at intervals of fourth preset time, and counting the health degree of the disk group;
if the health degree of the disk group is insufficient, calculating time required for counting the health degree of the disk group, and if the required time is larger than a preset value;
and transmitting the information of the migration result in the migration result queue to a result report queue.
2. The method of asynchronous processing of video migration according to claim 1, wherein the step of acquiring video object metadata of the metadata management service comprises:
acquiring the preset number of video object metadata of the metadata management service, wherein the preset number is the video object metadata which can be processed by the erasure coding single time;
creating a video object metadata cache queue, and migrating the video object metadata of the preset object number to the video object metadata cache queue.
3. The asynchronous processing method for video migration according to claim 1, wherein after the step of determining whether the number of objects exceeds a second preset number of objects, the asynchronous processing method further comprises:
if the number of the objects does not exceed the second preset number of objects, repeating the step of obtaining the number of the objects of the video object metadata in the cache region at intervals of second preset time;
and judging the time spent by repeatedly executing the scanning of the object number of the video object metadata in the buffer area at intervals of the second preset time, and if the spent time exceeds the preset third preset time, sending the object number to the magnetic disk.
4. The method of asynchronous processing of video migration according to claim 1, wherein the step of uploading the migration result to the metadata management service comprises:
and acquiring the information of the migration result in the result report queue, and sending the information of the migration result to a metadata management service in batches until the information of the migration result is sent.
5. The method of asynchronous processing of video migration according to claim 1, wherein prior to the step of obtaining video object metadata of the metadata management service, the method further comprises:
and applying for obtaining the disk group, and initializing a buffer zone with a preset size for each disk in the disk group.
6. An asynchronous processing system for video migration, wherein the asynchronous processing system is realized based on a distributed processing model, and the distributed processing model comprises a metadata management service and a thread pool; the asynchronous processing system includes:
the acquisition module is used for acquiring the video object metadata of the metadata management service and performing erasure coding processing on the video object metadata;
the counting module is used for acquiring the current capacity of the buffer area of each disk in the disk group by the thread pool and counting the number of disks of which the current capacity exceeds the preset capacity in the disk group;
the judging module is used for judging whether the number of the magnetic disks exceeding the preset capacity is smaller than a threshold value or not;
the migration module is used for delaying a first preset time to migrate the metadata of the video object subjected to erasure coding to the cache region if the number of the magnetic disks exceeding the preset capacity is smaller than a threshold value;
the arrangement module is used for migrating the video object metadata in the cache region to the disk according to the migration flow, arranging the migration result of the disk according to the arrangement flow, and uploading the migration result to the metadata management service;
the arrangement module comprises a judging unit, wherein the judging unit is used for acquiring the number of objects of the video object metadata in the cache region at intervals of a second preset time and judging whether the number of objects is equal to the first preset number of objects or not; if the number of the objects is not equal to the first preset number of objects, judging whether the number of the objects exceeds the second preset number of objects or not; if the number of the objects exceeds a second preset number of objects, the number of the objects is sent to the disk;
the arrangement module further comprises a creation unit, wherein the creation unit is used for creating a migration result queue and sending the migration result of the disk group to the migration result queue; acquiring information of the migration result in the migration result queue at intervals of fourth preset time, and counting the health degree of the disk group; if the health degree of the disk group is insufficient, calculating time required for counting the health degree of the disk group, and if the required time is larger than a preset value; and transmitting the information of the migration result in the migration result queue to a result report queue.
7. A computer comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the asynchronous processing method of video migration according to any one of claims 1 to 5 when the computer program is executed by the processor.
8. A storage medium having stored thereon a computer program which, when executed by a processor, implements the asynchronous processing method of video migration according to any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311228101.XA CN116980641B (en) | 2023-09-22 | 2023-09-22 | Asynchronous processing method, system, computer and storage medium for video migration |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311228101.XA CN116980641B (en) | 2023-09-22 | 2023-09-22 | Asynchronous processing method, system, computer and storage medium for video migration |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116980641A CN116980641A (en) | 2023-10-31 |
| CN116980641B true CN116980641B (en) | 2023-12-15 |
Family
ID=88477070
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311228101.XA Active CN116980641B (en) | 2023-09-22 | 2023-09-22 | Asynchronous processing method, system, computer and storage medium for video migration |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116980641B (en) |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CA2350721A1 (en) * | 2001-06-15 | 2002-12-15 | Ibm Canada Limited-Ibm Canada Limitee | Method and apparatus for chunk based transaction logging with asynchronous input/output for a database management system |
| CN111142788A (en) * | 2019-11-29 | 2020-05-12 | 浪潮电子信息产业股份有限公司 | Data migration method and device and computer readable storage medium |
| KR20200057409A (en) * | 2018-11-16 | 2020-05-26 | 네이버 주식회사 | Distributed file system and data processing method for live service |
| CN111629028A (en) * | 2020-04-10 | 2020-09-04 | 清华大学 | Data transmission scheduling system for distributed multi-cloud storage |
| CN112241398A (en) * | 2019-07-19 | 2021-01-19 | 北京京东尚科信息技术有限公司 | Data migration method and system |
| CN113296908A (en) * | 2021-04-30 | 2021-08-24 | 重庆紫光华山智安科技有限公司 | Method, device, equipment and medium for improving video image access speed |
| WO2023000670A1 (en) * | 2021-07-19 | 2023-01-26 | 华为技术有限公司 | Data writing method and apparatus, data reading method and apparatus, and device, system and medium |
| CN116661705A (en) * | 2023-07-24 | 2023-08-29 | 江西云眼视界科技股份有限公司 | Data management method, system, electronic equipment and storage medium based on kafka |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110096220B (en) * | 2018-01-31 | 2020-06-26 | 华为技术有限公司 | Distributed storage system, data processing method and storage node |
| US11055010B2 (en) * | 2019-09-05 | 2021-07-06 | Microsoft Technology Licensing, Llc | Data partition migration via metadata transfer and access attribute change |
-
2023
- 2023-09-22 CN CN202311228101.XA patent/CN116980641B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CA2350721A1 (en) * | 2001-06-15 | 2002-12-15 | Ibm Canada Limited-Ibm Canada Limitee | Method and apparatus for chunk based transaction logging with asynchronous input/output for a database management system |
| KR20200057409A (en) * | 2018-11-16 | 2020-05-26 | 네이버 주식회사 | Distributed file system and data processing method for live service |
| CN112241398A (en) * | 2019-07-19 | 2021-01-19 | 北京京东尚科信息技术有限公司 | Data migration method and system |
| CN111142788A (en) * | 2019-11-29 | 2020-05-12 | 浪潮电子信息产业股份有限公司 | Data migration method and device and computer readable storage medium |
| WO2021103596A1 (en) * | 2019-11-29 | 2021-06-03 | 浪潮电子信息产业股份有限公司 | Data migration method, device, and computer-readable storage medium |
| CN111629028A (en) * | 2020-04-10 | 2020-09-04 | 清华大学 | Data transmission scheduling system for distributed multi-cloud storage |
| CN113296908A (en) * | 2021-04-30 | 2021-08-24 | 重庆紫光华山智安科技有限公司 | Method, device, equipment and medium for improving video image access speed |
| WO2023000670A1 (en) * | 2021-07-19 | 2023-01-26 | 华为技术有限公司 | Data writing method and apparatus, data reading method and apparatus, and device, system and medium |
| CN116661705A (en) * | 2023-07-24 | 2023-08-29 | 江西云眼视界科技股份有限公司 | Data management method, system, electronic equipment and storage medium based on kafka |
Non-Patent Citations (4)
| Title |
|---|
| Scalable and secure virtualization of HSM with scaletrust;Juhyeng Han, et al.;《IEEE/ACM Transaction on Networking》;全文 * |
| 基于固态硬盘的云存储分布式缓存策略;李东阳;刘鹏;丁科;田浪军;;计算机工程(04);全文 * |
| 基于水平纠删码的云存储数据布局方法;蒋海波等.;《四川大学学报(工程科学版)》;全文 * |
| 海量信息分级存储数据迁移策略研究;吕帅;刘光明;徐凯;刘欣;;计算机工程与科学(S1);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116980641A (en) | 2023-10-31 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10826526B2 (en) | Memory system and information processing system | |
| CN110673798B (en) | Storage system and IO (input/output) tray-dropping method and device thereof | |
| JP2012521042A (en) | Web front end throttling | |
| CN110209502B (en) | Information storage method and device, electronic equipment and storage medium | |
| CN111651595A (en) | Abnormal log processing method and device | |
| CN110955390B (en) | Data processing method, device, electronic equipment and storage medium | |
| CN116980641B (en) | Asynchronous processing method, system, computer and storage medium for video migration | |
| CN101227689B (en) | Method and apparatus for reporting information | |
| CN112596985B (en) | IT asset detection method, device, equipment and medium | |
| CN117251275A (en) | Multi-application asynchronous I/O request scheduling method, system, equipment and medium | |
| US11194619B2 (en) | Information processing system and non-transitory computer readable medium storing program for multitenant service | |
| CN116089477B (en) | Distributed training method and system | |
| CN116909862A (en) | Method and device for collecting exception logs, electronic equipment and storage medium | |
| JP2001077813A (en) | Network information management system, network information management method and recording medium recording its processing program | |
| CN114817860B (en) | Data statistical method and device, solid state disk and medium | |
| CN112015527B (en) | Managing fetching and executing commands from a commit queue | |
| CN112131180A (en) | Data reporting method and device and storage medium | |
| CN115794446B (en) | Message processing method and device, electronic equipment and storage medium | |
| CN117648287B (en) | On-chip data processing system, method, server and electronic equipment | |
| CN116991781B (en) | Request processing device, method, chip, storage medium and electronic equipment | |
| CN114301927B (en) | Main node selection method, device and medium in distributed system | |
| CN117472597B (en) | Input/output request processing method, system, electronic device and storage medium | |
| CN116149557B (en) | Manufacturing-oriented slow disk detection strategy system and method | |
| CN110430264B (en) | Data reading method in cloud computing | |
| CN111399779B (en) | Flow control method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |