CN115934836A

CN115934836A - Data storage synchronization method of multi-content center

Info

Publication number: CN115934836A
Application number: CN202211562016.2A
Authority: CN
Inventors: 苏帅; 白松; 苏平宇
Original assignee: Tianyi Cloud Technology Co Ltd
Current assignee: Tianyi Cloud Technology Co Ltd
Priority date: 2022-12-07
Filing date: 2022-12-07
Publication date: 2023-04-07

Abstract

The invention provides a data storage synchronization method of a multi-content center, which comprises the following steps: slicing the multimedia file to form a subfile and an index file; saving the file and the index file; recording the index file to a preset database; the databases are synchronized. According to the method, the HLS content synchronization stored in the multi-content center is realized through the database record storage list, the consistency of the multi-content center storage is kept, a special key format for storing m3u8 is formulated by means of HLS characteristics, all ts list indexes are found through m3u8, and the resource pool where the storage is located can be clearly known.

Description

Data storage synchronization method of multi-content center

Technical Field

The application relates to the field of IT software storage, in particular to a data storage synchronization method of a multi-content center.

Background

After the content is introduced into the sky wing high definition content center, in order to better provide service for IPTV users, for the content injected on demand, the injected large on demand file is often stored in a storage (CEPH) in a manner of being converted into HLS, which is easy to distribute. However, a single content center is often difficult to meet the requirement of customer access nearby and fail-over, so that a plurality of content centers are required to be created to store and inject content to meet the requirement of multi-center disaster tolerance and user access nearby. The synchronization of the multi-center stored content becomes a difficult point for realizing, and if a mode of multiple copies in different places is adopted, the requirement on a network is high, and the whole cluster can be influenced when a fault occurs. According to the scheme, the data synchronization of the HLS of the multiple content centers is realized, the complete consistency of the data stored in the multiple content centers is ensured, the faults are not influenced mutually, and the real high availability of the remote center is realized.

Disclosure of Invention

The embodiment of the application provides a data storage synchronization method for a multi-content center, so as to improve the technical problem.

The embodiment of the application provides a data storage synchronization method for a multi-content center, which comprises the following steps: slicing the multimedia file to form a subfile and an index file; saving the file and the index file; recording the index file to a preset database; the databases are synchronized.

In some embodiments, the recording the index file to a preset database includes: recording the unique identification of the file; an index file is injected.

In some embodiments, the method further comprises: and deleting the unique identifier of the file.

In some embodiments, synchronizing the database content comprises: and acquiring a distributed lock, and traversing the live table if the distributed lock is not acquired.

In some embodiments, the traversal moive table is limited to 1000 entries.

In some embodiments, the method further includes comparing the storage information in the database with the storage information configured by the service, and taking a difference set by the two lists, and if the configured storage information is in the storage in the database, directly updating the synchronization task state to be completed without synchronization.

In some embodiments, if synchronization is required, the information is written to a synchronized message queue.

In some embodiments, it is determined whether the traversed data is the last data, and if not, the storage in the database is compared with the storage information configured by the service again.

In some embodiments, synchronizing the database further comprises reading a queue, obtaining tasks that need to be synchronized, and obtaining m3u8 index keys of HLSs and a storage center of content.

In some embodiments, the method further comprises: deleting the task acquisition process; acquiring traversed data information, putting the traversed data information into a queue, and updating the deletion state of the database into deletion; and deleting the task processing progress.

The application has the following advantages:

the HLS content stored in the CEPH is synchronized in real time based on a database mode, m3u8 index keys are stored in the database by virtue of HLS characteristics, objects in the CEPH do not need to be traversed, all content indexes can be obtained only by analyzing m3u8, a storage list is recorded by the database, the HLS content synchronization stored in a multi-content center is realized, the consistency of multi-content center storage is kept, a special key format for storing m3u8 is formulated by virtue of the HLS characteristics, all ts list indexes are found by virtue of m3u8, and a resource pool where the storage is located can be clearly known. The content synchronization and deletion are completed by independent processes, and the task acquisition and the dynamic expansion of the task processing process are supported through message queue decoupling. The centers needing synchronization can be configured in multiple numbers through configuration files, and incremental center content synchronization is supported.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.

Fig. 1 is a schematic diagram of field information according to an embodiment of the present application;

fig. 2 is a schematic diagram of another field information proposed in the embodiment of the present application;

FIG. 3 is a schematic diagram of a logical relationship of components according to an embodiment of the present application;

fig. 4 is a schematic diagram illustrating a component logic synchronization task obtaining process according to an embodiment of the present application;

fig. 5 is a schematic diagram illustrating a processing procedure of a component logic synchronization task according to an embodiment of the present application;

fig. 6 is a schematic diagram of a component logic deletion task obtaining process according to an embodiment of the present application;

fig. 7 is a schematic diagram of a processing procedure for deleting a task by component logic according to an embodiment of the present disclosure;

fig. 8 is a content diagram with a Contentid of 1 according to an embodiment of the present application.

Detailed Description

In order to make the technical solution better understood by those skilled in the art, the technical solution in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. It should be apparent that the described embodiments are only some embodiments of the present application, and not all embodiments. All other embodiments, which can be obtained by a person skilled in the art without making any inventive step based on the embodiments in the present application, are within the scope of protection of the present application.

Furthermore, the terms "first," "second," and the like are used merely for distinguishing between descriptions and not intended to imply or imply a particular structure. The description of the terms "some embodiments," "other embodiments," or the like, means that a particular feature, structure, material, or characteristic described in connection with the embodiments or examples is included in at least one embodiment or example of the application. In this application, the schematic representations of the terms used above are not necessarily intended to refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, the various embodiments or examples and features of the various embodiments or examples described in this application can be combined and combined by those skilled in the art without conflicting.

The HLS slices the multimedia file or direct stream to form a stack of ts files and m3u8 index files and saves to disk or storage. In colloquial, all ts file lists are recorded in the m3u8 index file, and all ts files can be found through the m3u8 file. According to the HLS technical principle, all ts files can be found by only recording the contents of the m3u8 index file in the database.

CEPH is a reliable, automatic rebalancing, automatic recovery distributed storage system, which can be divided into three large blocks according to scene division, namely object storage, block device and file system service. The scheme mainly uses object storage of CEPH.

According to the method, HLS contents stored in the CEPH are synchronized in real time based on a database mode, m3u8 index keys are stored in the database by virtue of HLS characteristics, objects in the CEPH do not need to be traversed, all content indexes can be obtained only by analyzing m3u8, the HLS contents stored in a plurality of content centers are synchronized by recording a storage list through the database, and the storage consistency of the plurality of content centers is kept. Injection database design

Referring to the HSL technical principle, the database records the following information when synchronizing:

contentid content unique identifier

url index File (m 3u 8) url of injected content

storage (list), which storage center the currently injected content is stored in, if the content is injected into Beijing content center, the record value is: [ "bj" ]

status (int): status is current task, 0: transcoding process, 1: the transcoding process is completed and synchronization can be performed

is _ sync, if synchronous to other content centers, 0: to be synchronized, 1: in synchronization, 2: synchronization is completed, 3: synchronization failure

The field information is as shown in FIG. 1

Deleting database designs

When deleting, the database deletion table records the following information:

ID: delete record id, auto-generate

Contentid content unique identifier

url index File (m 3u 8) url of injected content

storage (list) which storage centers the currently injected content is stored in,

is _ delete: whether deleted, 0: to be deleted, 1: in deletion, 2: deletion completion, 3: deletion failure

The field information is as shown in FIG. 2

In some embodiments, the recording the index file to a preset database includes:

recording the unique identification of the file;

an index file is injected.

In some embodiments, the traversal moive table is limited to 1000 entries.

Specifically, the synchronization service producer can configure the number of the storages to be synchronized, compare the configured number of the storages with the database storage, take out and place the contents to be synchronized into a queue, and update corresponding database fields, for example, the configuration storage information in the producer is as follows: and if a certain piece of information in the database is recorded as [ bj ], [ nj ], [ gz ], [ the information is recorded as [ bj ], [ the information is recorded as gz ], the record is synchronized to two content storage centers of nj and gz.

And the synchronization service worker consumes the queue content, executes specific writing (deleting) other storage center tasks, and updates the database field after the operation is finished.

Operations related to CEPH storage are all operated by using a native rados library, and are packaged without a gateway layer, so that the efficiency is higher.

The logical relationship of the components is as in figure 3.

6. Injection synchronization scheme implementation

The injected content synchronization mainly depends on a database to store injected content indexes and a storage center, the content center storage in the database is compared with the content center storage configured by a configuration file, a difference set (namely the storage in the product configuration file and the storage which does not exist in the database, such as the storage shown in fig. 7) is taken to obtain the storage to be synchronized, the index keys are analyzed from the central storage center of the stored data through the index keys recorded in the data, all the index keys (m 3u8 and ts) of the HLS content are obtained, an index list is traversed, the content in the storage is read, and the content is written into other un-synchronized storages.

The specific implementation process comprises the following steps:

synchronous task acquisition Process (independent Process), FIG. 4

1) And (3) acquiring the distributed lock (redis implementation), if the distributed lock is not acquired, re-acquiring, and if the distributed lock is acquired successfully, turning to the step 2).

2) Traversing the live table, limiting 1000 items (can be adjusted according to needs) for each traversal, increasing query speed, searching tasks which are completed in transcoding and are to be synchronized, if not, sleeping, turning to step 1), and if the tasks are to be synchronized, turning to step 3)

3) And aiming at the traversed data information, comparing the storage information in the database with the storage information configured through the service, and taking a difference set by two lists, wherein if the configured storage information is in the storage in the database, the synchronization is not needed, the state of the synchronization task is directly updated to be finished, and if the synchronization is needed, the information is written into a synchronous message queue.

4) And (3) judging whether the traversed data is the last data, if not, turning to the step 3, if so, turning to the step 1), and if so, turning to the step 1).

5) For the synchronized data, when a new storage needs to be synchronized, only the data is _ sync state needs to be modified to be synchronized, and a synchronization task can be set by self.

Synchronous task processing Process (independent Process), FIG. 5

1) And reading the queue, acquiring tasks needing synchronization, and acquiring an m3u8 index key of the HLS and a storage center of the content.

2) And reading the content of the content center key according to the HLS index key, acquiring all ts index key lists and m3u8 lists, integrating the ts index key lists and the m3u8 lists into an index list needing synchronization, and comparing the acquired storage with the configured storage to acquire a storage list needing synchronization.

3) Traversing the index list, reading index content, circularly writing storage needing synchronization, updating the synchronization state in the database movie table to be completed after writing is completed, adding the storage after synchronization to the database storage list, and directly updating the state to be a failure state without updating the storage list if the synchronization fails.

When the task amount is large, the expansion can be realized by directly increasing the synchronous task processing process, and the synchronous speed is improved.

7. Deletion synchronization scheme implementation

The basic idea of injected content deletion is the same as the synchronization scheme, the difference is that the storage information is not compared during deletion, ts indexes are preferentially deleted, and m3u8 is deleted finally

The index aims to avoid re-deletion when abnormality occurs in the process of deletion, ts index can be found well according to m3u8, dirty data is reduced, and when a central service receives a deletion instruction during deletion, a deletion task needs to be placed in a deletion table delete _ drive, and isdelet is set to be in a state to be deleted.

The specific implementation process comprises the following steps:

delete task get process (independent process), fig. 6.

1) And (4) acquiring the distributed lock (redis implementation), and if the distributed lock is not acquired, re-acquiring, and if the acquisition is successful, turning to the step 2).

2) Traversing the delete _ mive table, limiting 1000 items (can be adjusted according to needs) in each traversal, increasing the query speed, searching for the task to be deleted, if not found, sleeping, turning to the step 1), and if the task to be deleted, turning to the step 3)

3) And acquiring the traversed data information, putting the traversed data information into a queue, and updating the deletion state of the database into deletion (the delete is set to be 1).

Delete task processing process (independent process), fig. 7.

1) Reading the queue, acquiring the task needing synchronization, and acquiring the m3u8 index key of the HLS and the storage center of the content.

2) And reading the content of the content center key according to the HLS index key, and acquiring all ts index key lists and m3u8 lists to be integrated into an index list needing synchronization.

3) Traversing the index list, traversing the storage list, circularly deleting the storage key, and after the deletion is finished, updating the deletion state in the delete _ movie table of the database to be finished, if the deletion failure directly sets the state to be the deletion failure. And if the same Contentid exists in the movie table, updating the movie table storage field and removing the deleted storage.

Example (c):

content with Contentid of 1 (see fig. 8):

1) In the movie table, the storage is [ "bj", "nj", "gz" ], and in the delete _ movie table, the storage is [ "bj" ]

2) After the task is deleted, updating the storage data of the movie table except for the deletion of the task in the delete _ movie table, and realizing the dynamic deletion of the single-center data

When the task amount is large, the expansion can be realized by directly increasing the synchronous task processing process, and the synchronous speed is submitted.

Compared with the prior art, the application has the advantages and effects.

The synchronization method and the synchronization system provided by the invention have the following advantages that:

1. by means of HLS characteristics, a special key format for storing m3u8 is established, all ts list indexes are found through m3u8, and a resource pool where the storage is located can be clearly known.

2. And content synchronization and deletion are completed by independent processes, and task acquisition and dynamic expansion of a task processing process are supported through message queue decoupling.

3. The centers needing synchronization can be configured in multiple numbers through configuration files, and incremental center content synchronization is supported.

According to a fourth aspect of the present disclosure, there is also provided a computer program product containing instructions which, when run on a computer, make the computer perform the message middleware AOP based cluster load balancing method in the present disclosure.

According to a fifth aspect of the present disclosure, there is also provided a computer-readable storage medium having stored therein instructions, which when run on a computer, cause the computer to execute the message middleware AOP-based cluster load balancing method in the present disclosure.

The computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a Read-Only Memory (ROM), an Erasable Programmable Read-Only Memory (EPROM), a register, a hard disk, an optical fiber, a portable Compact Disc Read-Only Memory (CD-ROM), an optical storage device, a magnetic storage device, any suitable combination of the above, or any other form of computer readable storage medium known in the art. An exemplary storage medium is coupled to the processor such the processor can read information from, and write information to, the storage medium. Of course, the storage medium may also be integral to the processor. The processor and the storage medium may reside in an Application Specific Integrated Circuit (ASIC). In the present disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.

In the embodiments provided in the present disclosure, it should be understood that the disclosed system, apparatus and method may be implemented in other ways. For example, the above-described embodiments of the apparatus are merely illustrative, and for example, a division of a unit is merely a logical division, and an actual implementation may have another division, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.

In the present disclosure, units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may also be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.

The foregoing description is only exemplary of the preferred embodiments of the disclosure and is illustrative of the principles of the technology employed. It will be appreciated by those skilled in the art that the scope of the disclosure herein is not limited to the particular combination of features described above, but also encompasses other embodiments in which any combination of the features described above or their equivalents does not depart from the spirit of the disclosure. For example, the above features and (but not limited to) the features disclosed in this disclosure having similar functions are replaced with each other to form the technical solution.

It should be noted that, in the above embodiments, the various features described in the above embodiments may be combined in any suitable manner, and in order to avoid unnecessary repetition, various possible combinations will not be further described in the present disclosure.

Further, while operations are depicted in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Likewise, while several specific implementation details are included in the above discussion, these should not be construed as limitations on the scope of the disclosure. Certain features that are described in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination.

Furthermore, any combination between various embodiments of the disclosure may be made without departing from the spirit thereof, which is also to be considered as disclosed herein, and although the subject matter has been described in language specific to method logic acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims. With regard to the apparatus in the above-described embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be elaborated here.

The above embodiments are only for illustrating the technical solutions of the present application, and not for limiting the same; although the present application has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may be modified or some technical features may be equivalently replaced; such modifications and substitutions do not substantially depart from the spirit and scope of the embodiments of the present application and are intended to be included within the scope of the present application.

Claims

1. A data storage synchronization method of a multi-content center is characterized by comprising the following steps:

slicing the multimedia file to form a subfile and an index file;

saving the file and the index file;

recording the index file to a preset database;

the databases are synchronized.

2. The method of claim 1, wherein the recording the index file to a predetermined database comprises:

recording the unique identification of the file;

the index file is injected.

3. The method of claim 2, further comprising:

and deleting the unique identification of the file.

4. The method of claim 1, wherein synchronizing the database content comprises:

and acquiring a distributed lock, and traversing the live table if the distributed lock is not acquired.

5. The method of claim 4, wherein traversing the moive table is limited to 1000 pieces.

6. The method of claim 4, further comprising,

and comparing the storage information in the database with the storage information configured through service, wherein the difference set is obtained by the two lists, and if the configured storage information is in the storage in the database, the synchronization is not needed, and the synchronization task state is directly updated to be completed.

7. The method of claim 6, wherein if synchronization is required, writing the information to a synchronized message queue.

8. The method according to claim 4, characterized in that, whether the traversed data is the last data is judged, if not, the storage in the database is compared with the storage information configured by the service again.

9. The method of claim 8, wherein the synchronizing the database further comprises:

reading the queue, acquiring the task needing synchronization, and acquiring the m3u8 index key of the HLS and the storage center of the content.

10. The method of claim 4, further comprising:

deleting the task acquisition process;

acquiring traversed data information, putting the traversed data information into a queue, and updating the deletion state of the database into deletion;

and deleting the task processing progress.