CN110569217A - index data updating method and device in streaming file system - Google Patents
index data updating method and device in streaming file system Download PDFInfo
- Publication number
- CN110569217A CN110569217A CN201810470680.1A CN201810470680A CN110569217A CN 110569217 A CN110569217 A CN 110569217A CN 201810470680 A CN201810470680 A CN 201810470680A CN 110569217 A CN110569217 A CN 110569217A
- Authority
- CN
- China
- Prior art keywords
- index
- time
- data
- block
- updating
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/13—File access structures, e.g. distributed indices
Abstract
The embodiment of the invention provides an index data updating method and device in a streaming file system, wherein the index data updating method in the streaming file system comprises the following steps: acquiring index time of an index block; judging whether an intersection exists between the time index and the pre-updating time range; if the intersection exists, determining the index block as an index block to be processed; reading each index data stored in the index block to be processed; and updating the index data with the data writing time within the pre-updating time range. According to the scheme, the updating efficiency of the index data in the streaming file system can be improved.
Description
Technical Field
The present invention relates to the field of data management technologies, and in particular, to a method and an apparatus for updating index data in a streaming file system.
Background
In recent years, with the continuous development of video monitoring technology, the data volume of video data required to be processed is increasing, for example, in the security industry, the video data is collected in real time, the collected video data has continuity on a time axis, and the video data is infinitely increased along with the time, and this type of data is generally called streaming data.
Stream data is often managed by a stream file system, which mainly comprises a main start block, a standby start block, an object block group and a reserved block, where the object block group includes a main index block, a standby index block and an object block (the contents of the main index block and the standby index block are completely the same and are backed up with each other), a specific structure diagram is shown in fig. 1, where the object block is used to store stream data, the index block is used to store index data (such as data write time, data size, etc.) of stream data in the object block, and there is a corresponding relationship between the index block and the object block.
when the streaming data stored in the object block changes, the index data in the index block corresponding to the object block needs to be updated correspondingly, so as to achieve the purpose that the index data in the index block corresponds to the streaming data in the object block. Specifically, if the index data with the data writing time within the preset time range is to be updated, because the number of the index blocks is not unique, the index block where the index data to be updated is located cannot be directly determined, and therefore, an object block stored in the streaming data with the data writing time within the preset time range needs to be found first, then the index block to be processed is found through the corresponding relationship between the object block and the index block, and then the index data stored in the index block to be processed is updated. When a plurality of index data need to be updated, the updating operation needs to be performed for each index data, and the updating efficiency is low when the plurality of index data are updated due to the complex implementation of the updating operation of the index data.
disclosure of Invention
The embodiment of the invention aims to provide a method and a device for updating index data in a streaming file system, so as to solve the problem of low efficiency of updating the index data in the streaming file system. The specific technical scheme is as follows:
In a first aspect, an embodiment of the present invention provides a method for updating index data of a streaming file system, where the method includes:
Acquiring index time of an index block, wherein the index time is a time range determined based on data writing time of streaming data stored in an object block corresponding to the index block;
judging whether an intersection exists between the time index and a pre-updating time range;
if the intersection exists, determining the index block as an index block to be processed;
reading each index data stored in the index block to be processed, wherein the index data comprises data writing time of streaming data stored in an object block corresponding to the index block to be processed;
and updating the index data with the data writing time within the pre-updating time range.
Further, the pre-update time range is: and the time range is less than or equal to the acquired pre-updating time point.
Further, before the update data is written into the index data whose time is within the pre-update time range, the method further includes:
judging whether the time index of the index block to be processed is completely contained in the pre-updating time range;
the updating the index data of which the data writing time is within the pre-updating time range comprises:
And if the time index of the index block to be processed is completely contained in the pre-updating time range, updating all index data in the index block to be processed.
Further, the updating the index data of which the data writing time is within the pre-update time range includes:
and if the time index of the index block to be processed is not completely contained in the pre-updating time range, updating the index data of which the data writing time in the index block to be processed is within the overlapping time range of the time index of the index block to be processed and the pre-updating time range.
further, the obtaining the index time of the index block includes:
acquiring the earliest write-in time and the latest write-in time in data write-in time in index data of the index block;
and determining a time range which is greater than or equal to the earliest writing time and less than or equal to the latest writing time as the index time of the index block.
In a second aspect, an embodiment of the present invention provides an apparatus for updating index data in a streaming file system, where the apparatus includes:
The index time acquisition module is used for acquiring index time of an index block, wherein the index time is a time range determined based on data writing time of streaming data stored in an object block corresponding to the index block;
The first judgment module is used for judging whether an intersection exists between the time index and the pre-updating time range;
The index block to be processed determining module is used for determining the index block to be processed as an index block to be processed when intersection exists between the time index and a pre-updating time range;
a data reading module, configured to read each index data stored in the to-be-processed index block, where the index data includes data write time of streaming data stored in an object block corresponding to the to-be-processed index block;
And the data updating module is used for updating the index data of which the data writing time is within the pre-updating time range.
further, the pre-update time range is: and the time range is less than or equal to the acquired pre-updating time point.
further, the apparatus further comprises:
The second judgment module is used for judging whether the time index of the index block to be processed is completely contained in the pre-updating time range;
the data updating module is specifically configured to update all index data in the index block to be processed if the time index of the index block to be processed is completely included in the pre-update time range.
further, the data updating module is further configured to update the index data in the index block to be processed, in which the data writing time is within an overlapping time range of the time index of the index block to be processed and the pre-update time range, if the time index of the index block to be processed is not completely included in the pre-update time range.
Further, the index time obtaining module is specifically configured to:
acquiring the earliest write-in time and the latest write-in time in data write-in time in index data of the index block;
And determining a time range which is greater than or equal to the earliest writing time and less than or equal to the latest writing time as the index time of the index block.
In a third aspect, an embodiment of the present invention provides an electronic device, including a processor and a memory;
a memory for storing a computer program;
And the processor is used for realizing the steps of the index data updating method in any streaming file system when executing the program stored in the memory.
in a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, where instructions are stored, and when the instructions are executed on a computer, the instructions cause the computer to execute an index data updating method in any streaming file system described above.
The embodiment of the invention provides a method, a device and an electronic device for updating index data in a streaming file system, which are used for acquiring the index time of an index block, wherein the index time is a time range determined based on the data writing time of streaming data stored in an object block corresponding to the index block; judging whether an intersection exists between the time index and the pre-updating time range; if the intersection exists, determining the index block as an index block to be processed; reading each index data stored in the index block to be processed; the index data comprises data writing time of streaming data stored in an object block corresponding to the index block to be processed; and updating the index data with the data writing time within the pre-updating time range. The method comprises the steps of determining an index block to be processed by judging whether intersection exists between a time index and a pre-updating time range, and updating index data with data writing time within the pre-updating time range, wherein the index data with the data writing time within the pre-updating time range comprise a plurality of index data, namely, the plurality of index data are synchronously updated, the time required by updating the index data is reduced, and the updating efficiency of the index data in a streaming file system is improved.
drawings
in order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
FIG. 1 is a block diagram of a streaming file system;
Fig. 2 is a flowchart illustrating an index data updating method in a streaming file system according to an embodiment of the present invention;
Fig. 3 is a flowchart illustrating an index data updating method in a streaming file system according to another embodiment of the present invention;
Fig. 4 is a schematic structural diagram of an index data updating apparatus in a streaming file system according to an embodiment of the present invention;
fig. 5 is a schematic structural diagram of an index data updating apparatus in a streaming file system according to another embodiment of the present invention;
fig. 6 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
In order to improve the index data updating efficiency in the streaming file system, embodiments of the present invention provide an index data updating method and apparatus in the streaming file system, and an electronic device.
for streaming data with time continuity, a streaming file system is often used to manage the data, and first, the structural composition of the streaming file system is described with reference to fig. 1. The streaming file system is composed of a main starting block, a standby starting block, an object block group and a reserved block. The starting block is used for storing key starting information of the file system; the object block group includes a main index block, a standby index block and object blocks (the main index block and the standby index block have the same content and are backed up with each other), wherein the object blocks are used for storing streaming data, the index blocks are used for storing index data of the streaming data in the object blocks, and the index blocks and the object blocks have corresponding relations, generally, one index block can store the index data of a plurality of object blocks, for example, when the size of the index data of each object block is 4KB and the size of each index block is 1MB, each index block can store the index data of 256 object blocks; the reserved block is used for replacing a damaged starting block or index block when the starting block or index block in the object block group is damaged.
the following describes a method for updating index data in a streaming file system according to an embodiment of the present invention.
The index data update in the embodiment of the present invention may include deletion of the index data, or may include an overlay update of the index data.
as shown in fig. 2, the method for updating index data in a streaming file system according to an embodiment of the present invention may include the following steps:
Step 101, obtaining the index time of the index block.
The index time is a time range determined based on the data writing time of the streaming data stored in the object block corresponding to the index block. Generally, one index block can store index data of a plurality of object blocks, for example, in the streaming file system shown in fig. 1, the index data size of each object block is 4KB, and the index block size of each index block is 1MB, so that the index data of 256 object blocks can be stored in each index block.
The index time of each index block may be a time range that is pre-established and stored based on the data write time of the streaming data stored in the object block corresponding to the index block.
The index time of the index block is a time range representing the data writing time of the streaming data stored in the object block corresponding to the index block, and the data writing time of the streaming data stored in the object block is within the index time. For example, the index data of the streaming data in the object blocks No. 1 to No. 256 is stored in the index block No. 1, and the data writing time range of the streaming data is from 8 am to 9 am on 31 am of 12 th month in 2017, the index time of the index block may be the same as the time range, that is, from 8 am to 9 am on 31 am of 12 th month in 2017, or may be any time range including the time range, such as from 7 am to 10 am on 31 am of 12 th month in 2017, from 7 am to 9 am on 31 am of 12 th month in 2017, or from 8 am to 11 am on 31 am of 31 of 12 th month in 2017, and the like, and this is not limited.
By obtaining the index time of each index block, the time range to which the data write-in time of the streaming data stored in the object block corresponding to each index block belongs can be obtained, and the index block to which the index data with the write-in time within the preset time range is located can also be directly determined without searching the object block stored in the streaming data with the data write-in time within the preset time range, and then the index block to which the index data with the write-in time within the preset time range is located can be searched according to the corresponding relationship between the object block and the index block. The efficiency of index block searching is improved.
Further, the step of obtaining the index time of the index block may include:
acquiring the earliest write-in time and the latest write-in time in data write-in time in index data of an index block;
a time range greater than or equal to the earliest write time and less than or equal to the latest write time is determined as an index time of the index block.
because the streaming data has continuity on the time axis, the index time of the index blocks is set to be greater than or equal to the earliest writing time and less than or equal to the latest writing time, so that the condition that intersection exists between the index times corresponding to adjacent index blocks can be avoided, and meanwhile, when the data writing time in certain index data is known, the only index block for storing the index data can be determined according to the index time of each index block, and the index block searching efficiency is further improved.
And 102, judging whether an intersection exists between the time index and the pre-updating time range.
The pre-update time range may be a specific time range that needs to be updated by the user according to needs, for example, when the user needs to update the index data with the data writing time between 8 am and 9 am on 31 am of 12 and 31 months in 2017, the pre-update time range is: 8 am to 9 am on 31 am of 12 and 31 months in 2017; the pre-update time range may also be all time periods outside a specific time range specified by the user, for example, when the user needs to update all index data whose data write time is outside 8 am to 9 am on 31.12.2017, the pre-update time range is: a time range earlier than 8 am on 12/31/2017 and later than 9 am on 12/31/2017; the pre-update time range may also be a time range before a particular point in time specified by the user.
further, the pre-update time range may be: and the time range is less than or equal to the acquired pre-updating time point.
For example, the obtained pre-update time point is 8 am on 31/h in 2017, 12/h, and at this time, the pre-update time range may be a time range earlier than or equal to 8 am on 31/h in 2017, 12/h.
And 103, if the intersection exists, determining the index block as the index block to be processed.
There is an intersection between the time index and the pre-update time range, which may include the following 3 cases: the time index of the index block to be processed is completely contained in the pre-updating time range; the pre-updating time range is completely contained in the time index of the index block to be processed; there is only a partial overlapping time range between the time index of the block of indexes to be processed and the pre-update time range.
And 104, reading each index data stored in the index block to be processed.
The index data comprises data writing time of streaming data stored in an object block corresponding to the index block to be processed. When the stream data is written into the object block, the data writing time of the stream data is stored in the index block corresponding to the object block, and the data size, the verification information and the like of the stream data can also be stored in the index block, and the data writing time, the data size, the verification information and the like can be contained in the index data.
And step 105, updating the index data with the data writing time within the pre-updating time range.
the case of intersection between the time index and the pre-update time range in step 103 can be further summarized into two types: the time index of the index block to be processed is completely contained in the pre-updating time range; the time index of the pending index chunk is not completely contained within the pre-update time range.
further, before updating the index data whose data writing time is within the pre-update time range, the method further includes:
Judging whether the time index of the index block to be processed is completely contained in the pre-updating time range;
The updating the index data whose data writing time is within the pre-update time range may include:
And if the time index of the index block to be processed is completely contained in the pre-updating time range, updating all index data in the index block to be processed. For example, the index block to be processed is an index block 1, wherein the time index of the index block 1 is from 7 am to 8 am on 31 am in 12/month in 2017, and if the pre-update time range is from 0 am to 11 am on 31 am in 12/month in 2017, at this time, the time index of the index block to be processed is completely included in the pre-update time range, that is, the data writing time in the index data stored in the index block 1 is in the pre-update time range, all the index data in the index block to be processed (i.e., the index block 1) may be updated.
Further, if the time index of the index block to be processed is not completely included in the pre-update time range, the index data in the index block to be processed, the data writing time of which is within the overlapping time range of the time index of the index block to be processed and the pre-update time range, is updated. For example, the index block to be processed is the index block 2, wherein the time index of the index block 2 is 9 am to 11 am on 31 am in 12/31 month in 2017, and if the pre-update time range is 10 am to 11 am on 31 am in 12/31 month in 2017, at this time, the time index of the index block to be processed is not completely included in the pre-update time range, that is, only a part of the overlapping time range (10 am to 11 am on 31 am in 12/31 month in 2017) between the index time in the index block 2 and the pre-update time range, the index data in the overlapping time range can be updated.
after the index data stored in the index block to be processed is read, when the index data with the data writing time within the pre-updating time range is updated, whether the time index of the index block to be processed is completely contained in the pre-updating time range is judged, and when the time index of the index block to be processed is completely contained in the pre-updating time range, the data writing time of all the index data in the index block to be processed is inevitably within the pre-updating time range, at the moment, all the index data in the index block to be processed can be directly updated, so that the updating time of the index data can be further shortened, and the updating efficiency is improved.
In the method for updating index data in a streaming file system according to the embodiment of the present invention, the steps 101 to 105 may be sequentially performed for each index block in the file system according to a preset sequence; the updating of the index data may also be done as follows: after the index time of all index blocks in the file system is obtained, judging whether intersection exists between all the index time and a pre-updating time range; and determining the index blocks corresponding to the time indexes with intersection as index blocks to be processed, and executing the step 104 and the step 105 to the index blocks to be processed simultaneously, or selecting one index block from the index blocks to be processed each time according to a selection sequence set by a user, and executing the step 104 and the step 105 to the selected index block until the index blocks to be processed are completely updated.
In the index data updating method in the streaming file system shown in fig. 1 provided in the embodiment of the present invention, index time of an index block determined based on data write time of streaming data stored in an object block corresponding to the index block is obtained; determining an index block with intersection between the time index and the pre-updating time range as an index block to be processed, and reading each index data stored in the index block to be processed; and updating the index data with the data writing time within the pre-updating time range. The index blocks with intersection between the time index and the pre-updating time range are determined as the index blocks to be processed, and then the index data with data writing time within the pre-updating time range are updated, wherein the index data with the data writing time within the pre-updating time range comprise a plurality of index data, so that the synchronous updating of the plurality of index data is realized, the time required by updating the index data is reduced, and the updating efficiency of the index data in the streaming file system is improved.
As shown in fig. 3, another streaming file system index data updating method provided in the embodiment of the present invention specifically includes the following steps:
Step 201, obtaining an index time of an index block, where the index time is a time range determined based on a data writing time of streaming data stored in an object block corresponding to the index block.
Step 202, determine whether there is an intersection between the time index and the pre-update time range. If the intersection exists, go to step 203; if there is no intersection, go to step 206.
Step 203, if there is an intersection, the index block is determined as the index block to be processed.
Step 204, reading each index data stored in the index block to be processed, where the index data includes data write time of streaming data stored in the object block corresponding to the index block to be processed.
step 205, updating the index data with the data writing time within the pre-updating time range.
Step 201, step 202, step 203, step 204 and step 205 correspond to the specific contents in step 101, step 102, step 103, step 104 and step 105, respectively, and are not described herein again.
step 206, if there is no intersection, ending the intersection judgment process between the pre-update time range and the time index, and entering the intersection judgment process between the next pre-update time range and the time index.
when there is no intersection between the time index and the pre-update time range, it indicates that the index data that needs to be updated in the preset update time range is not stored in the streaming file system, and at this time, the intersection determination process between the pre-update time range and the time index may be ended, thereby ending the index data update process. When the user sets the next preset updating time range, the user can enter the intersection judgment flow aiming at the next preset updating time range and the time index, and then the next updating process of the index data is completed.
in the index data updating method in the streaming file system shown in fig. 2 provided in the embodiment of the present invention, the index time of an index block determined based on the data write time of streaming data stored in an object block corresponding to the index block is obtained; determining an index block with intersection between the time index and the pre-updating time range as an index block to be processed, and reading each index data stored in the index block to be processed; updating index data with the data writing time within the range of the pre-updating time; and when no intersection exists between the time index and the pre-updating time range, ending the intersection judgment process between the pre-updating time range and the time index, and entering the intersection judgment process between the next pre-updating time range and the time index. The index blocks with intersection between the time index and the pre-updating time range are determined as the index blocks to be processed, and then the index data with data writing time within the pre-updating time range are updated, wherein the index data with the data writing time within the pre-updating time range comprise a plurality of index data, so that the synchronous updating of the plurality of index data is realized, the time required by updating the index data is reduced, and the updating efficiency of the index data in the streaming file system is improved.
based on the same inventive concept, the above embodiment of the present invention provides an index data updating method in a streaming file system, and accordingly, an embodiment of the present invention provides an index data updating apparatus in a streaming file system, a schematic structural diagram of which is shown in fig. 4, and the method includes:
an index time obtaining module 301, configured to obtain index time of an index block, where the index time is a time range determined based on data writing time of streaming data stored in an object block corresponding to the index block;
a first determining module 302, configured to determine whether an intersection exists between the time index and the pre-update time range;
The to-be-processed index block determining module 303 determines the index block as the to-be-processed index block when an intersection exists between the time index and the pre-update time range;
A data reading module 304, configured to read each index data stored in the index block to be processed, where the index data includes data writing time of streaming data stored in an object block corresponding to the index block to be processed;
And a data updating module 305 for updating the index data with the data writing time within the pre-updating time range.
further, the pre-update time range is: and the time range is less than or equal to the acquired pre-updating time point.
Further, as shown in fig. 5, an embodiment of the present invention further provides an apparatus for updating index data in a streaming file system, where the apparatus may include:
An index time obtaining module 501, configured to obtain index time of an index block, where the index time is a time range determined based on data writing time of streaming data stored in an object block corresponding to the index block;
A first determining module 502, configured to determine whether an intersection exists between the time index and the pre-update time range;
The to-be-processed index block determining module 503 determines the index block as the to-be-processed index block when an intersection exists between the time index and the pre-update time range;
a data reading module 504, configured to read each index data stored in the index block to be processed, where the index data includes data writing time of streaming data stored in an object block corresponding to the index block to be processed;
a data updating module 505, configured to update index data with data writing time within a pre-updating time range;
A second determining module 506, configured to determine whether the time index of the index block to be processed is completely included in the pre-update time range.
The data updating module 505 is specifically configured to update all index data in the index block to be processed if the time index of the index block to be processed is completely included in the pre-update time range.
further, the data updating module 505 is further configured to update the index data in the index block to be processed, where the data writing time in the index block to be processed is within the overlapping time range of the time index of the index block to be processed and the pre-update time range, if the time index of the index block to be processed is not completely included in the pre-update time range.
further, the index time obtaining module 501 is specifically configured to: acquiring the earliest write-in time and the latest write-in time in data write-in time in index data of an index block; a time range greater than or equal to the earliest write time and less than or equal to the latest write time is determined as an index time of the index block.
In the index data updating apparatus in the streaming file system provided in the embodiment of the present invention, the index time obtaining module 501 obtains the index time of the index block; the first determining module 502 determines whether an intersection exists between the time index and the pre-update time range; when there is an intersection between the time index and the pre-update time range, the index block to be processed determining module 503 determines the index block as an index block to be processed; then, the data reading module 504 reads each index data stored in the index block to be processed; the data update module 505 updates the index data with a data write time within a pre-update time range. The method comprises the steps of determining an index block to be processed by judging whether intersection exists between a time index and a pre-updating time range, and updating index data with data writing time within the pre-updating time range, wherein the index data with the data writing time within the pre-updating time range comprise a plurality of index data, namely, the plurality of index data are synchronously updated, the time required by updating the index data is reduced, and the updating efficiency of the index data in a streaming file system is improved.
based on the same inventive concept, the method for updating index data in a streaming file system according to the above embodiment of the present invention correspondingly provides an electronic device, as shown in fig. 6, including a processor 601 and a memory 602, wherein,
a memory 602 for storing a computer program;
the processor 601 is configured to implement the index data updating method in the streaming file system according to the embodiment of the present invention when executing the program stored in the memory 602.
For example, the following steps may be included:
Acquiring index time of an index block, wherein the index time is a time range determined based on data writing time of streaming data stored in an object block corresponding to the index block;
judging whether an intersection exists between the time index and the pre-updating time range;
if the intersection exists, determining the index block as an index block to be processed;
Reading each index data stored in the index block to be processed, wherein the index data comprises data writing time of streaming data stored in an object block corresponding to the index block to be processed;
And updating the index data with the data writing time within the pre-updating time range.
Further, other processing flows in the index data updating method in the streaming file system provided by the embodiment of the present invention may also be included, and are not described in detail herein. The memory may include a Random Access Memory (RAM), or may include a Non-volatile memory (NVM), such as at least one disk memory. Further, the memory may be at least one memory device located remotely from the processor.
the Processor may be a general-purpose Processor, including a Central Processing Unit (CPU), a Network Processor (NP), and the like; the Integrated Circuit can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA), or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components.
the processor 601 and the memory 602 may be connected by a communication bus such as an address bus, a data bus, and a control bus, where the communication bus may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The electronic device may communicate with other external devices through the communication interface.
of course, the processor 601 and the memory 602, and the electronic device and other external devices may also communicate through a wireless connection of the wireless module.
In the electronic device provided by the embodiment of the invention, the adopted method is as follows: acquiring index time of an index block determined based on data writing time of streaming data stored in an object block corresponding to the index block; determining an index block with intersection between the time index and the pre-updating time range as an index block to be processed, and reading each index data stored in the index block to be processed; and updating the index data with the data writing time within the pre-updating time range. The index blocks with intersection between the time index and the pre-updating time range are determined as the index blocks to be processed, and then the index data with data writing time within the pre-updating time range are updated, wherein the index data with the data writing time within the pre-updating time range comprise a plurality of index data, so that the synchronous updating of the plurality of index data is realized, the time required by updating the index data is reduced, and the updating efficiency of the index data in the streaming file system is improved.
in yet another embodiment of the present invention, a computer-readable storage medium is further provided, which stores instructions that, when executed on a computer, cause the computer to execute the index data updating method in the streaming file system according to any one of the above embodiments.
In the computer-readable storage medium provided in the embodiment of the present invention, the method adopted is: acquiring index time of an index block determined based on data writing time of streaming data stored in an object block corresponding to the index block; determining an index block with intersection between the time index and the pre-updating time range as an index block to be processed, and reading each index data stored in the index block to be processed; and updating the index data with the data writing time within the pre-updating time range. The index blocks with intersection between the time index and the pre-updating time range are determined as the index blocks to be processed, and then the index data with data writing time within the pre-updating time range are updated, wherein the index data with the data writing time within the pre-updating time range comprise a plurality of index data, so that the synchronous updating of the plurality of index data is realized, the time required by updating the index data is reduced, and the updating efficiency of the index data in the streaming file system is improved.
In yet another embodiment of the present invention, there is also provided a computer program product containing instructions which, when run on a computer, cause the computer to perform the index data updating method in the streaming file system according to any one of the above embodiments.
In the computer program product including instructions provided by the embodiment of the present invention, the method adopted is: acquiring index time of an index block determined based on data writing time of streaming data stored in an object block corresponding to the index block; determining an index block with intersection between the time index and the pre-updating time range as an index block to be processed, and reading each index data stored in the index block to be processed; and updating the index data with the data writing time within the pre-updating time range. The index blocks with intersection between the time index and the pre-updating time range are determined as the index blocks to be processed, and then the index data with data writing time within the pre-updating time range are updated, wherein the index data with the data writing time within the pre-updating time range comprise a plurality of index data, so that the synchronous updating of the plurality of index data is realized, the time required by updating the index data is reduced, and the updating efficiency of the index data in the streaming file system is improved.
In the above embodiments, the implementation may be wholly or partially realized by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. The procedures or functions described above in accordance with the embodiments of the invention may be generated, in whole or in part, when the computer program instructions described above are loaded and executed on a computer. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored on a computer readable storage medium or transmitted from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wire (e.g., coaxial cable, fiber optic cable, Digital Subscriber Line (DSL)), or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device, such as a server, a data center, etc., that includes one or more of the available media. The usable medium may be a magnetic medium (e.g., a floppy Disk, a hard Disk, a magnetic tape), an optical medium (e.g., a Digital Video Disc (DVD)), a semiconductor medium (e.g., a Solid State Disk (SSD)), or the like.
It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. The term "comprising", without further limitation, means that the element so defined is not excluded from the group consisting of additional identical elements in the process, method, article, or apparatus that comprises the element.
all the embodiments in the present specification are described in a related manner, and the same and similar parts among the embodiments may be referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, as for the device, the electronic apparatus, and the storage medium embodiments, since they are substantially similar to the method embodiments, the description is relatively simple, and for the relevant points, reference may be made to part of the description of the method embodiments.
the above description is only for the preferred embodiment of the present invention, and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention shall fall within the protection scope of the present invention.
Claims (10)
1. A method for index data update in a streaming file system, the method comprising:
Acquiring index time of an index block, wherein the index time is a time range determined based on data writing time of streaming data stored in an object block corresponding to the index block;
Judging whether an intersection exists between the time index and a pre-updating time range;
if the intersection exists, determining the index block as an index block to be processed;
Reading each index data stored in the index block to be processed, wherein the index data comprises data writing time of streaming data stored in an object block corresponding to the index block to be processed;
and updating the index data with the data writing time within the pre-updating time range.
2. The method of claim 1, wherein the pre-update time range is: and the time range is less than or equal to the acquired pre-updating time point.
3. the method of claim 1, wherein before the update data writes index data whose time is within the pre-update time range, the method further comprises:
judging whether the time index of the index block to be processed is completely contained in the pre-updating time range;
the updating the index data of which the data writing time is within the pre-updating time range comprises:
and if the time index of the index block to be processed is completely contained in the pre-updating time range, updating all index data in the index block to be processed.
4. The method according to claim 3, wherein the updating the index data of which the data writing time is within the pre-update time range comprises:
and if the time index of the index block to be processed is not completely contained in the pre-updating time range, updating the index data of which the data writing time in the index block to be processed is within the overlapping time range of the time index of the index block to be processed and the pre-updating time range.
5. The method according to any one of claims 1-4, wherein the obtaining the index time of the index block comprises:
acquiring the earliest write-in time and the latest write-in time in data write-in time in index data of the index block;
And determining a time range which is greater than or equal to the earliest writing time and less than or equal to the latest writing time as the index time of the index block.
6. An apparatus for updating index data in a streaming file system, comprising:
The index time acquisition module is used for acquiring index time of an index block, wherein the index time is a time range determined based on data writing time of streaming data stored in an object block corresponding to the index block;
The first judgment module is used for judging whether an intersection exists between the time index and the pre-updating time range;
The index block to be processed determining module is used for determining the index block to be processed as an index block to be processed when intersection exists between the time index and a pre-updating time range;
a data reading module, configured to read each index data stored in the to-be-processed index block, where the index data includes data write time of streaming data stored in an object block corresponding to the to-be-processed index block;
And the data updating module is used for updating the index data of which the data writing time is within the pre-updating time range.
7. the apparatus of claim 6, wherein the pre-update time range is: and the time range is less than or equal to the acquired pre-updating time point.
8. The apparatus of claim 6, further comprising:
the second judgment module is used for judging whether the time index of the index block to be processed is completely contained in the pre-updating time range;
The data updating module is specifically configured to update all index data in the index block to be processed if the time index of the index block to be processed is completely included in the pre-update time range.
9. the apparatus of claim 8, wherein the data updating module is further configured to update index data in the index block to be processed, in which a data writing time of the index block to be processed is within an overlapping time range of the time index of the index block to be processed and the pre-update time range, if the time index of the index block to be processed is not completely included in the pre-update time range.
10. The apparatus according to any one of claims 6 to 8, wherein the index time obtaining module is specifically configured to:
acquiring the earliest write-in time and the latest write-in time in data write-in time in index data of the index block;
And determining a time range which is greater than or equal to the earliest writing time and less than or equal to the latest writing time as the index time of the index block.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810470680.1A CN110569217A (en) | 2018-05-16 | 2018-05-16 | index data updating method and device in streaming file system |
| PCT/CN2019/087002 WO2019219021A1 (en) | 2018-05-16 | 2019-05-15 | Index data updating method and device in stream-oriented file system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810470680.1A CN110569217A (en) | 2018-05-16 | 2018-05-16 | index data updating method and device in streaming file system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110569217A true CN110569217A (en) | 2019-12-13 |
Family
ID=68539541
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810470680.1A Pending CN110569217A (en) | 2018-05-16 | 2018-05-16 | index data updating method and device in streaming file system |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN110569217A (en) |
| WO (1) | WO2019219021A1 (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20030101183A1 (en) * | 2001-11-26 | 2003-05-29 | Navin Kabra | Information retrieval index allowing updating while in use |
| CN101989301A (en) * | 2010-10-22 | 2011-03-23 | 复旦大学 | Index maintenance method for supporting multiple data sources |
| US20130232175A1 (en) * | 2007-10-05 | 2013-09-05 | Masaki Kan | Information retrieval system, registration apparatus for indexes for information retrieval, information retrieval method and program |
| CN104794123A (en) * | 2014-01-20 | 2015-07-22 | 阿里巴巴集团控股有限公司 | Method and device for establishing NoSQL database index for semi-structured data |
| CN107315825A (en) * | 2017-07-05 | 2017-11-03 | 北京奇艺世纪科技有限公司 | A kind of index upgrade system, method and device |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103177116B (en) * | 2013-04-08 | 2016-05-04 | 国电南瑞科技股份有限公司 | A kind of distributed information log based on two-stage index is processed and querying method |
-
2018
- 2018-05-16 CN CN201810470680.1A patent/CN110569217A/en active Pending
-
2019
- 2019-05-15 WO PCT/CN2019/087002 patent/WO2019219021A1/en active Application Filing
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20030101183A1 (en) * | 2001-11-26 | 2003-05-29 | Navin Kabra | Information retrieval index allowing updating while in use |
| US20130232175A1 (en) * | 2007-10-05 | 2013-09-05 | Masaki Kan | Information retrieval system, registration apparatus for indexes for information retrieval, information retrieval method and program |
| CN101989301A (en) * | 2010-10-22 | 2011-03-23 | 复旦大学 | Index maintenance method for supporting multiple data sources |
| CN104794123A (en) * | 2014-01-20 | 2015-07-22 | 阿里巴巴集团控股有限公司 | Method and device for establishing NoSQL database index for semi-structured data |
| CN107315825A (en) * | 2017-07-05 | 2017-11-03 | 北京奇艺世纪科技有限公司 | A kind of index upgrade system, method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2019219021A1 (en) | 2019-11-21 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11853549B2 (en) | Index storage in shingled magnetic recording (SMR) storage system with non-shingled region | |
| CN109766362B (en) | Data processing method and device | |
| CN110018989B (en) | Snapshot comparison method and device | |
| CN107766374B (en) | Optimization method and system for storage and reading of massive small files | |
| CN111159436A (en) | Method and device for recommending multimedia content and computing equipment | |
| US20070156778A1 (en) | File indexer | |
| CN110147203B (en) | File management method and device, electronic equipment and storage medium | |
| CN109800181B (en) | Disk-based data writing method, data writing device and terminal equipment | |
| CN112597153A (en) | Data storage method and device based on block chain and storage medium | |
| CN107577775B (en) | Data reading method and device, electronic equipment and readable storage medium | |
| CN108255989B (en) | Picture storage method and device, terminal equipment and computer storage medium | |
| CN112463058B (en) | Fragmented data sorting method and device and storage node | |
| CN114020193A (en) | Cross-page checking determination method and device, electronic equipment and storage medium | |
| CN113407376A (en) | Data recovery method and device and electronic equipment | |
| WO2019228009A1 (en) | Lsm tree optimization method and device and computer equipment | |
| CN110569217A (en) | index data updating method and device in streaming file system | |
| CN110851411B (en) | DNS dynamic change system and method based on file synchronization | |
| CN114356856A (en) | Data deleting method, device, equipment and storage medium | |
| CN110490581B (en) | Distributed system critical data resource updating method and system | |
| CN111506250B (en) | Data deleting method and device, electronic equipment and storage medium | |
| CN115933995B (en) | Method and device for writing data into solid state disk, electronic equipment and readable medium | |
| CN113468105A (en) | Data structure of data snapshot, related data processing method, device and system | |
| CN110888643A (en) | Page processing method and device | |
| CN112543213B (en) | Data processing method and device | |
| CN111078714B (en) | Data processing method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |