CN112883036A - Index creation method, device, storage server and storage medium - Google Patents
Index creation method, device, storage server and storage medium Download PDFInfo
- Publication number
- CN112883036A CN112883036A CN202110250241.1A CN202110250241A CN112883036A CN 112883036 A CN112883036 A CN 112883036A CN 202110250241 A CN202110250241 A CN 202110250241A CN 112883036 A CN112883036 A CN 112883036A
- Authority
- CN
- China
- Prior art keywords
- data
- index
- stored
- time
- storage server
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 62
- 238000004891 communication Methods 0.000 description 12
- 238000012545 processing Methods 0.000 description 11
- 238000010586 diagram Methods 0.000 description 4
- 230000002093 peripheral effect Effects 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 3
- 238000013473 artificial intelligence Methods 0.000 description 2
- 238000013500 data storage Methods 0.000 description 2
- 238000013459 approach Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004590 computer program Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000001788 irregular Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 230000005055 memory storage Effects 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000009877 rendering Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2453—Query optimisation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Software Systems (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses an index creating method, which is applied to a storage server; the method comprises the following steps: when receiving data to be stored, storing the data to be stored in the storage server; when the data to be stored meet a preset condition, determining a selected index corresponding to the data to be stored in the available indexes of the storage server; and when the data volume of the selected stored data corresponding to the selected index reaches a preset threshold value, creating a new available index based on the to-be-written index information and the time information corresponding to the to-be-stored data. The invention also discloses an index creating device, a storage server and a computer readable storage medium. By using the index creating method, the query efficiency is higher when the obtained index is used for data query.
Description
Technical Field
The present invention relates to the field of data storage technologies, and in particular, to an index creation method, an index creation apparatus, a storage server, and a computer-readable storage medium.
Background
In a distributed big data processing scenario, if a large amount of continuous data is written into a data storage server, the data is convenient for a user to store and query.
In order to ensure the query speed of data, when the document amount in the index reaches a certain amount, index division operation needs to be carried out on the document amount, in the related technology, an index creation method is disclosed, and when the stored data amount reaches a certain value, the stored data is used as a data group to generate a corresponding index; along with the lapse of time, the data stored are more and more, the index obtained is more and more, the data bulk corresponding to each index is the same; when the data needs to be queried, the corresponding data is queried through the index.
However, when the existing index creating method is adopted and the obtained index is used for data query, the data query efficiency is low.
Disclosure of Invention
The invention mainly aims to provide an index creating method, an index creating device, a storage server and a computer readable storage medium, and aims to solve the technical problem that when the existing index creating method is adopted and the obtained index is used for data query in the prior art, the data query efficiency is low.
In order to achieve the above object, the present invention provides an index creating method, which is applied to a storage server; the method comprises the following steps:
when receiving data to be stored, storing the data to be stored in the storage server;
when the data to be stored meet a preset condition, determining a selected index corresponding to the data to be stored in the available indexes of the storage server;
and when the data volume of the selected stored data corresponding to the selected index reaches a preset threshold value, creating a new available index based on the to-be-written index information and the time information corresponding to the to-be-stored data.
Optionally, the step of creating a new available index based on the to-be-written index information and the time information corresponding to the to-be-stored data includes:
and creating the new available index based on the information of the index to be written, the current storage time of the data to be stored and preset time.
Optionally, the method further includes:
and when the data volume of the selected stored data corresponding to the selected index does not exceed a preset threshold value, updating the selected index by using the information of the index to be written to obtain the updated selected index.
Optionally, the selected index includes a time interval composed of an initial storage time of the selected stored data and the preset time; before the step of creating the new available index based on the information of the index to be written, the current storage time of the data to be stored, and the preset time, the method further includes:
and updating the preset time in the time interval by using the current storage time to obtain the updated selected index.
Optionally, the data to be stored has a sequence identifier; before the step of creating the new available index based on the information of the index to be written, the current storage time of the data to be stored, and the preset time, the method further includes:
replacing the sequence identification by using a preset identification to obtain new data to be stored;
the step of creating the new available index based on the information of the index to be written, the current storage time of the data to be stored and a preset time comprises:
and creating the new available index by using the index information to be written, the current storage time and the preset time based on the preset identification of the new data to be stored.
Optionally, the data to be stored is RDD data after spark streaming.
Optionally, the storage server is an Elasticsearch.
In addition, in order to achieve the above object, the present invention further provides an index creating apparatus, which is applied to a storage server; the device comprises:
the receiving module is used for storing the data to be stored in the storage server when the data to be stored is received;
the determining module is used for determining a selected index corresponding to the data to be stored in the available indexes of the storage server when the data to be stored meets a preset condition;
and the updating module is used for creating a new available index based on the information of the index to be written and the time information corresponding to the data to be stored when the data volume of the selected stored data corresponding to the selected index reaches a preset threshold value.
In addition, to achieve the above object, the present invention further provides a storage server, including: a memory, a processor and an index creation program stored on the memory and running on the processor, the index creation program when executed by the processor implementing the steps of the index creation method as claimed in any one of the above.
Further, to achieve the above object, the present invention also proposes a computer-readable storage medium having stored thereon an index creation program which, when executed by a processor, implements the steps of the index creation method as described in any one of the above.
The technical scheme of the invention provides an index creating method which is applied to a storage server; the method comprises the following steps: when receiving data to be stored, storing the data to be stored in the storage server; determining a selected index corresponding to the data to be stored in the available indexes of the storage server; and when the data volume of the selected stored data corresponding to the selected index reaches a preset threshold value, creating a new available index based on the index name to be written and the time information corresponding to the data to be stored.
In the existing index creating method, indexes are divided according to the data volume of stored data, that is, when the data volume of stored data corresponding to one index reaches a threshold value, a new available index is generated, but when data query is performed by using the generated index, all indexes corresponding to required index information need to be scanned based on the required index information of required data to query the required data, so that the efficiency of data query is low; by using the acquisition method of the invention, when the data volume of the stored data corresponding to the available index reaches the preset threshold value, namely, a new available index is created based on the to-be-written index information and the time information corresponding to the to-be-stored data, the available index has time information.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the structures shown in the drawings without creative efforts.
FIG. 1 is a schematic diagram of a storage server architecture of a hardware operating environment according to an embodiment of the present invention;
FIG. 2 is a flowchart illustrating a first embodiment of an index creating method according to the present invention;
FIG. 3 is a block diagram of a first embodiment of an index creating apparatus according to the present invention.
The implementation, functional features and advantages of the objects of the present invention will be further explained with reference to the accompanying drawings.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Referring to fig. 1, fig. 1 is a schematic structural diagram of a storage server in a hardware operating environment according to an embodiment of the present invention.
The storage server may be a User Equipment (UE) such as a Mobile phone, a smart phone, a laptop, a digital broadcast receiver, a Personal Digital Assistant (PDA), a tablet computer (PAD), a handheld device, a vehicle mounted device, a wearable device, a computing device or other processing device connected to a wireless modem, a Mobile Station (MS), etc. The storage server may be referred to as a user terminal, a portable terminal, a desktop terminal, etc.
In general, a storage server includes: at least one processor 301, a memory 302, and an index creation program stored on the memory and executable on the processor, the index creation program configured to implement the steps of the index creation method as previously described.
The processor 301 may include one or more processing cores, such as a 4-core processor, an 8-core processor, and so on. The processor 301 may be implemented in at least one hardware form of a DSP (Digital Signal Processing), an FPGA (Field-Programmable Gate Array), and a PLA (Programmable Logic Array). The processor 301 may also include a main processor and a coprocessor, where the main processor is a processor for processing data in an awake state, and is also called a Central Processing Unit (CPU); a coprocessor is a low power processor for processing data in a standby state. In some embodiments, the processor 301 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content required to be displayed on the display screen. The processor 301 may further include an AI (Artificial Intelligence) processor for processing operations related to the index creation method, so that the index creation method model may be trained autonomously for learning, improving efficiency and accuracy.
In some embodiments, the terminal may further include: a communication interface 303 and at least one peripheral device. The processor 301, the memory 302 and the communication interface 303 may be connected by a bus or signal lines. Various peripheral devices may be connected to communication interface 303 via a bus, signal line, or circuit board. Specifically, the peripheral device includes: at least one of radio frequency circuitry 304, a display screen 305, and a power source 306.
The communication interface 303 may be used to connect at least one peripheral device related to I/O (Input/Output) to the processor 301 and the memory 302. In some embodiments, processor 301, memory 302, and communication interface 303 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 301, the memory 302 and the communication interface 303 may be implemented on a single chip or circuit board, which is not limited in this embodiment.
The Radio Frequency circuit 304 is used for receiving and transmitting RF (Radio Frequency) signals, also called electromagnetic signals. The radio frequency circuitry 304 communicates with communication networks and other communication devices via electromagnetic signals. The rf circuit 304 converts an electrical signal into an electromagnetic signal to transmit, or converts a received electromagnetic signal into an electrical signal. Optionally, the radio frequency circuit 304 comprises: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a subscriber identity module card, and so forth. The radio frequency circuitry 304 may communicate with other terminals via at least one wireless communication protocol. The wireless communication protocols include, but are not limited to: metropolitan area networks, various generation mobile communication networks (2G, 3G, 4G, and 5G), Wireless local area networks, and/or WiFi (Wireless Fidelity) networks. In some embodiments, the rf circuit 304 may further include NFC (Near Field Communication) related circuits, which are not limited in this application.
The display screen 305 is used to display a UI (User Interface). The UI may include graphics, text, icons, video, and any combination thereof. When the display screen 305 is a touch display screen, the display screen 305 also has the ability to capture touch signals on or over the surface of the display screen 305. The touch signal may be input to the processor 301 as a control signal for processing. At this point, the display screen 305 may also be used to provide virtual buttons and/or a virtual keyboard, also referred to as soft buttons and/or a soft keyboard. In some embodiments, the display screen 305 may be one, the front panel of the electronic device; in other embodiments, the display screens 305 may be at least two, respectively disposed on different surfaces of the electronic device or in a folded design; in still other embodiments, the display screen 305 may be a flexible display screen disposed on a curved surface or a folded surface of the electronic device. Even further, the display screen 305 may be arranged in a non-rectangular irregular figure, i.e. a shaped screen. The Display screen 305 may be made of LCD (liquid crystal Display), OLED (Organic Light-Emitting Diode), and the like.
The power supply 306 is used to power various components in the electronic device. The power source 306 may be alternating current, direct current, disposable or rechargeable. When the power source 306 includes a rechargeable battery, the rechargeable battery may support wired or wireless charging. The rechargeable battery may also be used to support fast charge technology.
Those skilled in the art will appreciate that the architecture shown in FIG. 1 does not constitute a limitation of storage servers, and may include more or fewer components than those shown, or some components may be combined, or a different arrangement of components.
Furthermore, an embodiment of the present invention further provides a computer-readable storage medium, on which an index creating program is stored, and the index creating program, when executed by a processor, implements the steps of the index creating method as described above. Therefore, a detailed description thereof will be omitted. In addition, the beneficial effects of the same method are not described in detail. For technical details not disclosed in embodiments of the computer-readable storage medium referred to in the present application, reference is made to the description of embodiments of the method of the present application. It is determined that the program instructions may be deployed to be executed on one storage server, or on multiple storage servers located at one site, or on multiple storage servers distributed across multiple sites and interconnected by a communication network, as examples.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by a computer program, which can be stored in a computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. The computer-readable storage medium may be a magnetic disk, an optical disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), or the like.
Three index creation methods are disclosed in the prior art:
indexing by fixed time division: such as generating an index a day. The disadvantage is that if the amount of data per day is too small, the amount of stored data of a single slice is too small and wastes resources, while if the amount of data is too large, the amount of data of a single slice may be too large and the query efficiency may be too low, and indexing according to a fixed time may cause the index per day to be too large or the index per day to be too small. Thus, there is no way to ensure consistency in query efficiency.
Indexing by document amount: although the size of each index can be basically guaranteed to be consistent, the method is not advantageous when data in a certain time range is queried, because the time range contained in each index is not clear due to the lack of strict time division, and therefore, if conditional query is performed according to time, all indexes are scanned, and query efficiency is affected;
dividing indexes according to the storage size of the indexes: this approach is much more of a feature of indexing by document size, and has the disadvantage of scanning the entire index if queried over time.
Based on the above hardware structure, an embodiment of the index creation method of the present invention is provided.
Referring to fig. 2, fig. 2 is a schematic flowchart of a first embodiment of an index creating method according to the present invention, where the method is applied to a storage server; the method comprises the following steps:
step S11: and when receiving the data to be stored, storing the data to be stored in the storage server.
The main execution body of the present invention is a storage server, the storage server is installed with an index creation program, and the storage server implements the index creation method of the present invention when executing the index creation program of the present invention. The storage server may be any type of storage server and the present invention is not limited thereto.
In specific application, the storage server may be an Elasticsearch, and the data to be stored is RDD data after spark streaming. The data to be stored may be any type (without limitation to the kind of data); generally, the RDD data after spark streaming is sent in batches, each batch of RDD data has an ID (sequence number), and the IDs of the RDD data generally increase sequentially; for example, when one type of data (the one type of data may be all data corresponding to one commodity, or all data corresponding to a certain model of a certain commodity, or data corresponding to a certain object in different situations, etc.) is sent, the data is processed by spark streaming and then becomes multiple batches of RDD data, the ID of the RDD data of the first batch is 0, the ID of the RDD data of the second batch is 1, and so on, and the whole data batches have different sequence numbers (sequence identifiers) respectively.
Generally, when data to be stored is received, in addition to storing the data, the data to be stored needs to be associated with an index so as to facilitate query and the like of the data to be stored.
Step S12: and when the data to be stored meet a preset condition, determining a selected index corresponding to the data to be stored in the available indexes of the storage server.
It should be noted that the data to be stored satisfies the preset condition, that is, the data to be stored is the data of the type and is received by the storage server for the first time, for example, the data of one type is data of an article a, and the data to be stored is the data of the article a and is received by the storage server for the first time, then the data to be stored satisfies the preset condition.
It can be understood that the storage server stores a large amount of stored data, the stored data may also be associated with a plurality of available indexes, one available index corresponds to a part of the stored data, and the available index includes an available index corresponding to the data amount reaching the preset threshold (the index may not be associated with the new file to be stored any more), and an index corresponding to the data amount not reaching the preset threshold (the index may be associated with the new file to be stored any more). Generally, one type of data corresponds to a plurality of available indexes, and it is only necessary to determine a plurality of available indexes corresponding to stored data of the same type as the data to be stored in the storage server, and determine the nearest available index from the plurality of available indexes, that is, the selected index.
For example, the data to be stored is data of a product B, the data of the product B does not satisfy a preset condition (indicating that the storage server has stored data of a part of the product B), the available index corresponding to the product B includes 7 available indexes, where the 7 th available index is the nearest available index, and then the 7 available indexes corresponding to the stored data corresponding to the product B are determined at the storage server only according to the data of the product B, and the 7 th available index that is the nearest among the 7 available indexes is determined as the selected index.
Step S13: and when the data volume of the selected stored data corresponding to the selected index reaches a preset threshold value, creating a new available index based on the to-be-written index information and the time information corresponding to the to-be-stored data.
It should be noted that the preset threshold may be a threshold set by a user according to a requirement, and the present invention is not limited thereto. The step of creating a new available index based on the to-be-written index information and the time information corresponding to the to-be-stored data includes: and creating the new available index based on the information of the index to be written, the current storage time of the data to be stored and preset time.
Specifically, the data to be stored has a sequence identifier; before step S13, the method further includes: replacing the sequence identification by using a preset identification to obtain new data to be stored; correspondingly, the step of creating the new available index based on the information of the index to be written, the current storage time of the data to be stored, and the preset time includes: and creating the new available index by using the index information to be written, the current storage time and the preset time based on the preset identification of the new data to be stored.
When the data volume of the selected stored data corresponding to the selected index reaches a preset threshold value, the data volume of the stored data corresponding to the selected index is large, and at the moment, a new available index corresponding to the data to be stored needs to be generated; the preset identifier may be a sequence identifier corresponding to when the first batch of data corresponding to the data is stored, for example, when the data is RDD data, the preset identifier is 0. The generation of the new available index is performed with reference to the following steps:
the data to be stored is RDD data after spark streaming processing, and generally, the data to be stored includes two types, where the data to be stored is first batch data (data of the type is received for the first time), that is, the ID of the RDD data is 0, and the data to be stored is not first batch data (data of the type is not received for the first time), and the ID of the RDD data is not 0.
Here, the data to be stored whose RDD is not 0 is explained as an example. When the data volume of the selected stored data exceeds a preset threshold value, a new available index needs to be generated, at this time, it is determined that the sequential identification of the data to be stored (i.e. the ID of the RDD data) is not 0, and the sequential identification of the data to be stored is replaced by the preset identification (the ID of the RDD is replaced by 0), to obtain new data to be stored, the storage server determines that a new available index needs to be generated based on the sequential identification of the new data to be stored (the ID of the RDD is 0), the new available index is created based on the index information to be written (index information of the data to be stored, possibly including the name and type of the data, etc.), the current storage time of the data to be stored, and the preset time, in the new available index, the available index includes an index name, a current storage time (i.e. the current storage time of the data to be stored, which is also the creation time of the new available index), and a preset time; the preset time is usually the maximum time, and the maximum time is the time when the data amount of the stored data corresponding to the available index reaches the preset threshold. Generally, when an available index is created, the available index may continue to be associated with new data to be stored, the time when the amount of data reaches a preset threshold is unknown, and the preset time may be a time after the limit.
For example, the available index may include an index name and an index alias, the index name generally includes to-be-written index information of the to-be-stored data, and the index alias includes to-be-written index information of the to-be-stored data, a current storage time, and a preset time, for example, the index name of the available index is a (a may be index information of one type of data), and the index alias may be expressed as: a-1608280028000 and 99999999999, wherein the former 13 is the current storage time of the data to be stored, and the latter 13 digits is the preset time.
When the sequence identifier of the to-be-stored data (i.e., the ID of the RDD data) is 0 and the to-be-stored data satisfies the preset condition, the step of creating the new available index based on the to-be-written index information corresponding to the to-be-stored data, the current storage time of the to-be-stored data, and the preset time is performed, and the specific steps refer to the above description, and are not described again here.
Further, the available indexes in the storage server of the present invention may be obtained by the above method, and for any available index, when the data amount of the corresponding stored data reaches the preset threshold, the index name needs to be updated, that is, the preset time in the index is modified.
The selected index comprises a time interval consisting of the initial storage time of the selected stored data and the preset time; the step of modifying the preset time comprises the following steps: and updating the preset time in the time interval by using the current storage time to obtain the updated selected index.
When a new available index is generated, the selected index does not need to be continuously associated with new data, and at this time, the time interval (the time interval formed by the initial storage time and the preset time) of the selected index needs to be updated, because the selected index already corresponds to a sufficient amount of data at this time and cannot be continuously associated with new data, the maximum time is the current storage time of the data to be stored, the new time interval is the interval formed by the initial storage time and the current storage time of the data to be stored, and the initial storage time is the creation time of the selected index (the storage time of the first batch of data corresponding to the selected index).
Specifically, in the above embodiment, two times in the index alias are two end points of the time interval, the next string of numbers (preset time) is usually replaced with the current storage time of the data to be stored, and when the data to be stored is replaced, the number of bits of the data is usually the same, for example, the preset time is 13 bits of time data, that is, the current storage time of the data to be stored is also 13 bits.
Further, the method further comprises: and when the data volume of the selected stored data corresponding to the selected index does not exceed a preset threshold value, updating the selected index by using the information of the index to be written to obtain the updated selected index.
It should be noted that the available indexes in the storage server of the present invention may be obtained by using the above method, and when the data amount of the selected stored data does not exceed the preset threshold, the selected index may continue to be associated with the data to be stored, that is, the step of updating the selected index by using the name of the index to be written is performed to obtain an updated selected index, and the updated selected index is associated with the data to be stored.
It can be understood that, since the selected stored data corresponding to the selected index does not reach the preset threshold, the preset time in the alias of the index does not need to be updated, and the corresponding time interval is still the time interval formed by the initial storage time and the preset time.
The technical scheme of the invention provides an index creating method which is applied to a storage server; the method comprises the following steps: when receiving data to be stored, storing the data to be stored in the storage server; determining a selected index corresponding to the data to be stored in the available indexes of the storage server; and when the data volume of the selected stored data corresponding to the selected index reaches a preset threshold value, creating a new available index based on the index name to be written and the time information corresponding to the data to be stored.
In the existing index creating method, indexes are divided according to the data volume of stored data, that is, when the data volume of stored data corresponding to one index reaches a threshold value, a new available index is generated, but when data query is performed by using the generated index, all indexes corresponding to required index information need to be scanned based on the required index information of required data to query the required data, so that the efficiency of data query is low; by using the acquisition method of the invention, when the data volume of the stored data corresponding to the available index reaches the preset threshold value, namely, a new available index is created based on the to-be-written index information and the time information corresponding to the to-be-stored data, the available index has time information.
Referring to fig. 3, fig. 3 is a block diagram illustrating a first embodiment of an index creating apparatus according to the present invention, the apparatus being applied to a storage server; the device comprises:
the receiving module 10 is configured to store the data to be stored in the storage server when receiving the data to be stored;
a determining module 20, configured to determine, when the data to be stored meets a preset condition, a selected index corresponding to the data to be stored in the available indexes of the storage server;
and the updating module 30 is configured to create a new available index based on the to-be-written index information and the time information corresponding to the to-be-stored data when the data amount of the selected stored data corresponding to the selected index reaches a preset threshold.
The above description is only an alternative embodiment of the present invention, and not intended to limit the scope of the present invention, and all modifications and equivalents of the present invention, which are made by the contents of the present specification and the accompanying drawings, or directly/indirectly applied to other related technical fields, are included in the scope of the present invention.
Claims (10)
1. An index creation method is applied to a storage server; the method comprises the following steps:
when receiving data to be stored, storing the data to be stored in the storage server;
when the data to be stored meet a preset condition, determining a selected index corresponding to the data to be stored in the available indexes of the storage server;
and when the data volume of the selected stored data corresponding to the selected index reaches a preset threshold value, creating a new available index based on the to-be-written index information and the time information corresponding to the to-be-stored data.
2. The method according to claim 1, wherein the step of creating a new available index based on the to-be-written index information and the time information corresponding to the data to be stored comprises:
and creating the new available index based on the information of the index to be written, the current storage time of the data to be stored and preset time.
3. The method of claim 2, wherein the method further comprises:
and when the data volume of the selected stored data corresponding to the selected index does not exceed a preset threshold value, updating the selected index by using the information of the index to be written to obtain the updated selected index.
4. The method of claim 3, wherein the selected index comprises a time interval consisting of an initial storage time of the selected stored data and the preset time; before the step of creating the new available index based on the information of the index to be written, the current storage time of the data to be stored, and the preset time, the method further includes:
and updating the preset time in the time interval by using the current storage time to obtain the updated selected index.
5. The method of claim 4, wherein the data to be stored has an order identification; before the step of creating the new available index based on the information of the index to be written, the current storage time of the data to be stored, and the preset time, the method further includes:
replacing the sequence identification by using a preset identification to obtain new data to be stored;
the step of creating the new available index based on the information of the index to be written, the current storage time of the data to be stored and a preset time comprises:
and creating the new available index by using the index information to be written, the current storage time and the preset time based on the preset identification of the new data to be stored.
6. The method of any one of claims 1 to 5, wherein the data to be stored is spark streaming processed RDD data.
7. The method of claim 6, wherein the storage server is an Elasticsearch.
8. An index creation apparatus, applied to a storage server; the device comprises:
the receiving module is used for storing the data to be stored in the storage server when the data to be stored is received;
the determining module is used for determining a selected index corresponding to the data to be stored in the available indexes of the storage server when the data to be stored meets a preset condition;
and the updating module is used for creating a new available index based on the information of the index to be written and the time information corresponding to the data to be stored when the data volume of the selected stored data corresponding to the selected index reaches a preset threshold value.
9. A storage server, the storage server comprising: memory, a processor and an index creation program stored on the memory and running on the processor, the index creation program when executed by the processor implementing the steps of the index creation method of any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that an index creation program is stored thereon, which when executed by a processor implements the steps of the index creation method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110250241.1A CN112883036B (en) | 2021-03-05 | 2021-03-05 | Index creation method, device, storage server and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110250241.1A CN112883036B (en) | 2021-03-05 | 2021-03-05 | Index creation method, device, storage server and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112883036A true CN112883036A (en) | 2021-06-01 |
| CN112883036B CN112883036B (en) | 2024-05-28 |
Family
ID=76055698
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110250241.1A Active CN112883036B (en) | 2021-03-05 | 2021-03-05 | Index creation method, device, storage server and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112883036B (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113688142A (en) * | 2021-10-25 | 2021-11-23 | 北京金山云网络技术有限公司 | Index management method, device, storage medium and electronic equipment |
| CN114139021A (en) * | 2022-01-27 | 2022-03-04 | 云丁网络技术(北京)有限公司 | Index information management method and system |
| CN114661722A (en) * | 2022-03-23 | 2022-06-24 | 天津同阳科技发展有限公司 | Data storage method, data indexing method and device |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102024057A (en) * | 2010-12-24 | 2011-04-20 | 中兴通讯股份有限公司 | Method and device for building index of mass data record |
| CN104008111A (en) * | 2013-02-27 | 2014-08-27 | 深圳市腾讯计算机系统有限公司 | Data storage management method and device |
| CN106528717A (en) * | 2016-10-26 | 2017-03-22 | 中国电子产品可靠性与环境试验研究所 | Data processing method and system |
| CN106970935A (en) * | 2017-01-20 | 2017-07-21 | 朗坤智慧科技股份有限公司 | Real-time data memory structure, method for writing data and method for reading data |
| CN107391770A (en) * | 2017-09-13 | 2017-11-24 | 北京锐安科技有限公司 | A kind of method, apparatus of processing data, equipment and storage medium |
| US20190294602A1 (en) * | 2017-01-09 | 2019-09-26 | Tencent Technology (Shenzhen) Company Limited | Data scrubbing method and apparatus, and computer readable storage medium |
| CN110347674A (en) * | 2019-06-03 | 2019-10-18 | 阿里巴巴集团控股有限公司 | A kind of index creation method, device and equipment based on service identification |
| CN110427364A (en) * | 2019-06-21 | 2019-11-08 | 北京奇艺世纪科技有限公司 | A kind of data processing method, device, electronic equipment and storage medium |
| CN112241410A (en) * | 2020-10-22 | 2021-01-19 | 曙光信息产业(北京)有限公司 | Data storage method, data index construction method and device and computer equipment |
| CN112434039A (en) * | 2020-11-30 | 2021-03-02 | 浙江大华技术股份有限公司 | Data storage method, device, storage medium and electronic device |
-
2021
- 2021-03-05 CN CN202110250241.1A patent/CN112883036B/en active Active
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102024057A (en) * | 2010-12-24 | 2011-04-20 | 中兴通讯股份有限公司 | Method and device for building index of mass data record |
| CN104008111A (en) * | 2013-02-27 | 2014-08-27 | 深圳市腾讯计算机系统有限公司 | Data storage management method and device |
| CN106528717A (en) * | 2016-10-26 | 2017-03-22 | 中国电子产品可靠性与环境试验研究所 | Data processing method and system |
| US20190294602A1 (en) * | 2017-01-09 | 2019-09-26 | Tencent Technology (Shenzhen) Company Limited | Data scrubbing method and apparatus, and computer readable storage medium |
| CN106970935A (en) * | 2017-01-20 | 2017-07-21 | 朗坤智慧科技股份有限公司 | Real-time data memory structure, method for writing data and method for reading data |
| CN107391770A (en) * | 2017-09-13 | 2017-11-24 | 北京锐安科技有限公司 | A kind of method, apparatus of processing data, equipment and storage medium |
| CN110347674A (en) * | 2019-06-03 | 2019-10-18 | 阿里巴巴集团控股有限公司 | A kind of index creation method, device and equipment based on service identification |
| CN110427364A (en) * | 2019-06-21 | 2019-11-08 | 北京奇艺世纪科技有限公司 | A kind of data processing method, device, electronic equipment and storage medium |
| CN112241410A (en) * | 2020-10-22 | 2021-01-19 | 曙光信息产业(北京)有限公司 | Data storage method, data index construction method and device and computer equipment |
| CN112434039A (en) * | 2020-11-30 | 2021-03-02 | 浙江大华技术股份有限公司 | Data storage method, device, storage medium and electronic device |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113688142A (en) * | 2021-10-25 | 2021-11-23 | 北京金山云网络技术有限公司 | Index management method, device, storage medium and electronic equipment |
| CN114139021A (en) * | 2022-01-27 | 2022-03-04 | 云丁网络技术(北京)有限公司 | Index information management method and system |
| CN114139021B (en) * | 2022-01-27 | 2022-06-14 | 云丁网络技术(北京)有限公司 | Index information management method and system |
| CN114661722A (en) * | 2022-03-23 | 2022-06-24 | 天津同阳科技发展有限公司 | Data storage method, data indexing method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112883036B (en) | 2024-05-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112883036B (en) | Index creation method, device, storage server and storage medium | |
| CN108540965B (en) | Internet of things communication method and device based on LoRa technology and storage medium | |
| CN110381130B (en) | Keep-alive long connection method, device, communication terminal and storage medium | |
| CN114048168A (en) | Electronic archive filing method, device, equipment, system and storage medium | |
| CN112506904A (en) | Data quality evaluation method and device, terminal equipment and storage medium | |
| CN113038198A (en) | Video data downloading method and device, terminal equipment and storage medium | |
| CN112612526B (en) | Application program control method, device, terminal equipment and storage medium | |
| CN113918280A (en) | Dynamic adjustment method of virtual machine resources, terminal device and storage medium | |
| CN105721015A (en) | Dual-channel mobile terminal capable of multiplexing Flash ROM | |
| CN113220651A (en) | Operation data compression method and device, terminal equipment and storage medium | |
| CN112583907A (en) | Connection method and device, smart home equipment and computer readable storage medium | |
| CN113225234B (en) | Asset detection method, device, terminal equipment and computer readable storage medium | |
| CN113821252B (en) | Method and device for constructing applet, terminal equipment and storage medium | |
| CN112416940B (en) | Key value pair storage method, device, terminal equipment and storage medium | |
| CN112925744A (en) | Method and device for modifying flash package, terminal equipment and storage medium | |
| CN112346885A (en) | Electronic device control method, device, equipment and computer readable storage medium | |
| CN113391803A (en) | Method and device for creating object management engine, terminal equipment and storage medium | |
| CN112035036A (en) | Electronic whiteboard sharing method, system, terminal equipment and storage medium | |
| CN112583706A (en) | Gateway service processing method, gateway, receiving end, system and storage medium | |
| CN112800385A (en) | Data processing method, device, terminal equipment and storage medium | |
| CN113098902A (en) | Method and device for managing vulnerability of network equipment, management terminal equipment and storage medium | |
| CN112532971A (en) | Image processing method, image processing device, display terminal and computer readable storage medium | |
| CN113254184B (en) | Task scheduling method, device, scheduling system and storage medium | |
| CN114510168A (en) | Cursor control method and device, electronic device and computer readable storage medium | |
| CN111723123B (en) | Track prediction method, track prediction device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |