CN109033360B - Data query method, device, server and storage medium - Google Patents
Data query method, device, server and storage medium Download PDFInfo
- Publication number
- CN109033360B CN109033360B CN201810833389.6A CN201810833389A CN109033360B CN 109033360 B CN109033360 B CN 109033360B CN 201810833389 A CN201810833389 A CN 201810833389A CN 109033360 B CN109033360 B CN 109033360B
- Authority
- CN
- China
- Prior art keywords
- data
- index
- time period
- query request
- requested
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a data query method, a data query device, a server and a storage medium, wherein the method comprises the following steps: receiving a data query request; determining a data type of the business data requested by the data query request, wherein the data type comprises cold data and/or hot data; acquiring the service data requested by the data query request from a storage area of the service data based on the index data corresponding to the data type; the index data corresponding to the cold data comprises a bloom index, and the index data corresponding to the hot data comprises a binary index. By using the technical scheme provided by the embodiment of the invention, the data query speed can be effectively improved, and the storage resources are saved.
Description
Technical Field
The present invention relates to the field of information processing technologies, and in particular, to a data query method, apparatus, server, and storage medium.
Background
With the change of internet communication technology, the internet is widely applied to daily study, work and life of people. Various daily matters of people can be presented through a network. Therefore, huge amounts of data are being generated every day in the internet.
At present, a large amount of data generated by an internet service system is often stored and managed through a database, so that the data can be inquired and acquired from the database when the data is needed later. The HBase is a distributed and column-oriented open source database, data in the HBase is managed in a partition mode, different partitions need different partition server management, and when a user needs to query the data, the partition which contains the data needs to be determined from a data index table; then, determining the address of a partition server managing the partition; finally, the partitioned server is connected based on the address, and the data is searched. In the process of determining which partition contains the data from the data index table, because the data information recorded in the data index table is too much, the data in the data index table is often partitioned, and correspondingly, each partition has a corresponding partition manager. Therefore, a partition manager for managing the partition of the data index table corresponding to the data needs to be determined, which results in a complex data query process based on the HBase, seriously affects the query speed, and occupies a large amount of storage space. Therefore, there is a need to provide more efficient or faster solutions.
Disclosure of Invention
The invention provides a data query method, a data query device, a server and a storage medium, which can effectively improve the data query speed and save storage resources.
In a first aspect, the present invention provides a data query method, including:
receiving a data query request;
determining a data type of the business data requested by the data query request, wherein the data type comprises cold data and/or hot data;
acquiring the service data requested by the data query request from a storage area of the service data based on the index data corresponding to the data type;
the index data corresponding to the cold data comprises a bloom index, and the index data corresponding to the hot data comprises a binary index.
A second aspect provides a data query apparatus, including:
the data query request receiving module is used for receiving a data query request;
the data type determining module is used for determining the data type of the service data requested by the data query request, wherein the data type comprises cold data and/or hot data;
the query data acquisition module is used for acquiring the service data requested by the data query request from a storage area of the service data based on the index data corresponding to the data type;
the index data corresponding to the cold data comprises a bloom index, and the index data corresponding to the hot data comprises a binary index.
A third aspect provides a data query server comprising a processor and a memory, the memory having stored therein at least one instruction, at least one program, set of codes, or set of instructions, the at least one instruction, the at least one program, the set of codes, or the set of instructions being loaded and executed by the processor to implement the data query method according to the first aspect.
A fourth aspect provides a computer readable storage medium having stored therein at least one instruction, at least one program, a set of codes, or a set of instructions, which is loaded and executed by a processor to implement the data query method of the first aspect.
The data query method, the data query device, the server and the storage medium have the following technical effects:
the data type of the data to be inquired can be distinguished, and the data to be inquired can be acquired from the corresponding storage area by adopting different index data; the bloom index is used as index data of a large amount of cold data, so that the data query speed can be improved, the size of an index file can be reduced, and storage resources can be saved; the hot data which needs to be queried frequently adopts the binary index as index data, so that the data query speed can be improved better.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions and advantages of the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and other drawings can be obtained by those skilled in the art without creative efforts.
FIG. 1 is a schematic diagram of an application environment provided by an embodiment of the invention;
FIG. 2 is a flow chart of a data query method provided by an embodiment of the present invention;
FIG. 3 is an exemplary diagram of a data query interface provided by an embodiment of the invention;
FIG. 4 is a flowchart illustrating an embodiment of determining a data type of service data requested by the data query request according to the present invention;
FIG. 5 is a flowchart illustrating another embodiment of determining a data type of service data requested by the data query request according to the present invention;
FIG. 6 is a flowchart illustrating another embodiment of determining a data type of service data requested by the data query request according to the present invention;
FIG. 7 is an exemplary diagram of another data query interface provided by embodiments of the present invention;
FIG. 8 is a schematic structural diagram of a data query device according to an embodiment of the present invention;
FIG. 9 is a schematic structural diagram of another data query device according to an embodiment of the present invention;
FIG. 10 is a schematic structural diagram of another data query device according to an embodiment of the present invention;
fig. 11 is a schematic structural diagram of a server according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be obtained by a person skilled in the art without any inventive step based on the embodiments of the present invention, are within the scope of the present invention.
It should be noted that the terms "first," "second," and the like in the description and claims of the present invention and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the invention described herein are capable of operation in sequences other than those illustrated or described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or server that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
Referring to fig. 1, fig. 1 is a schematic diagram of an application environment according to an embodiment of the present invention, and as shown in fig. 1, the application environment may include a server 100 and a client 200.
In this embodiment of the present specification, the server 100 may be configured to collect service data of each service module of a service system, store the collected service data in the storage module 101, and when the service data is stored in the storage module 101, may establish first index data of the service data; and may store the first index data to the storage module 101.
Further, in order to improve the query speed, in this embodiment of the present specification, part of the service data may also be synchronously stored in the solid state disk 102, and correspondingly, when part of the service data is synchronously stored in the solid state disk 102, second index data of the part of the service data may be established, and the second index data may also be stored in the solid state disk 102.
In this embodiment, the storage module 101 and the solid state disk 102 may be located in the server 100, or may be located on a device accessible by other servers 100.
In practical application, the query demand of a user on service data in a service system in a recent period of time is often higher than the query demand of service data in a long-term time, in this specification embodiment, part of the service data stored in the solid state disk may include the service data in the recent period of time, in this specification embodiment, the service data in the recent period of time may be hot data, and correspondingly, data other than the hot data in the service data collected by the server 100 may be cold data.
In addition, it should be noted that, in order to control the storage cost, the storage module 101 storing the cold data in the embodiment of the present specification may include a mechanical hard disk (HDD conventional hard disk) with a low cost.
In the process of storing the service data, the cold data and the hot data are stored in different storage areas, the index data of the cold data and the hot data are established, the data to be inquired can be acquired from the corresponding storage areas according to the different index data, the hot data are stored in the solid state disk, and the speed of data inquiry can be greatly improved.
Specifically, the server 100 may include an electronic device having computing and network interaction functions; software may also be provided that runs in the electronic device to support data processing and network interaction.
In this embodiment of the present specification, the client 200 may be configured to provide a data query interface for a user, and send a data query request to the server 100 in response to a data query trigger instruction of the user on the data query interface; the server 100 may be configured to determine corresponding index data according to a data type (the data type may include cold data and/or hot data) corresponding to data requested by a data query request sent by the client 200, and obtain service data of the data query request from a corresponding storage area based on the index data; the server 100 may also be configured to send the acquired service data to the client 200, so that the client 200 presents the service data to the user.
Specifically, the client 200 may include a physical device such as a smart phone, a desktop computer, a tablet computer, a notebook computer, a digital assistant, a smart wearable device, and the like, and may also include software running in the physical device, such as a virtual machine. The operating system running on the client in the embodiment of the present invention may include, but is not limited to, an android system, an IOS system, linux, windows, and the like.
Further, in this embodiment, the first index data of the service data may be determined in combination with a data operation requirement of the service data, and in a specific embodiment, taking the service data as the pipeline data, the pipeline data generally does not need to be modified or deleted, but only needs to be inserted. Accordingly, in the embodiments of the present specification, the first index data may include, but is not limited to, a bloom index, and the bloom index may include: index files and data files. Specifically, the index file may be used to record information of the bloom filter. The data file may be used to record the existence of data to store information (e.g., data storage location, data size, etc.) in the source file.
A specific embodiment of the creation of the bloom index is described below.
In the embodiment of the present specification, assuming that the bloom filter has k layers, first, a hash value of a service data key (where the key may include some feature data in the service data, for example, the service data includes an input date, a birthday, a class, and a name; correspondingly, the key may be the feature data in any service data in the input date, the birthday, the class, and the name) is calculated according to a hash (hash) function of each layer, and then, a flag position 1 corresponding to each layer is set; and finally, writing the storage information of the service data in the source file into a data file, and recording the flag bit into the index file.
In practical application, the problem that the data volume of the running data is large often exists, and the establishment of the bloom index for the running data can greatly improve the data query speed, reduce the size of the index file and save the storage resources.
Further, in the embodiment of the present specification, the hot data is stored in the solid state disk, and the query speed of the binary index for the solid state disk which is read and written randomly is very fast. Therefore, in the embodiments of the present specification, the second index data may include, but is not limited to, a binary index. The binary index includes information sorted by index value.
The following introduces a specific embodiment of binary index establishment:
reading a source file of business data, and acquiring a key (the key may include certain characteristic data in the business data, for example, the business data includes an input date, a birthday, a class and a name; correspondingly, the key may be the characteristic data in any business data of the input date, the birthday, the class and the name) and storage information of the business data; storing the key and the storage information into a mapping table; then, the mapping table is sorted according to key to obtain the binary index of the service data.
The following describes an embodiment of a data query method provided by the present invention, and fig. 2 is a flowchart of a data query method provided by the embodiment of the present invention, and the present specification provides the method operation steps as described in the embodiment or the flowchart, but may include more or less operation steps based on conventional or non-inventive labor. The order of steps recited in the embodiments is merely one manner of performing the steps in a multitude of orders and does not represent the only order of execution. In practice, the system or server product may be implemented in a sequential or parallel manner (e.g., parallel processor or multi-threaded environment) according to the embodiments or methods shown in the figures. Specifically, as shown in fig. 2, the method may include:
s201: a data query request is received.
In this embodiment, the client side of the data query system may provide a data query interface for a user to trigger a data query request. Specifically, as shown in fig. 3, fig. 3 is an exemplary diagram of a data query interface according to an embodiment of the present invention. As can be seen from fig. 3, the user can enter the data content that needs to be queried and the time period of the query. In a specific embodiment, assume that the user needs to query the chronological data of service a from 2018, 20.5.20: 00:00 to 2018, 20.5.20: 19:00: 00. Accordingly, service a streaming data may be input at the query data content, the query time period is input from 2018, 5 month, 20 day 16:00:00 to 2018, 5 month, 20 day 19:00:00, and then the query button is clicked to trigger the data query request.
In addition, it should be noted that the query time period may include a time period as described above (20/2018/5/19: 00:00) or may include a time point, such as 16/00: 20/2018/5/20/2018/16/00: 00.
S203: determining a data type of the service data requested by the data query request, wherein the data type comprises cold data and/or hot data.
In this embodiment of this specification, after receiving a data query request, a data type of service data requested by the data query request may be determined, as shown in fig. 4, fig. 4 is a schematic flow chart of an embodiment of determining a data type of service data requested by the data query request provided by the present invention, and specifically, the method may include:
s401: and acquiring a time period corresponding to the service data requested by the data query request.
S403: and judging whether the corresponding time period comprises a time period in a preset latest time period.
In this embodiment of the present specification, the preset latest time period may include a preset time period before the current time. Such as one week ahead of the current time.
S405: and when the judgment result is negative, determining that the data type of the service data requested by the data query request is cold data.
In a specific embodiment, it is assumed that a time period corresponding to service data requested by a data query request is: 20/2018 at 16:00:00 to 20/5/20/19: 00: 00/2018 at a predetermined recent time period: the current time is one week ahead, and the current time is: 6/2018, 21/21 at 12:00: 00; accordingly, the preset latest time period may be a time period: 14/6/2018 at 12:00:00 to 21/6/2018 at 12:00: 00. Correspondingly, the time period from 2018, 5 month, 20 th day 16:00:00 to 2018, 5 month, 20 th day 19:00:00 corresponding to the service data requested by the data query request does not include the time period from 2018, 6 month, 14 th day 12:00:00 to 2018, 6 month, 21 th day 12:00:00, and correspondingly, the data type of the service data requested by the data query request can be determined to be cold data.
In other embodiments, as shown in fig. 5, the method may further include:
s407: and when the judgment result is yes, judging whether the corresponding time period comprises a time period other than the time period in the preset latest time period.
S409: and when the corresponding time period is judged not to include the time period except the time period in the preset latest time period, determining that the data type of the service data requested by the data query request is thermal data.
In a specific embodiment, it is assumed that a time period corresponding to service data requested by a data query request is: 16:00:00 at 20.6.2018 to 19:00:00 at 20.6.2018, with the predetermined recent time period: the current time is one week ahead, and the current time is: 6/2018, 21/21 at 12:00: 00; accordingly, the preset latest time period may be a time period: 14/6/2018 at 12:00:00 to 21/6/2018 at 12:00: 00. Correspondingly, the time period from 2018, 6, 20, 16:00:00 to 2018, 6, 20, 19:00:00 of the service data requested by the data query request includes a time period from 2018, 6, 14, 12:00:00 to 2018, 6, 21, 12:00:00, and correspondingly, whether the corresponding time period (from 2018, 6, 20, 16:00:00 to 2018, 6, 20, 19:00:00) includes a time period other than the time period from 2018, 6, 14, 12:00:00 to 2018, 21, 12:00:00) or not can be judged, and correspondingly, the time period other than the time period from 2018, 6, 20, 16:00:00 to 2018, 6, 19:00:00 does not include the time period other than the time period from 2018, 6, 14, 12:00:00 to 2018, 6, 21, 12:00: 00; accordingly, it may be determined that the data type of the service data requested by the data query request is hot data.
In other embodiments, as shown in fig. 6, the method may further include:
s411: when the corresponding time period is judged to comprise the time periods except the time period in the preset latest time period, determining that the data type of the service data requested by the data query request comprises cold data and hot data;
the data type of the requested service data corresponding to the other time periods is cold data, and the data type of the requested service data corresponding to the time period in the preset latest time period included in the corresponding time period is hot data.
In a specific embodiment, it is assumed that a time period corresponding to service data requested by a data query request is: 20/2018 at 16:00:00 to 20/6/2018 at 19:00:00, with the predetermined recent time period: the current time is one week ahead, and the current time is: 6/2018, 21/21 at 12:00: 00; accordingly, the preset latest time period may be a time period: 14/6/2018 at 12:00:00 to 21/6/2018 at 12:00: 00. Correspondingly, the time slot corresponding to the service data requested by the data query request is from 2018, 5, 20, 16:00:00 to 2018, 6, 20, 19:00:00, and includes the time slot from 2018, 6, 14, 12:00:00 to 2018, 6, 21, 12:00:00, and correspondingly, it is possible to judge whether or not the corresponding time zone (20/2018/16: 00: 00-20/2018/20/19: 00:00) includes a time zone other than the time zone in the preset latest time zone (14/2018/14/12: 00: 00-2018/21/12: 00:00), and accordingly, here, the time period other than the time period in the range from 2018 year 5 month 20 day 16:00:00 to 2018 year 6 month 20 day 19:00:00 includes 2018 year 6 month 14 day 12:00:00 to 2018 year 6 month 21 day 12:00:00 (2018 year 5 month 20 day 16:00:00 to 2018 year 6 month 14 day 12:00: 00); correspondingly, the data type of the service data requested by the data query request can be determined to comprise cold data and hot data, wherein the data type of the service data requested corresponding to the other time periods (20/5/2018 and 16:00: 00-2018 and 6/14/2018 and 12:00:00) is cold data; the data type of the requested service data corresponding to a time period (12: 00:00 in 14/6/2018 to 19:00:00 in 20/6/2018) in a preset latest time period included in the corresponding time period is hot data.
S205: and acquiring the service data requested by the data query request from a storage area of the service data based on the index data corresponding to the data type.
The index data corresponding to the cold data comprises a bloom index, and the index data corresponding to the hot data comprises a binary index.
In the embodiment of the present specification, service data of different data types are stored in different storage areas, and different index data are set. After determining the data type of the requested service data, the service data requested by the data query may be obtained from the storage area of the service data based on the index data corresponding to the data type.
In a specific embodiment, taking the first index data as a bloom index as an example, when the data type corresponding to the data requested by the data query request includes cold data, the obtaining, from the storage area of the service data, the service data requested by the data query request based on the index data corresponding to the data type may include:
and acquiring the service data requested by the data query request from a preset hard disk based on the bloom index.
In an embodiment of the present specification, the preset hard disk may include a storage module storing cold data.
Specifically, the following describes specific steps of obtaining the service data requested by the data query request from the storage area of the service data based on the bloom index:
calculating a hash value of a key of the service data according to a hash function of each layer of the bloom filter, checking whether corresponding zone bits are all 1 according to the hash value of each layer, and determining that the service data exists if k layers (assuming that the bloom filter has k layers) are all 1; then, searching out the storage information of the service data in the source file from the data file; and finally, acquiring the service data based on the source file of the storage area corresponding to the storage information.
In addition, if the k layer has a flag bit not equal to 1, the data is considered to be absent, and the information of query failure can be returned.
In another specific embodiment, taking the second index data as a binary index as an example, when the data type corresponding to the data requested by the data query request includes hot data, the obtaining, from the storage area of the business data, the business data requested by the data query request based on the index data corresponding to the data type includes:
and acquiring the service data requested by the data query request from the solid state disk based on the binary index.
Specifically, the following introduces specific steps of obtaining the service data requested by the data query request from the solid state disk based on the binary index:
the two-division index is a mapping table which is sorted according to the keys of the service data, so that the keys can be used for performing two-division search to find corresponding storage information, and the storage information is given to find the service data corresponding to the keys from the source file in the solid state disk.
In another specific embodiment, taking the first index data as a bloom index and the second index data as a binary index as an example, when the data type corresponding to the data requested by the data query request includes hot data and cold data, the obtaining the service data requested by the data query request from the storage area of the service data based on the index data corresponding to the data type may include:
and acquiring inter-cooling data in the service data requested by the data query request from a preset hard disk based on the bloom index.
And acquiring hot data in the service data requested by the data query request from the solid state disk based on the binary index.
In another embodiment, the method may further include:
acquiring service data, and storing the service data to a preset hard disk;
and establishing a bloom index of the service data, and storing the bloom index to the preset hard disk.
In another embodiment, the method may further include:
storing the hot data in the service data to a solid state disk;
and establishing a binary index of the hot data, and storing the binary index to the solid state disk.
According to the technical scheme provided by the embodiment of the specification, the bloom indexes are established for a large amount of cold data for subsequent query indexes, so that the data query speed is greatly improved, the size of the index file is reduced, and the storage resources are saved. In addition, the hot data which needs to be inquired frequently is stored in the solid state disk, the binary index is used as index data of the hot data, the advantage that the binary index is high in inquiring speed of the solid state disk which is read and written randomly is fully utilized, and the data inquiring speed can be improved better; and the cold and hot data are stored in different storage areas in combination with actual requirements, so that the cost can be effectively controlled.
Another example of a data query interface provided by an embodiment of the present specification is described below in conjunction with fig. 7:
in the embodiment of the description, different index data are adopted for cold and hot data and stored in different storage areas, and for the hot data, a mode of inquiring the hot data from the solid state disk by storing the hot data in the solid state disk and adopting a binary index can realize a faster inquiring speed than the cold data. Therefore, in order to improve the experience of the user in data query, in this embodiment of the specification, prompt information for accelerating query may be set on the data query interface, and specifically, as shown in fig. 7, the prompt information for accelerating query may be information of the current speed time period.
In addition, the embodiment of the present specification may further provide a trigger interface for accelerating query on the data query interface, and only when a user triggers an acceleration query instruction, data query based on binary index from the solid state disk is adopted for the hot data that can be accelerated; otherwise, the service data can be directly acquired from the preset hard disk according to the bloom index.
An embodiment of the present invention further provides a data query apparatus, as shown in fig. 8, the apparatus includes:
a data query request receiving module 810, which may be configured to receive a data query request;
a data type determining module 820, configured to determine a data type of the service data requested by the data query request, where the data type includes cold data and/or hot data;
the query data obtaining module 830 may be configured to obtain, based on the index data corresponding to the data type, service data requested by the data query request from a storage area of the service data;
the index data corresponding to the cold data comprises a bloom index, and the index data corresponding to the hot data comprises a binary index.
In another embodiment, the data type determining module 820 may include:
a time period obtaining unit, configured to obtain a time period corresponding to service data requested by the data query request;
the first judging unit may be configured to judge whether the corresponding time period includes a time period in a preset latest time period;
the first data type determining unit may be configured to determine that the data type of the service data requested by the data query request is cold data when the result of the determination by the first determining unit is negative.
In another embodiment, the data type determining module 820 may further include:
a second determining unit, configured to determine whether a time period other than the time period in the preset latest time period is included in the corresponding time period when the result of the determination by the first determining unit is yes;
the second data type determining unit may be configured to determine that the data type of the service data requested by the data query request is hot data when the result of the determination by the second determining unit is negative.
In another embodiment, the data type determining module 820 may further include:
a third data type determining unit, configured to determine that the data type of the service data requested by the data query request includes cold data and hot data when the result of the determination by the second determining unit is yes;
the data type of the requested service data corresponding to the other time periods is cold data, and the data type of the requested service data corresponding to the time period in the preset latest time period included in the corresponding time period is hot data.
The device and method embodiments in the device embodiment described are based on the same inventive concept.
An embodiment of the present invention further provides another data query apparatus, as shown in fig. 9, the apparatus includes:
the data obtaining module 910 may be configured to obtain service data.
The first data storage module 920 may be configured to store the service data in a preset hard disk.
The first index establishing module 930 may be configured to establish a bloom index of the service data.
A first index data storage module 940, configured to store the bloom index to the preset hard disk.
The data query request receiving module 950 may be configured to receive a data query request.
The data type determining module 960 may be configured to determine the data type of the service data requested by the data query request, where the data type includes cold data and/or hot data.
The query data obtaining module 970 may be configured to obtain, based on the index data corresponding to the data type, the service data requested by the data query request from the storage area of the service data.
The index data corresponding to the cold data comprises a bloom index, and the index data corresponding to the hot data comprises a binary index.
The device and method embodiments in the device embodiment described are based on the same inventive concept.
An embodiment of the present invention further provides another data query apparatus, as shown in fig. 10, the apparatus includes:
the data obtaining module 1010 may be configured to obtain service data.
The first data storage module 1020 may be configured to store the service data in a preset hard disk.
The first index creating module 1030 may be configured to create a bloom index of the service data.
The first index data storage module 1040 may be configured to store the bloom index to the preset hard disk.
The second data storage module 1050 may be configured to store hot data in the service data to a solid state disk;
a second index establishing module 1060, which may be configured to establish a binary index of the thermal data;
a second index data storage module 1070 may be configured to store the binary index to the solid state disk.
The data query request receiving module 1080 may be configured to receive a data query request.
A data type determining module 1090 may be configured to determine a data type of the service data requested by the data query request, where the data type includes cold data and/or hot data.
The query data obtaining module 1100 may be configured to obtain the service data requested by the data query request from a storage area of the service data based on the index data corresponding to the data type.
The device and method embodiments in the device embodiment described are based on the same inventive concept.
An embodiment of the present invention provides a data query server, which includes a processor and a memory, where the memory stores at least one instruction, at least one program, a set of codes, or a set of instructions, and the at least one instruction, the at least one program, the set of codes, or the set of instructions is loaded and executed by the processor to implement the data query method provided by the above method embodiment.
The memory may be used to store software programs and modules, and the processor may execute various functional applications and data processing by operating the software programs and modules stored in the memory. The memory can mainly comprise a program storage area and a data storage area, wherein the program storage area can store an operating system, application programs needed by functions and the like; the storage data area may store data created according to use of the apparatus, and the like. Further, the memory may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid state storage device. Accordingly, the memory may also include a memory controller to provide the processor access to the memory.
The method provided by the embodiment of the invention can be executed in a mobile terminal, a computer terminal, a server or a similar operation device. Taking the server as an example, fig. 11 is a hardware block diagram of the server of the data query method according to the embodiment of the present invention. As shown in fig. 11, the server 1100 may have a relatively large difference due to different configurations or performances, and may include one or more Central Processing Units (CPUs) 1110 (the processors 1110 may include but are not limited to Processing devices such as a microprocessor MCU or a programmable logic device FPGA), a memory 1130 for storing data, and one or more storage media 1120 (e.g., one or more mass storage devices) for storing applications 1123 or data 1122. The memory 1130 and the storage medium 1120 may be, among other things, transient storage or persistent storage. The program stored in the storage medium 1120 may include one or more modules, each of which may include a series of instruction operations for a server. Still further, the central processor 1110 may be configured to communicate with the storage medium 1120, and execute a series of instruction operations in the storage medium 1120 on the server 1100. The server 1100 may also include one or more power supplies 1160, one or more wired or wireless network interfaces 1150, one or more input-output interfaces 1140, and/or one or more operating systems 1121, such as Windows Server, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.
The input output interface 1140 may be used to receive or transmit data via a network. Specific examples of the network described above may include a wireless network provided by a communication provider of the server 1100. In one example, i/o Interface 1140 includes a Network adapter (NIC) that may be coupled to other Network devices via a base station to communicate with the internet. In one example, the input/output interface 1140 can be a Radio Frequency (RF) module, which is used for communicating with the internet in a wireless manner.
It will be understood by those skilled in the art that the structure shown in fig. 11 is only an illustration and is not intended to limit the structure of the electronic device. For example, server 1100 may also include more or fewer components than shown in FIG. 11, or have a different configuration than shown in FIG. 11.
Embodiments of the present invention also provide a storage medium, which may be disposed in a server to store at least one instruction, at least one program, a code set, or a set of instructions related to implementing a data query method in the method embodiments, where the at least one instruction, the at least one program, the code set, or the set of instructions is loaded and executed by the processor to implement the data query method provided by the above method embodiments.
Alternatively, in this embodiment, the storage medium may be located in at least one network server of a plurality of network servers of a computer network. Optionally, in this embodiment, the storage medium may include, but is not limited to: a U-disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a removable hard disk, a magnetic or optical disk, and other various media capable of storing program codes.
According to the data query method, the data query device, the server or the storage medium, the bloom indexes are established for a large amount of cold data for subsequent query indexes, so that the data query speed is greatly improved, the size of the index file is reduced, and the storage resources are saved. In addition, the hot data which needs to be inquired frequently is stored in the solid state disk, the binary index is used as index data of the hot data, the advantage that the binary index is high in inquiring speed of the solid state disk which is read and written randomly is fully utilized, and the data inquiring speed can be improved better; and the cold and hot data are stored in different storage areas in combination with actual requirements, so that the cost can be effectively controlled.
It should be noted that: the precedence order of the above embodiments of the present invention is only for description, and does not represent the merits of the embodiments. And specific embodiments thereof have been described above. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims may be performed in a different order than in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing may also be possible or may be advantageous.
The embodiments in the present specification are described in a progressive manner, and the same and similar parts among the embodiments are referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, as for the device and server embodiments, since they are substantially similar to the method embodiments, the description is simple, and the relevant points can be referred to the partial description of the method embodiments.
It will be understood by those skilled in the art that all or part of the steps for implementing the above embodiments may be implemented by hardware, or may be implemented by a program instructing relevant hardware, where the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disk, etc.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents, improvements and the like that fall within the spirit and principle of the present invention are intended to be included therein.
Claims (10)
1. A method for data query, the method comprising:
acquiring service data, and storing the service data to a preset hard disk; establishing a bloom index of the service data, and storing the bloom index to the preset hard disk; storing the hot data in the service data to a solid state disk; establishing a binary index of the hot data, and storing the binary index to the solid state disk;
receiving a data query request;
when the data query request carries an acceleration query instruction, determining the data type of the service data requested by the data query request, wherein the data type comprises cold data and/or hot data; the service data is flow data;
acquiring the service data requested by the data query request from a storage area of the service data based on the index data corresponding to the data type;
the index data corresponding to the cold data comprises a bloom index, and the index data corresponding to the hot data comprises a binary index;
when the data type corresponding to the data requested by the data query request includes cold data, the obtaining the service data requested by the data query request from the storage area of the service data based on the index data corresponding to the data type includes: acquiring service data requested by the data query request from a preset hard disk based on the bloom index, wherein the preset hard disk comprises a mechanical hard disk;
when the data type corresponding to the data requested by the data query request includes hot data, the obtaining, from a storage area of business data, the business data requested by the data query request based on the index data corresponding to the data type includes: acquiring the service data requested by the data query request from the solid state disk based on the binary index;
and when the data query request does not carry an acceleration query instruction, acquiring the service data requested by the data query request from the mechanical hard disk according to the bloom index.
2. The method of claim 1, wherein the determining the data type of the service data requested by the data query request comprises:
acquiring a time period corresponding to the service data requested by the data query request;
judging whether the corresponding time period comprises a time period in a preset latest time period or not;
and when the judgment result is negative, determining that the data type of the service data requested by the data query request is cold data.
3. The method of claim 2, further comprising:
when the judgment result is yes, judging whether the corresponding time period comprises a time period other than the time period in the preset latest time period;
and when the corresponding time period is judged not to include the time period except the time period in the preset latest time period, determining that the data type of the service data requested by the data query request is thermal data.
4. The method of claim 3, further comprising:
when the corresponding time period is judged to comprise the time periods except the time period in the preset latest time period, determining that the data type of the service data requested by the data query request comprises cold data and hot data;
the data type of the requested service data corresponding to the other time periods is cold data, and the data type of the requested service data corresponding to the time period in the preset latest time period included in the corresponding time period is hot data.
5. A data query apparatus, characterized in that the apparatus comprises:
the data acquisition module is used for acquiring service data; the first data storage module is used for storing the service data to a preset hard disk; the first index establishing module is used for establishing a bloom index of the service data; the first index data storage module is used for storing the bloom index to the preset hard disk; the second data storage module is used for storing hot data in the service data to the solid state disk; the second index establishing module is used for establishing a binary index of the thermal data; the second index data storage module is used for storing the binary index to the solid state disk;
the data query request receiving module is used for receiving a data query request;
a data type determining module, configured to determine a data type of service data requested by the data query request when the data query request carries an acceleration query instruction, where the data type includes cold data and/or hot data; the service data is flow data;
the query data acquisition module is used for acquiring the service data requested by the data query request from a storage area of the service data based on the index data corresponding to the data type;
the index data corresponding to the cold data comprises a bloom index, and the index data corresponding to the hot data comprises a binary index;
when the data type corresponding to the data requested by the data query request includes cold data, the query data acquisition module is specifically configured to acquire service data requested by the data query request from a preset hard disk based on the bloom index, where the preset hard disk includes a mechanical hard disk;
when the data type corresponding to the data requested by the data query request includes hot data, the query data obtaining module is specifically configured to: acquiring the service data requested by the data query request from the solid state disk based on the binary index;
and when the data query request does not carry an acceleration query instruction, acquiring the service data requested by the data query request from the mechanical hard disk according to the bloom index.
6. The apparatus of claim 5, wherein the data type determination module comprises:
a time period obtaining unit, configured to obtain a time period corresponding to service data requested by the data query request;
the first judging unit is used for judging whether the corresponding time period comprises a time period in a preset latest time period or not;
a first data type determining unit, configured to determine that the data type of the service data requested by the data query request is cold data when the result of the determination by the first determining unit is negative.
7. The apparatus of claim 6, wherein the data type determination module further comprises:
a second judging unit, configured to, when the result of the judgment by the first judging unit is yes, judge whether the corresponding time period includes a time period other than a time period in the preset latest time period;
a second data type determining unit, configured to determine that the data type of the service data requested by the data query request is hot data when a result of the determination by the second determining unit is negative.
8. The apparatus of claim 7, wherein the data type determination module further comprises:
a third data type determining unit, configured to determine that the data type of the service data requested by the data query request includes cold data and hot data when the result of the determination by the second determining unit is yes;
the data type of the requested service data corresponding to the other time periods is cold data, and the data type of the requested service data corresponding to the time period in the preset latest time period included in the corresponding time period is hot data.
9. A data query server, comprising a processor and a memory, wherein the memory has stored therein at least one instruction, at least one program, a set of codes, or a set of instructions, which is loaded and executed by the processor to implement the data query method of any one of claims 1 to 4.
10. A computer readable storage medium having stored therein at least one instruction, at least one program, a set of codes, or a set of instructions, which is loaded and executed by a processor to implement the data query method of any one of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810833389.6A CN109033360B (en) | 2018-07-26 | 2018-07-26 | Data query method, device, server and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810833389.6A CN109033360B (en) | 2018-07-26 | 2018-07-26 | Data query method, device, server and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109033360A CN109033360A (en) | 2018-12-18 |
| CN109033360B true CN109033360B (en) | 2020-11-17 |
Family
ID=64646407
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810833389.6A Active CN109033360B (en) | 2018-07-26 | 2018-07-26 | Data query method, device, server and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109033360B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113190507B (en) * | 2021-05-14 | 2022-06-03 | 杭州海康威视数字技术股份有限公司 | Index information synchronization method and device and electronic equipment |
Families Citing this family (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111475506B (en) * | 2020-03-30 | 2024-03-01 | 广州虎牙科技有限公司 | Method, device, system, equipment and storage medium for data storage and query |
| CN113704346A (en) * | 2020-05-20 | 2021-11-26 | 杭州海康威视数字技术股份有限公司 | Hbase table cold and hot data conversion method and device and electronic equipment |
| CN112650453A (en) * | 2020-12-31 | 2021-04-13 | 北京千方科技股份有限公司 | Method and system for storing and inquiring traffic data |
| CN113163003B (en) * | 2021-04-09 | 2023-05-30 | 北京汇钧科技有限公司 | File transmission method, cloud server, client and cloud node router |
| CN114513373B (en) * | 2022-04-20 | 2022-11-15 | 北京掌趣无限科技有限公司 | Trusted data exchange method, device, system, electronic equipment and storage medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104850564A (en) * | 2014-02-18 | 2015-08-19 | 腾讯科技（深圳）有限公司 | Index search method and system of data file |
| CN105630847A (en) * | 2014-11-21 | 2016-06-01 | 深圳市腾讯计算机系统有限公司 | Data storage method as well as data query method, apparatus and system |
| CN108021585A (en) * | 2016-10-28 | 2018-05-11 | 腾讯科技（深圳）有限公司 | Distributed data storage method and device |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR101933455B1 (en) * | 2011-11-23 | 2019-01-02 | 삼성전자주식회사 | Autostereoscropic display device |
-
2018
- 2018-07-26 CN CN201810833389.6A patent/CN109033360B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104850564A (en) * | 2014-02-18 | 2015-08-19 | 腾讯科技（深圳）有限公司 | Index search method and system of data file |
| CN105630847A (en) * | 2014-11-21 | 2016-06-01 | 深圳市腾讯计算机系统有限公司 | Data storage method as well as data query method, apparatus and system |
| CN108021585A (en) * | 2016-10-28 | 2018-05-11 | 腾讯科技（深圳）有限公司 | Distributed data storage method and device |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113190507B (en) * | 2021-05-14 | 2022-06-03 | 杭州海康威视数字技术股份有限公司 | Index information synchronization method and device and electronic equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109033360A (en) | 2018-12-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109033360B (en) | Data query method, device, server and storage medium | |
| US11068441B2 (en) | Caseless file lookup in a distributed file system | |
| US11036690B2 (en) | Global namespace in a heterogeneous storage system environment | |
| CN107704202B (en) | Method and device for quickly reading and writing data | |
| CN106649403B (en) | Index implementation method and system in file storage | |
| CN105468642A (en) | Data storage method and apparatus | |
| US10909086B2 (en) | File lookup in a distributed file system | |
| CN107103011B (en) | Method and device for realizing terminal data search | |
| WO2014110940A1 (en) | A method, apparatus and system for storing, reading the directory index | |
| US10824612B2 (en) | Key ticketing system with lock-free concurrency and versioning | |
| US10417192B2 (en) | File classification in a distributed file system | |
| CN105227672A (en) | The method and system that data store and access | |
| US11157456B2 (en) | Replication of data in a distributed file system using an arbiter | |
| CN109471860B (en) | Large-scale charging pile data processing method and device for electric vehicle charging network | |
| CN114610680A (en) | Method, device and equipment for managing metadata of distributed file system and storage medium | |
| EP3343395B1 (en) | Data storage method and apparatus for mobile terminal | |
| CN112272137A (en) | Mass data management in communication applications through multiple mailboxes | |
| CN107844483B (en) | File management method and device | |
| CN110347656B (en) | Method and device for managing requests in file storage system | |
| CN115129789A (en) | Bucket index storage method, device and medium of distributed object storage system | |
| EP3572951A1 (en) | A method for handling requests in a storage system and a storage node for a storage system | |
| CN115794876A (en) | Fragment processing method, device, equipment and storage medium for service data packet | |
| US20170116219A1 (en) | Efficient differential techniques for metafiles | |
| CN108509438A (en) | A kind of ElasticSearch fragments extended method | |
| CN112685557A (en) | Visualized information resource management method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |